Issue with VCS configured with uploaded SSH while getting List remote refs
Hi,
In a last couple of week we have noticed that we are having issues with getting list of remote refs on our VCS on our builds. This is causing inconsistent random build fails (random build will fail at random time). VCS are configured to use azure git repos over ssh with uploaded private key with checkout on agents, with pooling mechanism (different projects use different timers to avoid rate limiting issues). We have checked Azure DevOps logs for the user that we are using inside our builds, and there are no issues reported on that side. Can someone please help?
TeamCity version and build number: Teamcity Professional 2020.2.2 (build 85899)
Operating System:
Server: Windows Server 2012
Agents: Windows 10
Kind regards,
Tarik Eminagic
Here is a Debug log that we are getting when this error occurs (url to git repo is omitted due privacy reasons and replaced with ssh_git_repo_url placeholder):
[2021-02-04 01:02:09,604] DEBUG [l executor 1442] - ggers.vcs.git.OperationContext - The error during GIT vcs operation list remote refs
org.eclipse.jgit.errors.TransportException: ssh_git_repo_url: Session.connect: java.net.SocketException: Connection reset
at org.eclipse.jgit.transport.JschConfigSessionFactory.getSession(JschConfigSessionFactory.java:191)
at org.eclipse.jgit.transport.SshTransport.getSession(SshTransport.java:140)
at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.<init>(TransportGitSsh.java:280)
at org.eclipse.jgit.transport.TransportGitSsh.openFetch(TransportGitSsh.java:170)
at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport$1.call(GitVcsSupport.java:531)
at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport$1.call(GitVcsSupport.java:519)
at jetbrains.buildServer.buildTriggers.vcs.git.Retry.retry(Retry.java:58)
at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.lambda$getRemoteRefs$8(GitVcsSupport.java:519)
at jetbrains.buildServer.serverSide.impl.BaseAccessChecker.runWithDisabledChecks(BaseAccessChecker.java:7)
at jetbrains.buildServer.serverSide.impl.SecondaryNodeSecurityManager.executeSafe(SecondaryNodeSecurityManager.java:9)
at jetbrains.buildServer.serverSide.IOGuardInitializer$IOGuardDelegateImpl.allowNetworkCall(IOGuardInitializer.java:5)
at jetbrains.buildServer.serverSide.IOGuard.allowNetworkCall(IOGuard.java:82)
at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.getRemoteRefs(GitVcsSupport.java:517)
at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.getRemoteRefs(GitVcsSupport.java:503)
at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.lambda$getCurrentState$0(GitVcsSupport.java:182)
at jetbrains.buildServer.buildTriggers.vcs.git.RepositoryManagerImpl.runWithDisabledRemove(RepositoryManagerImpl.java:256)
at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.getCurrentState(GitVcsSupport.java:178)
at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.getCurrentState(GitVcsSupport.java:173)
at jetbrains.buildServer.buildTriggers.vcs.git.GitCollectChangesPolicy.getCurrentState(GitCollectChangesPolicy.java:111)
at jetbrains.vcs.api.services.impl.RepositoryStateServiceProvider$1.getCurrentState(RepositoryStateServiceProvider.java:9)
at jetbrains.buildServer.vcs.impl.VcsRootInstanceImpl.lambda$getCurrentState$0(VcsRootInstanceImpl.java:127)
at jetbrains.buildServer.serverSide.impl.BaseAccessChecker.runWithDisabledChecks(BaseAccessChecker.java:57)
at jetbrains.buildServer.serverSide.impl.SecondaryNodeSecurityManager.executeSafe(SecondaryNodeSecurityManager.java:30)
at jetbrains.buildServer.serverSide.IOGuardInitializer$IOGuardDelegateImpl.allowNetworkAndCommandLine(IOGuardInitializer.java:14)
at jetbrains.buildServer.serverSide.IOGuard.allowNetworkAndCommandLine(IOGuard.java:117)
at jetbrains.buildServer.vcs.impl.VcsRootInstanceImpl.getCurrentState(VcsRootInstanceImpl.java:119)
at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector$CurrentStateOperation.call(VcsChangesStatesCollector.java:1)
at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector$CurrentStateOperation.call(VcsChangesStatesCollector.java:2)
at jetbrains.buildServer.vcs.VcsOperationProgressProviderImpl.runWithProgress(VcsOperationProgressProviderImpl.java:29)
at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector.getCurrentState(VcsChangesStatesCollector.java:14)
at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector.lambda$null$1(VcsChangesStatesCollector.java:38)
at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:76)
at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector.lambda$getCollectStatesTasks$2(VcsChangesStatesCollector.java:76)
at jetbrains.buildServer.vcs.impl.ImmediateFutureExecService$2.call(ImmediateFutureExecService.java:2)
at jetbrains.buildServer.serverSide.impl.ImmediateFuture.get(ImmediateFuture.java:63)
at jetbrains.buildServer.serverSide.impl.ImmediateFuture.get(ImmediateFuture.java:73)
at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector.collectStatesForAllRoots(VcsChangesStatesCollector.java:72)
at jetbrains.buildServer.vcs.impl.VcsChangesStatesCollector.getCurrentStateSnapshot(VcsChangesStatesCollector.java:119)
at jetbrains.buildServer.vcs.impl.VcsChangesFetcher.getStateAndCreateChangesCollectingTasks(VcsChangesFetcher.java:89)
at jetbrains.buildServer.vcs.impl.VcsChangesLoader.doLoadChanges(VcsChangesLoader.java:116)
at jetbrains.buildServer.vcs.impl.VcsChangesLoader.tryLoadChanges(VcsChangesLoader.java:114)
at jetbrains.buildServer.serverSide.impl.VcsModificationChecker$CollectChangesAction$1.run(VcsModificationChecker.java:23)
at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:76)
at jetbrains.buildServer.serverSide.impl.VcsModificationChecker$CollectChangesAction.run(VcsModificationChecker.java:9)
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.jcraft.jsch.JSchException: Session.connect: java.net.SocketException: Connection reset
at com.jcraft.jsch.Session.connect(Session.java:565)
at org.eclipse.jgit.transport.JschConfigSessionFactory.getSession(JschConfigSessionFactory.java:145)
... 47 more
Please sign in to leave a comment.
I am having what looks to be the same issue. I think it has been since I upgraded to 2020.2.2 but it could have been before. I am about to reach out to support since it is getting worse and we are dead in the water.
I think it has to do with upgrade, since it have started happening on my project after the upgrade now that I think of. Before this version we had the similar error, but on azure side we have noticed that we were hitting the rate limit of the azure devops calls. After we increased pooling time + added some delay so that all builds do not pool at the same time (we have around 100 builds) it has removed the issue. And when this happened again (after the upgrade) I have first taught that it was the same issue again. But, while on old issue, there were logs on azure devops indicating the rate limiting issue, here there was no logs like that. And exception itself doesn't point me to the right direction.
Anyhow, I have also created the ticket to their issue tracker (https://youtrack.jetbrains.com/issue/TW-70157). Hopefully they will fix it soon, since it is getting to that point where it gets annoying.