'git fetch' command failed. exception: Timeout exception
I've moved a Git repository from a shared folder on the network to a SSH server running gitolite. I've added a public/private key for TeamCity to use and pointed the VCS root at it. This all works OK as *Test Connection* is successful and my build configuration shows the pending changes.
However, when TeamCity attempts to fetch those pending changes I get the following error:
'git fetch' command failed.
exception: Timeout exception
Switching to verbose logging I see that it timed out after 90 seconds - I would expect under normal circumstances that it would be a matter of a couple of seconds to fetch these changes.
Since I started trying to troubleshoot the problem I have changed my VCS root to "Default Private Key" and added the required .ssh folder with config, id_rsa and known_hosts files. I saw the fingerprint and then password messages in the log until I put the necessary stuff in these files, and now my server is back to giving the Timeout exception again just as it does when I specify the key in the VCS root.
If I run Git from the command line I am able to clone the repository without any prompts (when I have the .ssh folder set up as above). This is from a different user account as TC is running as a service under the default SYSTEM account, but I don't think that's the cause because as I said I can see that TC has picked up the changes to the .ssh folder for that account.
So, I've spent the whole evening on this, now I am stuck and I've run out of things to try. Any ideas? Is there any way I can see the command executed and any response that is being returned from the server?
:(
Please sign in to leave a comment.
Hi Graeme,
what version of TeamCity do you use? Also please enable debug-vcs logging preset and proved teamcity-vcs.log and build log related to the issue.
Hi Dmitry - sorry I missed off that vital information! I'm running TeamCity 6.5.3 and here is a portion of the log:
[2011-08-09 07:32:37,422] DEBUG [tor 344 {id=43}] - ggers.vcs.git.FetchCommandImpl - Start fetch process for (\\myserver\builds\TeamCity\BuildServer\system\caches\git\git-7F9FCF32.git, git@192.168.0.77:MyRepo#+refs/heads/release-5.1:refs/heads/release-5.1) [2011-08-09 07:32:37,581] DEBUG [tor 344 {id=43}] - ggers.vcs.git.FetchCommandImpl - Fetch process for (\\myserver\builds\TeamCity\BuildServer\system\caches\git\git-7F9FCF32.git, git@192.168.0.77:MyRepo#+refs/heads/release-5.1:refs/heads/release-5.1) started [2011-08-09 07:34:07,664] DEBUG [tor 344 {id=43}] - t.FetchCommandImpl.Performance - [fetch in separate process] root= (\\myserver\builds\TeamCity\BuildServer\system\caches\git\git-7F9FCF32.git, git@192.168.0.77:MyRepo#+refs/heads/release-5.1:refs/heads/release-5.1), took 90242ms [2011-08-09 07:34:07,664] DEBUG [tor 344 {id=43}] - ggers.vcs.git.OperationContext - The error during GIT vcs operation retrieving current version jetbrains.buildServer.vcs.VcsException: 'git fetch' command failed. exception: Timeout exception at jetbrains.buildServer.buildTriggers.vcs.git.CommandLineUtil.getCommandLineError(CommandLineUtil.java:41) at jetbrains.buildServer.buildTriggers.vcs.git.FetchCommandImpl.fetchInSeparateProcess(FetchCommandImpl.java:169) at jetbrains.buildServer.buildTriggers.vcs.git.FetchCommandImpl.fetch(FetchCommandImpl.java:67) at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.fetch(GitVcsSupport.java:1078) at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.fetchBranchData(GitVcsSupport.java:1060) at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.getCurrentVersion(GitVcsSupport.java:991) at jetbrains.buildServer.vcs.impl.VcsRootInstancesManagerImpl$SVcsRootInstance.getCurrentRevision(VcsRootInstancesManagerImpl.java:25) at jetbrains.buildServer.vcs.impl.VcsManagerImpl.getRevisionsForAllRoots(VcsManagerImpl.java:181) at jetbrains.buildServer.vcs.impl.VcsManagerImpl.loadChanges(VcsManagerImpl.java:709) at jetbrains.buildServer.serverSide.impl.auth.SecuredVcsManager.loadChanges(SecuredVcsManager.java:64) at jetbrains.buildServer.serverSide.impl.VcsModificationChecker$1.run(VcsModificationChecker.java:20) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source)I just tried changing back to my previous repository (on a shared folder) and I still get the Timeout exception! I upgraded to 6.5.3 and then changed my repositories over - maybe this problem is to do with 6.5.3 and not the git repository?
Error might be caused by upgrade. In 6.5.2 we changed how git timeouts work. We introduced an internal property "teamcity.git.idle.timeout.seconds", it's default value is 600. This means that when there is no data from remote repository upto 10 minutes we throw a Timeout Exception. Could you please try to increase value of this property and check if it helps? To do that, create a file .BuildServer/config/internal.properties (if it doesn't exist), put teamcity.git.idle.timeout.seconds=1800 there and restart TeamCity.
Restarting now. I notice from the log that the fetch is timing out after 90 seconds which is obviously less than the teamcity.git.idle.timeout.seconds default of 600 seconds. I had already tried setting teamcity.git.fetch.timeout to a larger number but didn't notice that it had changed in 6.5.3!
Is there anywhere I can verify what value is being used for teamcity.git.idle.timeout.seconds by TeamCity?
Unfortunatelly we don't log used values, but seems like we should do that. I will add such logs
It's still timing out after 90 seconds :(
To check that it was using my settings I also set teamcity.git.fetch.separate.process=false and notice that the error stack still includes jetbrains.buildServer.buildTriggers.vcs.git.FetchCommandImpl.fetchInSeparateProcess(FetchCommandImpl.java:169) which suggests that I haven't created the file in the correct place. I will investigate...
Well, I fixed the problem but not in the most straightforward manner - I migrated the TeamCity server from Windows to Ubuntu (which we were doing anyway) and Git is now behaving itself very well, using the Default Private Key option.
Hi Dmitry,
I have some trouble - 90 seconds timeout on initial git fetch. I can't report version now (server unaviable, but more than 6.5.2, win2008). teamcity.git.idle.timeout.seconds setted in all places i found (internal.properties, tomcat teamcity config, agent config).
Logs looks like timeout set on entry task, not a git operation (can't quote log now).
Alexander, Graeme,
I found a bug. It turns out that separate fetch process was terminated even though it doesn't exceed idle timeout. I created an issue http://youtrack.jetbrains.net/issue/TW-17910 and will attach a build of plugin with fix there.
Fantastic! Good stuff, hope this saves people some trouble :)
Thanks!
Thanks Dmitry!
Can you describe new plugin version deploing process?
To deploy plugin, put the zip file attached to the issue into .BuildServer/plugins and restart the server.
Hi - I'm on 6.5.3 (on Windows) & have been having the same issues described above.
I've deleted the old plugin, added the new version you've linked to above and added the
teamcity.git.fetch.timeout = 600
teamcity.git.idle.timeout.seconds = 600
to the internal.properties file, but it's still no use, it is still happening - after around 90 seconds I get:
[17:28:57]: Unable to collect changes java.util.concurrent.ExecutionException: jetbrains.buildServer.vcs.VcsException: Problem collecting changes for 'SG :: SG' : 'git fetch' command failed. exception: Timeout exception at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source) at java.util.concurrent.FutureTask.get(Unknown Source)
...etc. It's strange because the other builds (using Git) in the server work fine. If I just change the fetch URL to one I know works, then all's fine (I can fetch this repo from the command line fine).
Any ideas on why this may be?
Hi Paul,
try to increase the value of teamcity.git.fetch.timeout property. As our doc says (http://confluence.jetbrains.net/display/TCD65/Git+%28JetBrains%29#Git%28JetBrains%29-InternalProperties) this property is the maximum time for fetch process to run. And fetch could be longer than 10 minutes.
If that doesn't help, please enable debug-vcs loggin preset at Administration > Server Configuration > Diagnostics and provide contents of a teamcity-vcs.log for the time when error occurs.
Hi Dmitry,
Thaks for the quick reply - I've changed the teamcity.git.fetch.timeout property, but this makes no difference. I've attached the log file (Looking at it, there seems to be an issue with "git-upload-pack", but everything is working fine if I change the Git repo in the TC project to a different one?).
Many Thanks,
Paul
Attachment(s):
teamcity-vcs.log.zip
Paul,
please check the version of the git-plugin at Administration > Server Configuration > Plugins, it should be SNAPSHOT-20110811022111. Maybe TeamCity didn't pick up the latest version.
Hi Dmitry - you were correct, the plugin was the old version (I had put the Zip file in "webapps\ROOT\plugins" and not "webapps\ROOT\WEB-INF\plugins").
It seems to work fine now!
Many Thanks,
Paul
I am experiencing this same issue on an osx install of version 6.5.3. I did get the patch version of the plugin to appear, but when I ran a build on the default agent I got "error whhile applying patch", after diggin in a bit more the rest of the error said to change auth to default key. After making that change and successfully testing I now get "the ref refs/heads/master" could not be resolved. Any help would be great.
Hi Dirk,
please enable 'debug-vcs' logging preset and provide a teamcity-vcs.log.
First run after start up (only changed logging mode)
[08:15:50]: Updating sources: agent side checkout...
[08:15:51]: [Updating sources: agent side checkout...] Failed to perform checkout on agent: TeamCity doesn't support authentication method Private Key with agent checkout. Please use 'Anonymous' or 'Default Private Key' methods.0
Second run
Unable to collect changes: jetbrains.buildServer.vcs.VcsException: Problem collecting changes for 'ROX iOS :: RMU Continuous Integration' : The ref 'refs/heads/master' could not be resolved
Attachment(s):
teamcity-vcs.log.zip
Dirk,
error "ref 'refs/heads/master' could not be resolved" can happen only when we cannot find such a branch in the repository. I guess your repository has such branch and for some reason our local clone doesn't. Could you please check that? You can find all git local clones at .BuildServer/system/caches/git. Please find directory for your repository and check if it have file refs/heads/master inside.
The caches/git dir was empty, so I cloned the repo there. I am now seeing pending items which is progress. I get the "error applying patch" error now with:
[16:55:50]:
Updating sources: agent side checkout...
[16:55:50]:
[Updating sources: agent side checkout...] Failed to perform checkout on agent: TeamCity doesn't support authentication method Private Key with agent checkout. Please use 'Anonymous' or 'Default Private Key' methods.
The vcs log looks like we are getting changes.
Also when I chagne to default private key I do not pass connection test.
At the moment, we don't support private keys in custom location or protected with passphrase on the agents. Can you change your key to be passphrase-less and put it the ./ssh with default name id_rsa/id_rsa.pub?
If you forced to use a passphrase-protected key, you can switch to server-side checkout.
Well I set VCS checkout mode to "Automatically on Server" with the VCS Root using Private Key. Teamcity is now working. Thanks for your help!