It takes long time to checkout submodules

After resolved this issue http://devnet.jetbrains.com/message/5502730#5502730,
It takes several hours to checkout submodule for the first build with github VCS Root and it does not finished.
Here is the build log

[18:55:37]Checking for changes
[18:55:39]Publishing internal artifacts
[18:55:39][Publishing internal artifacts] Sending build.start.properties.gz file
[18:55:39]Clearing temporary directory: C:\TeamCity\buildAgent\temp\buildTmp
[18:55:39]Clean build enabled: removing old files from C:\TeamCity\buildAgent\work\d5bd6e88e6d8f4c6
[18:55:39]Checkout directory: C:\TeamCity\buildAgent\work\d5bd6e88e6d8f4c6
[18:55:39]Updating sources: agent side checkout (running for 6m:33s)
[18:55:39][Updating sources] Will perform clean checkout. Reason: Checkout directory is empty or doesn't exist
[18:55:39][Updating sources] Cleaning C:\TeamCity\buildAgent\work\d5bd6e88e6d8f4c6
[18:55:39][Updating sources] VCS Root: https://github.com/myteam/myrepo.git (running for 6m:33s)
[18:55:39][VCS Root: https://github.com/myteam/myrepo.git] revision: 9ce7c339672bb2a7b113f0513231e9d7cb198c98
[18:55:39][VCS Root: https://github.com/myteam/myrepo.git] Cleaning C:\TeamCity\buildAgent\work\d5bd6e88e6d8f4c6
[18:55:39][VCS Root: https://github.com/myteam/myrepo.git] The .git directory is missing in 'C:\TeamCity\buildAgent\work\d5bd6e88e6d8f4c6'. Running 'git init'...
[18:55:40][VCS Root: https://github.com/myteam/myrepo.git] Commit '9ce7c339672bb2a7b113f0513231e9d7cb198c98' is not found in repository. Running 'git fetch'...
[18:56:00][VCS Root: https://github.com/myteam/myrepo.git] Checking out branch refs/heads/branch6_image_server_iis in https://github.com/myteam/myrepo.git in C:\TeamCity\buildAgent\work\d5bd6e88e6d8f4c6 with revision 9ce7c339672bb2a7b113f0513231e9d7cb198c98
[18:56:15][VCS Root: https://github.com/myteam/myrepo.git] Cleaning https://github.com/myteam/myrepo.git in C:\TeamCity\buildAgent\work\d5bd6e88e6d8f4c6 the file set ALL_UNTRACKED
[18:56:15][VCS Root: https://github.com/myteam/myrepo.git] Checkout submodules in C:\TeamCity\buildAgent\work\d5bd6e88e6d8f4c6

I don't have any idea to investigate the manner.
Please let us know if threr is any information.

13 comments
Comment actions Permalink

Hi Takayoshi,

What TeamCity version do you use?

There is a related issue fixed in 8.0.1: http://youtrack.jetbrains.com/issue/TW-26924

If still actual in the latest TeamCity version:

> It takes several hours to checkout submodule for the first build with github VCS Root and it does not finished.

So does it finally finish or not?

How long does it take if you use native git client on the same machine and under the same user to clone to an empty directory?

What authentication is used for the subrepositories? (as there is some issues with that: http://youtrack.jetbrains.com/issue/TW-24809 )

0
Comment actions Permalink

Thanks Yegor,

TeamCity version is 8.0.4.

> > It takes several hours to checkout submodule for the first build with github VCS Root and it does not finished.
> So does it finally finish or not?

It doesn't finish. I tried continue building, it fiinally failed by timed out after a day.


> How long does it take if you use native git client on the same machine and under the same user to clone to an empty directory?

It takes less than one minute.

> What authentication is used for the subrepositories? (as there is some issues with that: http://youtrack.jetbrains.com/issue/TW-24809 )

We use 4 subrepositories. 3 of them are same authentication (github private repository) and the other one is no authentication (public repository).
0
Comment actions Permalink

Could you please attach a full build log and teamcity-vcs.log from the agent to the thread or send them to teamcity-feedback@jetbrains.com.

0
Comment actions Permalink

Hello,

I'm having the same problem. I'm able to pull everything using native git.exe in 1 minute.

[20:57:58]Checking for changes
[20:58:00]Publishing internal artifacts
[20:58:00][Publishing internal artifacts] Sending build.start.properties.gz file
[20:58:00]Clearing temporary directory: D:\Program Files\TeamCity\buildAgent\temp\buildTmp
[20:58:00]Checkout directory: D:\Program Files\TeamCity\buildAgent\work\2dee800d0d6d36a4
[20:58:00]Updating sources: agent side checkout (30m:18s)
[20:58:00][Updating sources] Will perform clean checkout. Reason: Agent doesn't have any version of the project sources
[20:58:00][Updating sources] Cleaning D:\Program Files\TeamCity\buildAgent\work\2dee800d0d6d36a4
[20:58:00][Updating sources] Failed to delete empty directory: D:\Program Files\TeamCity\buildAgent\work\2dee800d0d6d36a4
[20:58:00][Updating sources] VCS Root: SDK @ BitBucket (30m:18s)
[20:58:00][VCS Root: SDK @ BitBucket] revision: 4c12c9203feda6dad621c1b5771a6e9a471c9d6c
[20:58:00][VCS Root: SDK @ BitBucket] Cleaning D:\Program Files\TeamCity\buildAgent\work\2dee800d0d6d36a4
[20:58:00][VCS Root: SDK @ BitBucket] Failed to delete empty directory: D:\Program Files\TeamCity\buildAgent\work\2dee800d0d6d36a4
[20:58:00][VCS Root: SDK @ BitBucket] The .git directory is missing in 'D:\Program Files\TeamCity\buildAgent\work\2dee800d0d6d36a4'. Running 'git init'...
[20:58:00][VCS Root: SDK @ BitBucket] Commit '4c12c9203feda6dad621c1b5771a6e9a471c9d6c' is not found in repository. Running 'git fetch'...
[21:28:18][VCS Root: SDK @ BitBucket] No output from git during 1800 seconds. Try increasing idle timeout by setting parameter 'teamcity.git.idle.timeout.seconds' either in build or in agent configuration.
[21:28:18][Updating sources] Failed to perform checkout on agent: Timeout exception
[21:28:18]Publishing internal artifacts
[21:28:18][Publishing internal artifacts] Sending build.finish.properties.gz file
[21:28:18]Build failed to start. Artifacts will not be published for this build
[21:28:18]Build finished

0
Comment actions Permalink

Hi Arik,

> I'm able to pull everything using native git.exe in 1 minute.
If that is a clean featch of the entire repository ru ont he same machine under the same user, then could you please attach corresponding teamcity-vcs.log from the TeamCity agent?

In your case the git invoked seems to produce no output for 30 minutes:
> [21:28:18][VCS Root: SDK @ BitBucket] No output from git during 1800 seconds. Try increasing idle timeout by setting parameter 'teamcity.git.idle.timeout.seconds' either in build or in agent configuration.
> [21:28:18][Updating sources] Failed to perform checkout on agent: Timeout exception

Adding the parameter to the build confiuration will make TeamCity wait longer.

0
Comment actions Permalink

Arik, please check if the problem still happens in TeamCity 8.0.5.

0
Comment actions Permalink

I am still seeing the same problem on Teamcity 8.0.5.

[15:28:29]Checking for changes
[15:28:29]Publishing internal artifacts
[15:28:29]Clearing temporary directory: /Users/aquto/Teamcity/temp/buildTmp
[15:28:29]Checkout directory: /Users/aquto/Teamcity/work/3a54df07b17e6a2
[15:28:29]Updating sources: agent side checkout (running for 13m:28s)
[15:28:29][Updating sources] Will perform clean checkout. Reason: "Clean all files before build" turned on
[15:28:29][Updating sources] Cleaning /Users/aquto/Teamcity/work/3a54df07b17e6a2
[15:28:29][Updating sources] VCS Root: client-build (master) (running for 13m:28s)
[15:28:29][VCS Root: client-build (master)] revision: 8bf531e0b59e5af8fbbaf3d594ecf1698c0ba78b
[15:28:29][VCS Root: client-build (master)] Cleaning /Users/aquto/Teamcity/work/3a54df07b17e6a2
[15:28:29][VCS Root: client-build (master)] The .git directory is missing in '/Users/aquto/Teamcity/work/3a54df07b17e6a2'. Running 'git init'...
[15:28:29][VCS Root: client-build (master)] Commit '8bf531e0b59e5af8fbbaf3d594ecf1698c0ba78b' is not found in repository. Running 'git fetch'...


and it just hangs.

This same git repo is checked out properly on other build agents. This particular build agent is on Max OS X Mavericks (others that have worked are on CentOS 6.4).

Scott.
0
Comment actions Permalink

I was able to work around this issue by stopping the agent, manually removing the entire contents of the Teamcity/work/ directory, and starting the agent.

0
Comment actions Permalink

We have recently found that the way we pass a username for submodules doesn't work starting from git 1.7.11. If you still have this problem please install the latest build of git plugin from http://teamcity.jetbrains.com/viewType.html?buildTypeId=TeamCityPluginsByJetBrains_JetBrainsGitPluginTeamCity80x&tab=buildTypeStatusDiv&branch_TeamCityPluginsByJetBrains_Git=__all_branches__. To install it put the zip from artifacts into <TEAMCITY_DATA_DIR>/plugins and restart the server.

0
Comment actions Permalink

Thnaks for your update!
But the latest version (#snapshot-54) in http://teamcity.jetbrains.com/viewType.html?buildTypeId=TeamCityPluginsByJetBrains_JetBrainsGitPluginTeamCity80x&tab=buildTypeStatusDiv&branch_TeamCityPluginsByJetBrains_Git=__all_branches__,
it does not work for submodule of the submodule respotiory.

> [Updating sources] Failed to perform checkout on agent: Error while reading C:\TeamCity\buildAgent\work\3ce7df0f5f7cd386\MySubmodule\.gitmodules

MySubmodule is the submodule of top level repository. And my submodule has submodules and .gitmodules is written about it.
Is this a known issue?

Thanks.

0
Comment actions Permalink

Please attach a full build log and teamcity-vcs.log from the agent.

0
Comment actions Permalink

Sorry for not responding long time, but finally this problem is resolved at the latest TeamCity v8.1 released February 12, 2014.
Thanks for a lot of your comments and cooperation.

0
Comment actions Permalink

Great! Thanks for update! Please create an issue in the tracker if you encounter any other problems.

0

Please sign in to leave a comment.