Collecting git changes takes too long time
The build log excerpt:
[23:19:53][Collecting changes in 3 VCS roots] Processing combined checkout rule for '<redacted_repo>'
[23:24:39][Collecting changes in 3 VCS roots] VCS root '<redacted_repo>': git fetch git@github.com:<redacted_org>/<redacted_repo>.git
[23:45:24][Collecting changes in 3 VCS roots] VCS root '<redacted_repo>': Updating references 100%
[23:45:24][Collecting changes in 3 VCS roots] VCS root '<redacted_repo>': Updating references 100%
[23:45:24][Collecting changes in 3 VCS roots] VCS root '<redacted_repo>': update ref remote name: refs/heads/release-candidate-08-05-2019, local name: refs/heads/release-candidate-08-05-2019, old object id: 0000000000000000000000000000000000000000, new object id: 3f84f2d6c7e1316dc8ffe591c9037ef0b0f05297, result: NEW
[23:45:24][Collecting changes in 3 VCS roots] VCS root '<redacted_repo>': git fetch git@github.com:<redacted_org>/<redacted_repo>.git finished
[23:46:06][Collecting changes in 3 VCS roots] Done collecting changes for '<redacted_repo>': 1 changes collected 1 changes persisted, total time: 26m:12s,662ms, persisting time: 39ms
The agent log:
[2019-08-04 23:24:16,243] INFO - jetbrains.buildServer.AGENT - Updating agent parameters on the server: AgentDetails{Name='<redacted>',
AgentId=217, BuildId=null, AgentOwnAddress='<redacted>', AlternativeAddresses=[<redacted>, <redacted>, <redacted>], Port=9090, Version='42677', Plugins
Version='42677-md5-fa2f1e26eb5611e26d5ab757cdad0712', AvailableRunners=[Ant, cargo-deploy-runner, Duplicator, ftp-deploy-runner, gradle-runner, Inspection, Ipr, JPS, matrix-build-runner, Maven2, rake-runner, SBT, simpleRunner, smb-deploy-runner, ssh-deploy-runner, ssh-exec-runner], AvailableVcs=[tfs, cvs, jetbrains.git, mercurial, svn, perforce], AuthorizationToken='<redacted>', PingCode='ZdFiX2r08hfGRE1mXTBfxCjX72Z0QLR1'}
[2019-08-04 23:46:10,536] INFO - ldServer.AGENT.PollingProtocol - New command is received from server "runBuild" {id = 815}
During the slowness time there are some threadDums generated on the server with "java.lang.Thread.State: BLOCKED", although none of them mentions the problem build:
https://gist.github.com/maxnasonov/b0381b545b118a5f6692039b83bb590e
Please sign in to leave a comment.
We still have this issue. Sometimes collecting changes takes up to 1 hour.
What should we check to resolve it?
TeamCity Professional 10.0.5 (build 42677)
OS: Linux, version 4.4.0-1050-aws, amd64
Java: 1.8.0_101, Java HotSpot(TM) 64-Bit Server VM (25.101-b13, mixed mode), Java(TM) SE Runtime Environment (1.8.0_101-b13), Oracle Corporation
JVM args: -Djava.util.logging.config.file=/opt/TeamCity/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xmx4g -Xmx4g -XX:ReservedCodeCacheSize=350m -Dteamcity.configuration.path=../conf/teamcity-startup.properties -Dlog4j.configuration=file:/opt/TeamCity/bin/../conf/teamcity-server-log4j.xml -Dteamcity_logs=../logs/ -Djava.awt.headless=true -Djava.endorsed.dirs=/opt/TeamCity/endorsed -Dcatalina.base=/opt/TeamCity -Dcatalina.home=/opt/TeamCity -Djava.io.tmpdir=/opt/TeamCity/temp
Database type: MySQL
Hi Maxim,
the collecting changes step at that point takes place on the server, so the agent logs won't really provide any useful information. This step connects to the repository, performs a fetch to pull the information about new revisions and updates them in the database and the local cache. There is a wide array of possible reasons for this process to be slow. From a very large repository/set of changes, to slow connection to the server, to not enough cpu cores assigned to the VM, to having configured the server with many branches, to the local cache being very large, slow database or slow I/O.
There is very little configuration that can help here, as the most critical elements are external to teamcity, but there is always the possibility that the application is doing something in a non-optimal manner for your scenario. This said, it won't really help that you are using teamcity 10.0.5, as the version is a few years old. We'd recommend upgrading as many improvements have been brought in since.
Thank you for the input, Denis. The thread dump is copied from the server logs but I got your point.
Hi Maxim,
yes sorry, my comment was about your initial posting of the agent logs, was just to clarify the place where it would be best to look for the answers.
I also missed to say that the teamcity-vcs.log, in particular if you enable debug-vcs under Administration - Diagnostics, there might be some more extra information that might give a hint at what is taking the longest.