"Skip checking for changes - changes are already collected" long running

Hello Anthony,

   The matter is, between messages "Skip checking for changes" and "Updating sources" TeamCity shouldn't perform any lengthy VCS operations.
   To understand, what makes your builds slow, please enable Debug logging in administration interface, and also please make a couple of thread dumps from TeamCity server UI.

   Please send us the logs and the thread dumps, we'll try to figure out what's going on.

   Regards,
   KIR

This is what came from the Admin interface to download the logs as a zip.

I took several thread dumps while builds exhibiting the issue were occuring.

Not sure if this is helpful, but we recently switched TeamCity to listen on Port 80 instead of port 8080. Also, during troubleshooting, we moved from one templetated VCS root for the project with exclusions and server side checkout to one VCS root per build configuration with no exclusions and Agent Side Checkout. Before we did this, TeamCity would seem to just hang with nothing being processed out of the queue and all build agents idle. Since this change, The build agents have been reporting "Running..." for long periods of time (ie. the issue described above).

Anthony, could you please describe dependencies of this build, both snapshot and artifact?

After the further analyzing of the thread dumps, it seems that upgrade to 6.0 would help here, at least it should be faster in 6.0. If you do not plan to upgrade, please submit issue to our tracker, we will try to provide you with a fix.

We are planning on upgrading to V6 early next week. I did a test install that has a subset of our builds and it seems to resolve dependencies and start building faster. I'll let you know

Thanks for your help!

Pavel,

I upgraded TeamCity to V6 today. I'm not seeing the "Skip Changes" issue. However, my build queue now seems to be stuck. We use a lot of snapshot dependencies to manage our build chains. I created a "Master Build" that has a snapshot dependency on all the other builds. The master build is a simple script ("echo success") that has the only trigger in the build chain - a VCS trigger that checks VCS changes on snapshot dependencies. I queued up a "Master Build" and 60 minutes later, nothing has been removed (or added) to the queue and no builds have occurred. The load on the server has been staying at a consistent 33% processor usage.

If I manually kick off smaller build (lower in the dependency chain) manually, they seem to build relatively quickly. But when the complete build is queued up, it just seems hang. We divide up our build agents and assign them exclusively to certain projects. When a build that has no snapshot dependencies and a available build agent is queued, it won't build while this master build is queued.

The only way I can remove the hanging snapshot dependency builds is to archive the project they are in and then unarchive it. Even after that, the other non-snapshot dependent builds don't budge from the queue. After a restart of the TeamCity Web server, those builds go through.

After upgrading to TC6, I switched the JRE and tomcat to 64 bit.

Thanks for you help!

We just upgraded to 6.0 today and we are seeing similar issues.  Our build queue gets filled up and take a long time to empty even though the build agents are sitting there idle.  We upgraded from a 6.0 EAP and didn't see those issues until today's upgrade.

When we do get builds to run, they sit there in the "collecting changes" step for a long time.  Most of our builds are experiencing this problem while a few others just seem to run normally.

-jasonm

Hello,

Could you please describe your dependency graph somehow? Or maybe you could send us your config files (.BuildServer/config)? You can use teamcity-feedback[at]jetbrains.com email address.

Jason,

I do not think your problem is related, because in your case builds are starting but spend a lot of time in checking for chages stage, right? What version control do you use?

Pavel,

We are seeing both problems.  We first see our jobs sit in the queue for a long time for no reason.  After the jobs start we are seeing huge times on the checking for changes steps.  we use SVN.

Pavel,

We have registered this bug within your YouTRACK system:

http://youtrack.jetbrains.net/issue/TW-14907?projectKey=TW

Would it be possible to have this reviewed and updated accordingly?

Thank you,

Shawn.

We are still experiencing issues with slow dependency resolution. I usally have to archive our main build project several times a day to get the queue moving again. Is there any performance improvement in v6.01?


Here is an example of the kind of dependencies we have. In this scenario, we only have VCS triggers (with "Trigger on changes in snapshot dependencies" enabled) on G, I, K, L. Build C has a lot of dependencies. If a checkin is made for build C, a large of amount of builds are added to the queue due to snapshot dependencies.

Anthony, sorry for delay and thank you for dependency graph. It's quite complex. So the problem is still the same, right? When the whole chain is added to the queue, server starts eating the CPU and nothing starts, correct?

The dependency graph I posted is actaully much less complex than what we really have, I just don't have an easy tool to get it out of TeamCity and we don't have it documented in our shop. We are working to consolidate dependency to make it less complex, but that won't be happening too soon.

When a high level code checkin is made, say on build configuration I, the chain is quite simple and the build generally happens quickly. But if a code checkin is made for build configuration C, the amount of builds queued for C and the entire build chain explodes. I believe in the example, 5 builds would be queued for C alone. In our situation, this number is over 10. In our setup, if a code change is made to a low level dependency, we can easily get 250 builds added to the queue from our build project which contains about 55 configurations. When this happens, TeamCity grinds to a halt. Even build configurations that have dedicated build agents get stuck in the queue. It can sometimes be 10 - 20 minutes between builds occurring on an agent.

In this case, we usually archive the build project to clean the queue since manually deleting items from the queue does not work. We then do a build of the low level dependency manually. When that completes, we kick off the high level builds. While still slow, it doesn't seem to stall and results in less overall builds being queued. I my experience, once our queue goes higher that 200, performance decreases drastically.

CPU usage for the Teamcity webserver process usually hovers around 10 - 15% with 3 - 5GB memory usage while this is happening. Just before builds actually start occurring on agents, the usage can spike to 100% for short periods. There are no build agents installed on the machine that hosts the web server and the machine is dedicated to the TeamCity webserver.

As I understand you upgraded to TeamCity 6.0 version. Could you please provide me with fresh thread dumps of your server for the case when server eats too much CPU? Ideally, there should be several thread dumps taken with some interval. These thread dumps can be saved to the build log directory right on the Diagnostics page.

I made a code change in a low level dependency. 8 builds of that configuration were queued up. In total 179 builds were queued. After 10 minutes or so, 163 builds were still queued and no builds had actually been built. I got tired of waiting and archived the build project and restarted TeamCity web server and built the build configuration manual. The build I made a code checkin to was 'Panda Systems | Core References' (bt343).

Attached are server logs with thread dumps.

Anthony, thank you for logs. Could you please install this plugin: http://confluence.jetbrains.net/display/TW/Server+Profiling
then take CPU snapshot for the time while the server is slow and send it to us (you can use: teamcity-feedback[at]jetbrains.com)

Pavel,

We have a software deployment coming up, so I will not be able to post the results until next week.

Thank you for your help!

I understand. I will try to further investigate it based on thread dumps. But I would really appreciate if you could take a CPU snapshot at some point. This would help me ensure that we are on the right track if any optimizations will be done.

I have some good news. We were able to speedup code working with large build chains. Without the CPU snapshot I still can't say for sure whether such speedup will help you or not, but I'll appreciate if you install TeamCity 6.0.2 once it will be released (the fix will be included in 6.0.2) and give us your feedback.

I uploaded pre-6.0.2 build with speedup in build chain to our ftp server:
ftp://ftp.intellij.net/pub/.teamcity/15837/

Pavel,

I thought I should reply about our success with your changes since Tony is off on vacation.
We were able to deploy the 6.0.2 update on Tuesday afternoon to our production systems, and since then everything has been running much faster.

We haven't had to manually clear the build queue via archiving since that update. We still have (expected) surges in the number of builds added to the queue, but the processing of these happens much more quickly, and in almost no time things are building. I know there are still some slowdowns that seem unnecessary, but I'll let Tony follow up with more on that when he returns.

Thanks for those changes and your responsiveness to our issues!

6.02 has dramatically improved performance.

Thank you for all your help!

Regards,
Tony