TeamCity - Build Queue size increased since upgrade to version 9.0

Hi Guys,

I have recently upgraded to TeamCity version 9 for the company I work for - they are currently using the Enterprise version.

Everything has been great so far except that the build queues have exploded from 200-300 build configurations to 900+ daily...

Is there something changed in the way build configurations are being merged/calculated in build queues compared to version 8?

Anyone else ran into this issue?

Thanks in advance!

Gavin

15 comments
Comment actions Permalink

Hi Gavin,

There were no changes that would increase the queue. Could you please investigate what causes such behavior? Perhaps more builds are triggered, or builds hang before start, or build is slower than before.

0
Comment actions Permalink

Hi Alina,

Thanks for the suggestions.

I will investigate and let you know what I find.

Cheers,
Gavin

0
Comment actions Permalink

Hi Alina,

I have managed to get the build queue to stabilize e.g. it will now decrease slowly rather than staying at 900+ all the time.

Originally we would have a VCS trigger on our build configurations that does the code build, then we would rely on "finished build" triggers down the chain to trigger other build configurations to run -

e.g. code build (VCS trigger) -> package build (code build finished triggers) -> database build (package build finished triggers) -> test runs (database build finished triggers)

We also have snapshot + artifact dependcies setup on all build configurations, with "Build from the same chain" set on the artifact dependencies.

We now rely on just the build chains from snapshot dependencies to work out which build configurations need to be run, by putting a VCS trigger on the build configurations that run the actual tests.

e.g. code build (no trigger) <- package build (no triggers, depends on code build) <- database build (no trigger, depends on code and package builds) <- test runs (VCS trigger, depends on code, package and database builds)

Note that we have artifacts that we need to use from all 3 dependent code, package and database builds, hence test runs have snapshot dependencies on them.

"Build from the same chain" is still set on the artifact dependencies in build configurations.

Now I am wondering if "Build from the same chain" is causing the slow down.

But removing "Build from the same chain" option in the test runs can cause issues (e.g. Artifacts from different svn revision of code builds are used with artifacts created from a different svn revision from package builds).

Do you have any other suggestions?

Cheers,
Gavin

0
Comment actions Permalink

Hi Gavin,

Sorry for delay in replying.
Your current setup with one VCS trigger is recommended one. It can look like this:

code (no trigger) <- package (no triggers, snapshot+artifact dependency on code) <- database (no trigger, snapshot+artifact dependency on package) <- test runs (VCS trigger, snapshot dependency on code build database, artifacts dependency on code/package/database)

Artifact dependencies on all build configurations with "Build from the same chain" option.

> Note that we have artifacts that we need to use from all 3 dependent code, package and database builds, hence test runs have snapshot dependencies on them.
Actually snapshot dependencies from test to all build configurations are not needed, one snapshot dependency to database is enough. In this case TeamCity will resolve dependencies correctly.

> Now I am wondering if "Build from the same chain" is causing the slow down.

"Build from the same chain" option should not cause slow down.

Please attach teamcity-server.log file and several server thread dumps.
0
Comment actions Permalink

Hi Alina,

Thanks for the details.

I have uploaded the thread dump from our TeamCity server (have also uploaded our usage stats)

Note that our build durations averages out at 15 minutes, times that by about 100 configurations x 3 branches.

Also, are there anyway to turn on "Build from same Chain" option on artifact dependencies for the test configurations without causing a huge increase in the build queue? Currently when I setup the test configurations to use this we would end up with multiple versions of the same test configurations on the queue, each for the different build chain. (e.g. if I check in 5 times in 1 hour, we would have A Test build configuration queued 5 times)

Note that I did not encounter this issue when using version 8 (not sure if it was due to some configuration / settings)

The reason why we need build from same chain on test configurations are because we also have VCS checkout setup in the tests, which are responsible for checking out thousands of test files from svn for test runs to use (It was not feasible to package test files in an earlier build configuration which we have tested initially, we had issues with file path length popping up constantly when packaging and unpacking to agents etc). Having build from same chain option on artifact dependencies allowed VCS checkouts on the test configurations to correctly checkout test files from SVN with the revision number that the artifacts were built from.

I am hoping there is some sort of option I may have missed that will optimize the build queue, so that if TeamCity sees that if there are multiples of the same queued test configuration, although it is for different build chain, it would merge and only run the latest build chain (e.g. similar to "Latest Successful Build" option but on the build configuration level - "Latest Successful Build Chain" etc)

Cheers,
Gavin



Attachment(s):
tc-usage-statistics-20150212-065004.properties.zip
2015-02-12_10.11.25-manual.txt.zip
TeamCityServerLog.zip
0
Comment actions Permalink

TeamCity automatically optimizes queue, see https://confluence.jetbrains.com/display/TCD9/Build+Queue#BuildQueue-BuildQueueOptimizationbyTeamCity.

There are few options that can help:

1. Snapshot dependency option “Do not run new build if there is a suitable one”

2. VCS trigger settings:

0
Comment actions Permalink

Hi Alina,

Thanks for the update.

We did have "Do not run new build if there is a suitable one" turned on when we had build on same chain turned on with snapshot dependencies set for the tests (back in version 8).

Also VCS trigger was made sure "Trigger a build on each check-in" is not selected as well.

Quiet period is default to 60 secs.

What I am interested in is this line here regarding build queue automatic optimization -

  • if an automatically triggered build chain has more changes than a build chain that is already queued, the latter will be replaced with the automatically triggered build chain, if such replacement will not not delay obtaining the build chain results


This seems to be the correct behaviour I have been seeing when our build configurations are being run on version 8, but since the upgrade to version 9 this rule does not seem to be kicking in for us anymore...

Is there something that have changed on the way TeamCity determines whether a build / replacement will not delay obtaining the build chain result?

Cheers,

Gavin

0
Comment actions Permalink

Hi Gavin,

There was the related issue https://youtrack.jetbrains.com/issue/TW-39717 fixed in TeamCity 9.0.2. Please try to upgrade the latest TeamCity version.
If the issue is repruduced after upgrade then please enable debug-all logging preset and send us zipped server logs folder.

0
Comment actions Permalink

Thanks Alina,

I will arrange to update the build configurations next week to test the build queue :)

Will let you know once I get the results.

Cheers,
Gavin

0
Comment actions Permalink

Hi Alina,

I have added the build chains / snapshot dependencies back into the test configurations.

Unfortunately looks like the issue I am currently facing still persists :p

I have attached the logs as per request.

I have also attached a screenshot of an example the same test configuration queued x 3 where I would have expected all 3 to have been merged.

I noticed alot of the builds have been removed from merge / replacement due to it being determined as causing a delay in finishing a build chain -

[2015-02-23 16:57:35,886]  DEBUG - serverSide.impl.BuildQueueImpl - Build bt2408 (promo id: 1118323) removed from the list of candidates to replace: bt2408 (promo id: 1117078), because such replacement would delay finishing of the chain.
[2015-02-23 16:57:35,886]  DEBUG - serverSide.impl.BuildQueueImpl - Build bt2401 (promo id: 1118330) removed from the list of candidates to replace: bt2401 (promo id: 1117085), because such replacement would delay finishing of the chain.
[2015-02-23 16:57:35,886]  DEBUG - serverSide.impl.BuildQueueImpl - Build bt2402 (promo id: 1118337) removed from the list of candidates to replace: bt2402 (promo id: 1117092), because such replacement would delay finishing of the chain.
[2015-02-23 16:57:35,886]  DEBUG - serverSide.impl.BuildQueueImpl - Build bt2403 (promo id: 1118344) removed from the list of candidates to replace: bt2403 (promo id: 1117099), because such replacement would delay finishing of the chain.

I wonder if there are any flags I can set to force it to use the latest build chain regardless?

Cheers,
Gavin



Attachment(s):
TeamCityLogs.zip
Build Chain Queued x 3.zip
0
Comment actions Permalink

Hi Alina,

I have left the build chain setup running overnight to collect further logs from TeamCity server (attached)

The build queue have jumped from 300-400 to 800 overnight for a single branch (the new build configurations was only updated for one branch, we had 2-3 other branches that uses the same set of configurations but did not have build chain set on the test configurations).

Hope the logs will help.

Thanks,
Gavin



Attachment(s):
TeamCityLogs_24022015.zip
0
Comment actions Permalink

Hi Alina,

Do you have an update for the setup?

Thanks,
Gavin

0
Comment actions Permalink

I think I'm seeing the same issue. We just upgraded from 8.0.3 to 9.0.2. Now every build chain is added separately to the queue. I've reviewed the documentation on "Suitable Builds", and think we're complying with all the conditions - in any case, it worked in 8.0.3.
Thanks
Brian

0
Comment actions Permalink

I managed to reproduce similar problem and created request in our tracker: https://youtrack.jetbrains.com/issue/TW-40270

0
Comment actions Permalink

Thanks Guys.

Will use the tracker instead for updates :)

Cheers,
Gavin

0

Please sign in to leave a comment.