Investigating frequent SVN agent clean checkout in server-side checkout mode

OK, I found proper documentation for 2017.1.5 https://confluence.jetbrains.com/display/TCD10/Reporting+Issues#ReportingIssues-VersionControldebugloggingvcs :)

I wonder what is the criterion to trigger cleanPatch=true for server side checkout?

According to https://teamcity-support.jetbrains.com/hc/en-us/community/posts/206256019/comments/206591045 it happens at least once per day

The problem is that it happens too often for us, almost on every 2nd CI build.

As a result our Incredibuild farm takes 40-50 minutes to rebuild everything. Not only it's too long but also takes Incredibuild resources from others.

To guarantee that the folder is used just by 1 configuration on the agent, not by all configurations with same VCS hash, I used custom checkout dir.

Still it requests clean patch. And it's 17 GB for 3 roots.

Hi Vyachelav,

sorry for the inconvenience. We have the description on why clean checkouts are produced in our docs below, could you check whether you can trace some of those to your scenario?

https://confluence.jetbrains.com/display/TCD10/Clean+Checkout

Unfortunately nothing applies.

Using "recommended" Automatic checkout directory results in several configurations using same folder (same hash) on the agent.

Using same folder for 2 configurations leads to frequent clean checkouts for all 3 build roots.

After I explicitly designate different folders, the problem goes away (15 builds in a row without clean checkout).

According to TeamCity documentation this should not happen.

During compilation some files under source control can be touched (time changed). Can it so happen that if 2 or more configurations share same checkout folder, TeamCIty watches integrity of working copy? And if folder is used exclusively by just one configuration - not?

Ok. Unfortunately, it does not help any more.

I have 3 builds in a row with clean checkout every time :(

I reckon last time what actually helped is cleaning SVN cache and SVN External Cache, but not for long.

Build '* MAGNET * :: MAGNET Field 5.0 Continuous Integration :: Win32 - VS2017 Release' #build 1070 - r151871
Started 2017-10-28 17:35:06 on 'mos-tc-agent-01' by 'Vyacheslav Lanovets (vlanovets)'
Is running with status NORMAL 'Transferring repository sources: 4.84 GB so far...'
VCS revisions: 'magnet_root' (Subversion): 151871, 'stlport' (Subversion): 8775, 'boost' (Subversion): 8880
TeamCity URL https://*********/viewLog.html?buildId=596645&buildTypeId=MF_CI_Win32_Release
TeamCity server version is 2017.1.5 (build 47175), server timezone: MSK (UTC+03:00)

[17:34:27] : bt1191 (running for 5m:16s)
[17:34:27]i: TeamCity server version is 2017.1.5 (build 47175)
[17:34:28] : The build is removed from the queue to be prepared for the start
[17:34:28] : Collecting changes in 3 VCS roots (37s)
[17:34:28] : [Collecting changes in 3 VCS roots] VCS Root details
[17:34:28] : [VCS Root details] "magnet_root" {instance id=593, parent internal id=1, parent id=magnet_root, description: "svn: svn+ssh://*********/MAGNET"}
[17:34:28] : [VCS Root details] "boost" {instance id=728, parent internal id=8, parent id=boost_root, description: "svn: svn+ssh://*********/ExtLibs/boost"}
[17:34:28] : [VCS Root details] "stlport" {instance id=650, parent internal id=20, parent id=StlportRoot, description: "svn: svn+ssh://*********/ExtLibs/stlport"}
[17:34:28]i: [Collecting changes in 3 VCS roots] Detecting changes in VCS root 'magnet_root' (used in '2T WinCE 6.0 VS2008 (ARMV4I) - Release', '2T WinCE 6.0 VS2008 (ARMV4I) - Release (CMake)' and 112 other configurations)
[17:34:28]i: [Collecting changes in 3 VCS roots] Will collect changes for 'magnet_root' starting from revision 151871_2017/10/28 17:33:43 +0300
[17:34:28]i: [Collecting changes in 3 VCS roots] Detecting changes in VCS root 'stlport' (used in '2T WinCE 6.0 VS2008 (ARMV4I) - Release', '2T WinCE 6.0 VS2008 (ARMV4I) - Release (CMake)' and 101 other configurations)
[17:34:28]i: [Collecting changes in 3 VCS roots] Detecting changes in VCS root 'boost' (used in '2T WinCE 6.0 VS2008 (ARMV4I) - Release', '2T WinCE 6.0 VS2008 (ARMV4I) - Release (CMake)' and 174 other configurations)
[17:34:28]i: [Collecting changes in 3 VCS roots] Will collect changes for 'boost' starting from revision 8892_2017/10/28 17:27:58 +0300
[17:34:28]i: [Collecting changes in 3 VCS roots] Will collect changes for 'stlport' starting from revision 8892_2017/10/28 17:26:32 +0300
[17:34:28]i: [Collecting changes in 3 VCS roots] VCS revisions for 'boost' - 8892_2017/10/28 17:27:58 +0300..8892_2017/10/28 17:34:28 +0300
[17:34:28]i: [Collecting changes in 3 VCS roots] Processing combined checkout rule for 'boost' with 3 include rules
[17:34:28]i: [Collecting changes in 3 VCS roots] VCS revisions for 'stlport' - 8892_2017/10/28 17:26:32 +0300..8892_2017/10/28 17:34:28 +0300
[17:34:28]i: [Collecting changes in 3 VCS roots] Processing combined checkout rule for 'stlport' with 2 include rules
[17:34:28]i: [Collecting changes in 3 VCS roots] VCS revisions for 'magnet_root' - 151871_2017/10/28 17:33:43 +0300..151871_2017/10/28 17:34:28 +0300
[17:34:28]i: [Collecting changes in 3 VCS roots] Processing combined checkout rule for 'magnet_root' with 12 include rules
[17:34:29]i: [Collecting changes in 3 VCS roots] Done collecting changes for 'stlport': 0 changes collected 0 changes persisted, total time: 313ms, persisting time: < 1ms
[17:34:29]i: [Collecting changes in 3 VCS roots] Done collecting changes for 'boost': 0 changes collected 0 changes persisted, total time: 532ms, persisting time: < 1ms
[17:35:05]i: [Collecting changes in 3 VCS roots] Done collecting changes for 'magnet_root': 0 changes collected 0 changes persisted, total time: 37s,264ms, persisting time: < 1ms
[17:35:06] : [Collecting changes in 3 VCS roots] Compute revision for 'boost'
[17:35:06] : [Compute revision for 'boost'] Upper limit revision: 8892_2017/10/28 17:34:28 +0300
[17:35:06]i: [Compute revision for 'boost'] MaxModId = 156912
[17:35:06] : [Compute revision for 'boost'] Latest commit attached to build configuration: 8880_2017/10/12 14:33:28 +0300
[17:35:06] : [Compute revision for 'boost'] Computed revision: 8880_2017/10/12 14:33:28 +0300
[17:35:06] : [Collecting changes in 3 VCS roots] Compute revision for 'magnet_root'
[17:35:06] : [Compute revision for 'magnet_root'] Upper limit revision: 151871_2017/10/28 17:34:28 +0300
[17:35:06]i: [Compute revision for 'magnet_root'] MaxModId = 156912
[17:35:06] : [Compute revision for 'magnet_root'] Latest commit attached to build configuration: 151871_2017/10/28 14:18:59 +0300
[17:35:06] : [Compute revision for 'magnet_root'] Computed revision: 151871_2017/10/28 14:18:59 +0300
[17:35:06] : [Collecting changes in 3 VCS roots] Compute revision for 'stlport'
[17:35:06] : [Compute revision for 'stlport'] Upper limit revision: 8892_2017/10/28 17:34:28 +0300
[17:35:06]i: [Compute revision for 'stlport'] MaxModId = 156912
[17:35:06] : [Compute revision for 'stlport'] No modification from VCS root is attached to build configuration, use first revision of changes collecting 8775_2017/06/06 17:10:03 +0300
[17:35:06] : [Compute revision for 'stlport'] Computed revision: 8775_2017/06/06 17:10:03 +0300
[17:35:06] : Starting the build on the agent mos-tc-agent-01
[17:35:06]i: Agent time zone: Europe/Moscow
[17:35:05]i: Agent is running under JRE: 1.8.0_121-b13
[17:35:06]i: Performance monitor is using command line: [cscript, C:\BuildAgent\system\perfmon\scripts/perfmon.vbs, C:\BuildAgent\system\perfmon\temp\perfmon.csv, 1000]
[17:35:06]i: Starting performance monitoring process
[17:35:09]i: Performance monitoring process started
[17:35:09] : Clearing temporary directory: C:\temp\BuildAgent\buildTmp
[17:35:09] : Publishing internal artifacts
[17:35:09] : [Publishing internal artifacts] Publishing 1 file using [WebPublisher]
[17:35:09] : [Publishing internal artifacts] Publishing 1 file using [ArtifactsCachePublisher]
[17:35:09] : Using vcs information from agent file: MF-CI-WIN32-vs17-Release.xml
[17:35:09] : Transferring cached clean patch for VCS root: magnet_root
[17:35:09] : Checkout directory: C:\X\MF-CI-WIN32-vs17-Release
[17:35:09] : Updating sources: server side checkout (running for 4m:34s)
[17:37:09] : [Updating sources] Transferring repository sources: 2.47 GB so far...
[17:37:28] : [Updating sources] Building incremental patch over the cached patch
[17:37:38] : [Updating sources] Transferring cached clean patch for VCS root: boost
[17:39:09] : [Updating sources] Transferring repository sources: 4.84 GB so far...
Current time: 17:39:44

Final conclusion.

It was same folder after all.

Hence *It is better to not checkout to 1 folder 2 different configurations, no matter what. And better avoid automatic checkout folder.*

system\checkoutdir-revisions on the teamCity agent is tracking build history correctly.

buildTypes folder on TeamCity server machine has same settings for 2 conflicting configurations (compared with diff tool).

Files under version control are not modified during the build.

Still 2 configurations fight each other.

But If I copy one of the configurations, it is not fighting the original any more. As I don't understand what was wrong with orignal configuration, I would not experiment. I configuration = 1 checkout directory.

Unrelated: When I look into the configuration XML in folder buildTypes, I see that "order" attribute of build-runners tag is not good, as it mentions build runners not present in the XML. After Reordering build steps back and forth, XML is repaired.

OK, from now on, our IT will be buying 2TB NMVe SSDs instead of 1.2TB for the agents. This should compensate extra copies of source code on the agent machines. $$$ :(

 Hello Vyacheslav,

  It is hardly possible that the automatic checkout directory would be reused for unrelated build configurations which do not share VCSRoot + checkout rules combination. Simply because the name of the checkout directory is built using a hash of important VCS Root parameters, and the content of checkout rules.

  There could be some bug in what TeamCity understands as "important VCS parameters". For SVN this list includes URL, username, externals mode, working copy format (for checkout on agent). VCS Root id, name, type, checkout rules mapping are participating in the hash as well. I.e. only if all those parameters match, TeamCity should reuse checkout directory between builds.

  If TeamCity reuses checkout directory but runs clean checkout often, it should indicate the reason of clean checkout in the build log. If you're unsure what the reason is, please download and provide the full build log of the builds where the problem occurred, as well as debug teamcity-vcs.log and teamcity-server.log from TeamCity for the corresponding timeframe. You may want to create a YouTrack issue for that: https://youtrack.jetbrains.com/issues/TW.

  Thank you,