Cache source code with worker or 3-d party server

Can I cache source code with worker or 3-d party server instead of caching it on master?
I want to put I/O load from master (single point of load) to other servers. All my agents are using server-side checkout, so when I add an agent the I/O load on TC master increases proportionally

15 comments
Comment actions Permalink

Hi Bulat,

As for now you can map the .BuildServer/system/caches directory to the desired location using OS-specific means (filesystem directory links / symlinks). Please read the realeted blog post.
Also please watch/vote for the related feature request.

0
Comment actions Permalink

Great! I've voted.

Also I mean you already have a vcs-worker server. Let it cahce sources, not master. I think it should be another feature request, don't you?
My I/O problem is network, not disks, since we're using raid10, he-he.



Attachment(s):
vcs-worker as chache server.png
0
Comment actions Permalink

Yes, you are right, it is not the same. Please feel free to create new feature request. However won't it be an option to use agent-side checkout?

0
Comment actions Permalink

I've tried to use agent-side checkout, but it forces to download full sources every time I run build. Also, I still want to have caching server which stores already processed with checkout rules sources to make for end-side agent downloading it as just bunch of files. No need mappings/workspaces for it. Additionally I'll be able to make my TFS happy not loading it so much with get sources requests/

0
Comment actions Permalink

Cyberbolt schrieb:

I've tried to use agent-side checkout, but it forces to download full sources every time I run build.


I believe that this is due to your configuration. We also use agent-side checkout and have configured it so that only an update is necessary.

0
Comment actions Permalink

That's great. Do you mean I have 'Clean sources on each build' checkbox checked or you did some deeper config tuning to avoid sources full uploading?
I use TFS as version control system.

0
Comment actions Permalink

Hi Bulat,

Please check that none of Automatic Clean Checkout cases are met. Also please check that advanced TFS VCS root option "Enforce overwrite all files" is set to false and no internal properties related to TFS are configured.

0
Comment actions Permalink

Hi Alina,
I've checked, none of cases are met.



I've run one configuration more than 6 times with different settings.
I tried custom checkout directory. The agent was disabled and I run the same config again and again. Every time I didn't expected clean checkout - I got it.
I tried with automaticaly named chekout dir. I run that config couple of times and got different dirs. The third time the build was started in the same dir as previous, but I still got clean checkout.

Please take a look into agent teamcity and build logs attached, I don't know how to make agent not to perform clean checkout each time. Probably it happens because of checkout rules, but it shouldnt.



Attachment(s):
teamcity-build.logs.zip
teamcity-agent.zip
0
Comment actions Permalink

Do you have numbers that your config is working? E.g. each time I've run the same build (even no changes commited) I got ~25 minutes for total build (19-27min) and 10-19 minutes for sources downloading, all the time - clean checkout.

0
Comment actions Permalink

We use Subversion, so it certainly is not 1:1 to compare.

clean/full checkout

Updating sources: agent side checkout (7m:57s)
[14:25:07]Will perform clean checkout. Reason: Checkout directory is empty or doesn't exist
[14:25:07]Cleaning D:\BuildAgent1_1\work\XXX
[14:25:08]Publishing to local artifacts cache is disabled, skipping.
[14:25:09]Using vcs information from server. Reason: no revision information for buildtype build and checkout directory D:\BuildAgent1_1\work\XXX on agent
[14:25:09]VCS Root: VCS_root_name (7m:55s)
[14:25:09]checkout rules: build/teamcity=>; =>source; revision: 37503_2015/01/30 14:23:06 +0100
[14:25:09]Using working copy format 1.7
[14:33:04]Using working copy format 1.7


update (with revert, without it would be faster)

Updating sources: agent side checkout (26s)
[12:13:29]Using vcs information from agent file: XXX.xml
[12:13:29]Publishing to local artifacts cache is disabled, skipping.
[12:13:30]VCS Root: VCS_root_name (25s)
[12:13:30]checkout rules: build/teamcity=>; =>source; revision: 37496_2015/01/29 11:37:29 +0100
[12:13:30]Using working copy format 1.7
[12:13:30]Fast SVN update is enabled
[12:13:30]Reverting local changes for working directory (14s)
[12:13:30]svn revert in D:\BuildAgent1_1\work\XXX\source
[12:13:54]Using working copy format 1.7
[12:13:54]Fast SVN update is enabled
[12:13:54]Reverting local changes for working directory
[12:13:54]svn revert in D:\BuildAgent1_1\work\XXX



Can you copy the part of the build logs with the checkout here?

0
Comment actions Permalink

We use TFS:

Clean checkout (agnent-side custom dir):

[15:12:16]Updating sources: agent side checkout (10m:47s)
[15:12:16]Will perform clean checkout. Reason: Checkout directory is empty or doesn't exist
[15:12:16]Cleaning E:\w\DVLP_MONITOR_4_ARCHITEC_WITHOUT_BRAND
[15:12:17]Using  vcs information from server. Reason: no revision information for build  configuration "50 - base_lib Compile Win32 Debug" and checkout directory  E:\w\DVLP_MONITOR_4_ARCHITEC_WITHOUT_BRAND on agent
[15:12:17]VCS Root: CITools (1m:28s)
[15:12:17]revision: 254515
[15:13:45]VCS Root: TFS (9m:18s)
[15:13:45]checkout rules: main=>src/main; iplir=>src/iplir; revision: 254514


The next build, just Run was clicked:

[15:40:57]Updating sources: agent side checkout (21m:50s)
[15:40:57]Using vcs information from agent file: DVLP_MONITOR_4_ARCHITEC_WITHOUT_BRAND.xml
[15:40:58]VCS Root: CITools (41s)
[15:40:58]revision: 254515
[15:41:39]VCS Root: TFS (21m:07s)
[15:41:39]checkout rules: main=>src/main; iplir=>src/iplir; revision: 254514
[15:41:39]Directory E:\w\DVLP_MONITOR_4_ARCHITEC_WITHOUT_BRAND\src\iplir is empty, forcing full checkout
[15:41:39]Directory E:\w\DVLP_MONITOR_4_ARCHITEC_WITHOUT_BRAND\src\main is empty, forcing full checkout


Please note, the revision number is the same.

Then I was experimenting with swabra, with auto named dir (with VCS settings hash) haven't got any different results


Today I tried server-side checkout:
Clean:

[12:50:28]Updating sources: server side checkout (1h:32m:25s)
[12:50:28]Using  vcs information from server. Reason: no revision information for build  configuration "50 - base_lib Compile Win32 Debug" and checkout directory  E:\w\ee39a1da6943b37 on agent
[12:50:35]Will perform clean checkout
[12:50:35]Clean checkout reasons
[12:50:35]Agent doesn't have any version of the project sources
[12:50:35]Checkout directory is empty or doesn't exist
[12:50:35]Transferring cached clean patch for VCS root: CITools
[12:50:35]Building incremental patch over the cached patch
[12:51:29]Building and caching clean patch for VCS root: TFS
[14:20:36]Transferring cached clean patch for VCS root: TFS
[14:20:36]Transferring repository sources: 3.21 MB so far...
[14:22:18]Repository sources transferred: 652.52 MB total
[14:22:18]Average transfer speed: 121.4 KB per second
[14:22:19]Removing E:\w\ee39a1da6943b37
[14:22:19]Updating E:\w\ee39a1da6943b37
[14:22:19]Updating E:\w\ee39a1da6943b37\src\iplir
[14:22:19]Updating E:\w\ee39a1da6943b37\src\main
[14:22:21]Updating E:\w\ee39a1da6943b37\src\iplir
[14:22:26]Updating E:\w\ee39a1da6943b37\src\main



Just Run btn clicked:
[14:41:30]Updating sources: server side checkout
[14:41:30]Using vcs information from agent file: ee39a1da6943b37.xml
[14:41:31]Building incremental patch for VCS root: CITools; checkout rules: =>; revision: 255357 --> 255357
[14:41:31]Building incremental patch for VCS root: TFS; checkout rules: iplir=>src/iplir main=>src/main; revision: 255358 --> 255358
[14:41:31]Repository sources transferred


For server-side option the revision number is also the same.

Above I know why it makes clean checkout with agent-side option. It makes it because it forces deletion of all sources in checkout dir. But I don't get why it deletes all sources, agent doesn't put any info into log about it. Also there is no reason to make deletion, I even haven't commited any changes to code, just run the same build multiple times on the same agent.


With the server-side checkout I see that the second run was fast but the first one was extreamly slow - that's why I am trying to use agent-side checkout more than server-side. I am running thousands of builds a day, most of them are different, so source-code downloading is very critical to me because it takes much part of time the whole build takes.


In this case I complain of agent-side checkout behavior for TFS. It differs with server-side one. I thought that checkout rules were the key to such behavior, but when I disabled it I got the same code deletion without any notify.

I removed checkout rules for testing:
Clean build:

[15:00:07]Updating sources: agent side checkout (29m:47s)
[15:00:07]Will perform clean checkout. Reason: Checkout directory is empty or doesn't exist
[15:00:07]Cleaning E:\w\75a221f5ae7215b9
[15:00:08]Using  vcs information from server. Reason: no revision information for build  configuration "50 - base_lib Compile Win32 Debug" and checkout directory  E:\w\75a221f5ae7215b9 on agent
[15:00:08]VCS Root: TFS (29m:03s)
[15:00:08]revision: 255505
[15:29:11]VCS Root: CITools (43s)
[15:29:11]revision: 255357



Just run that build again:
[15:34:48]Updating sources: agent side checkout (15m:18s)
[15:34:48]Using vcs information from agent file: 75a221f5ae7215b9.xml
[15:34:58]VCS Root: TFS (13m:55s)
[15:34:58]revision: 255505
[15:48:53]VCS Root: CITools (1m:13s)
[15:48:53]revision: 255357








0
Comment actions Permalink

Is it possible that your build deletes the folder E:\w\DVLP_MONITOR_4_ARCHITEC_WITHOUT_BRAND\src\iplir ?
The folder exists immediately after the build?

Otherwise, you could try agent-side checkout without custom dir (we had here in combination with Swabra a problem that should be fixed in 9.0.2, however). Just to isolate the error.

0
Comment actions Permalink

>Is it possible that your build deletes the folder E:\w\DVLP_MONITOR_4_ARCHITEC_WITHOUT_BRAND\*
The build doesn't delete source code inside it's folder. As you can notice in my message above the second build run with server-side checkout doesn't download whole sources again.
>The folder exists immediately after the build?
The whole folder E:\w\DVLP_MONITOR_4_ARCHITEC_WITHOUT_BRAND\* exists after the build.

>Otherwise, you could try agent-side checkout without custom dir (we had  here in combination with Swabra a problem that should be fixed in 9.0.2,  however). Just to isolate the error.
I tried agent-side checkout without custom dir. It shows the same behavior as you may find in my huge message above in  custom dir build logs, just  E:\w\DVLP_MONITOR_4_ARCHITEC_WITHOUT_BRAND\ is replaced with E:\w\75a221f5ae7215b9 there.

Also I am using 9.0.2 and did all the test runs in this version.

0
Comment actions Permalink

Do you have anything to add?

0
Comment actions Permalink

Alina, do you have any new about my problem above mentioned in discussion with mar99?

0

Please sign in to leave a comment.