Cache source code with worker or 3-d party server
Can I cache source code with worker or 3-d party server instead of caching it on master?
I want to put I/O load from master (single point of load) to other servers. All my agents are using server-side checkout, so when I add an agent the I/O load on TC master increases proportionally
Please sign in to leave a comment.
Hi Bulat,
As for now you can map the .BuildServer/system/caches directory to the desired location using OS-specific means (filesystem directory links / symlinks). Please read the realeted blog post.
Also please watch/vote for the related feature request.
Great! I've voted.
Also I mean you already have a vcs-worker server. Let it cahce sources, not master. I think it should be another feature request, don't you?
My I/O problem is network, not disks, since we're using raid10, he-he.
Attachment(s):
vcs-worker as chache server.png
Yes, you are right, it is not the same. Please feel free to create new feature request. However won't it be an option to use agent-side checkout?
I've tried to use agent-side checkout, but it forces to download full sources every time I run build. Also, I still want to have caching server which stores already processed with checkout rules sources to make for end-side agent downloading it as just bunch of files. No need mappings/workspaces for it. Additionally I'll be able to make my TFS happy not loading it so much with get sources requests/
I believe that this is due to your configuration. We also use agent-side checkout and have configured it so that only an update is necessary.
That's great. Do you mean I have 'Clean sources on each build' checkbox checked or you did some deeper config tuning to avoid sources full uploading?
I use TFS as version control system.
Hi Bulat,
Please check that none of Automatic Clean Checkout cases are met. Also please check that advanced TFS VCS root option "Enforce overwrite all files" is set to false and no internal properties related to TFS are configured.
Hi Alina,
![]()
![]()
I've checked, none of cases are met.
I've run one configuration more than 6 times with different settings.
I tried custom checkout directory. The agent was disabled and I run the same config again and again. Every time I didn't expected clean checkout - I got it.
I tried with automaticaly named chekout dir. I run that config couple of times and got different dirs. The third time the build was started in the same dir as previous, but I still got clean checkout.
Please take a look into agent teamcity and build logs attached, I don't know how to make agent not to perform clean checkout each time. Probably it happens because of checkout rules, but it shouldnt.
Attachment(s):
teamcity-build.logs.zip
teamcity-agent.zip
Do you have numbers that your config is working? E.g. each time I've run the same build (even no changes commited) I got ~25 minutes for total build (19-27min) and 10-19 minutes for sources downloading, all the time - clean checkout.
We use Subversion, so it certainly is not 1:1 to compare.
clean/full checkout
Updating sources: agent side checkout (7m:57s)
[14:25:07]Will perform clean checkout. Reason: Checkout directory is empty or doesn't exist
[14:25:07]Cleaning D:\BuildAgent1_1\work\XXX
[14:25:08]Publishing to local artifacts cache is disabled, skipping.
[14:25:09]Using vcs information from server. Reason: no revision information for buildtype build and checkout directory D:\BuildAgent1_1\work\XXX on agent
[14:25:09]VCS Root: VCS_root_name (7m:55s)
[14:25:09]checkout rules: build/teamcity=>; =>source; revision: 37503_2015/01/30 14:23:06 +0100
[14:25:09]Using working copy format 1.7
[14:33:04]Using working copy format 1.7
update (with revert, without it would be faster)
Updating sources: agent side checkout (26s)
[12:13:29]Using vcs information from agent file: XXX.xml
[12:13:29]Publishing to local artifacts cache is disabled, skipping.
[12:13:30]VCS Root: VCS_root_name (25s)
[12:13:30]checkout rules: build/teamcity=>; =>source; revision: 37496_2015/01/29 11:37:29 +0100
[12:13:30]Using working copy format 1.7
[12:13:30]Fast SVN update is enabled
[12:13:30]Reverting local changes for working directory (14s)
[12:13:30]svn revert in D:\BuildAgent1_1\work\XXX\source
[12:13:54]Using working copy format 1.7
[12:13:54]Fast SVN update is enabled
[12:13:54]Reverting local changes for working directory
[12:13:54]svn revert in D:\BuildAgent1_1\work\XXX
Can you copy the part of the build logs with the checkout here?
We use TFS:
Clean checkout (agnent-side custom dir):
The next build, just Run was clicked:
Please note, the revision number is the same.
Then I was experimenting with swabra, with auto named dir (with VCS settings hash) haven't got any different results
Today I tried server-side checkout:
Clean:
Just Run btn clicked:
For server-side option the revision number is also the same.
Above I know why it makes clean checkout with agent-side option. It makes it because it forces deletion of all sources in checkout dir. But I don't get why it deletes all sources, agent doesn't put any info into log about it. Also there is no reason to make deletion, I even haven't commited any changes to code, just run the same build multiple times on the same agent.
With the server-side checkout I see that the second run was fast but the first one was extreamly slow - that's why I am trying to use agent-side checkout more than server-side. I am running thousands of builds a day, most of them are different, so source-code downloading is very critical to me because it takes much part of time the whole build takes.
In this case I complain of agent-side checkout behavior for TFS. It differs with server-side one. I thought that checkout rules were the key to such behavior, but when I disabled it I got the same code deletion without any notify.
I removed checkout rules for testing:
Clean build:
Just run that build again:
Is it possible that your build deletes the folder E:\w\DVLP_MONITOR_4_ARCHITEC_WITHOUT_BRAND\src\iplir ?
The folder exists immediately after the build?
Otherwise, you could try agent-side checkout without custom dir (we had here in combination with Swabra a problem that should be fixed in 9.0.2, however). Just to isolate the error.
>Is it possible that your build deletes the folder E:\w\DVLP_MONITOR_4_ARCHITEC_WITHOUT_BRAND\*
The build doesn't delete source code inside it's folder. As you can notice in my message above the second build run with server-side checkout doesn't download whole sources again.
>The folder exists immediately after the build?
The whole folder E:\w\DVLP_MONITOR_4_ARCHITEC_WITHOUT_BRAND\* exists after the build.
>Otherwise, you could try agent-side checkout without custom dir (we had here in combination with Swabra a problem that should be fixed in 9.0.2, however). Just to isolate the error.
I tried agent-side checkout without custom dir. It shows the same behavior as you may find in my huge message above in custom dir build logs, just E:\w\DVLP_MONITOR_4_ARCHITEC_WITHOUT_BRAND\ is replaced with E:\w\75a221f5ae7215b9 there.
Also I am using 9.0.2 and did all the test runs in this version.
Do you have anything to add?
Alina, do you have any new about my problem above mentioned in discussion with mar99?