Getting sources from large repository

Hi. I am new to TeamCity but I am very interested in using it for my CI.
From the docs I understand that the full repository is fetched from VCS root despite setting exclude/include checkout rules - and then after all the sources have been fetched the exclude/include rules are applied to the full checkout. Is this true?

Here is the problem. We have a repository around ~3Gb and I simply want to build a very small portion of that for each project setup. Right now the checkout is extremely slow (hardly no IO usage on builder machine) and constantly fails after a few hours. I am assuming the size is a problem?

Does the build agent re-get all sources for each build or does it cache the sources locally and then only get changes?

Is there a workaround to this? Do I need to do something awkward like setting up multiple VCS roots, one for each directory to be included in a build configuration?

I have TeamCity Enterprise Version 3.1.1 (build 6828) and currently only one build agent.

12 comments

The stage when checkout rules are applied depends on specifc VCS plugin. In your case I would do the following: read this document http://www.jetbrains.net/confluence/display/TCD3/Reporting+Issues#ReportingIssues-vcs and enable debug logging for VCS. When next time checkout fails submit an issue to our tracker with logs and the following information:
- you VCS root settings (screenshot would be enough)
- checkout type (on server or on agent)

If checkout is performed on agent attach agent logs too.

When clean checkout is performed result of the checkout is cached on disk. So any subsequent attempts to checkout sources should work much faster.

--
Pavel Sher

0

I am using Team Foundation (TFS). Looking at teamcity-vcs.log all I see are exceptions because the disk is full.

VCS Checkout Mode is set to "Automatically on Server".

Teamcity and BuildAgent are set up on the same server.

Ran another build last night that failed after approx 10 hours getting sources from the same repository mentioned in my previous post ~3Gb. The disk which had 40Gb free at the start of the build gets filled up by temp files under <Teamcity-home-dir>/temp.

Mind you that connecting to smaller repositories >100Mb everything works perfectly.

Is this working as intended?

Is there a way for me to see which exact files are being checked out of TFS?

Edited by: joejoe13 on Jul 3, 2008 2:25 PM

0

Hello, I have created an issue for your request in our tracker at http://www.jetbrains.net/tracker/issue/TW-5374

It is possible to see the list of files. Please have a look to the commandline of TFS-NATIVE process. (You may use ProcessExplorer for that).

The usage of the utility is the following:

Usage:
tfs-native (http|https)://: [/(u|user): [/(p|password):]] Commands: (t|ConnectionTest) (v|GetCurrentVersion) (c|CollectBuildChanges) ((c|current)|) (b|BuildPatch) ((c|current)|) (g|GetContent) The results are being saved to ]]> file. To have a look on the list of checked files, please try looking there.

Could you please attach to the issue(http://www.jetbrains.net/tracker/issue/TW-5374) logs from
running TFS-NATIVE program with enabled logging.
Please start the process with /log key for BuildPatch command from version 0 to c

Thanks!

0

tfs-native is giving me errors on /log parameter.

TeamCity\webapps\ROOT\WEB-INF\classes\bin\tfs-native.exe /log http://source:8080 /u:joe /p:mypass test.txt v

TFS Native Verifier v1.0 Copyright (C) 2006-2008 JetBrains s.r.o. by Mikhail Pil
in and Eugene Petrenko
TFS Native Accessor v1.3 Copyright (C) 2006-2008 JetBrains s.r.o. by Mikhail Pil
in and Eugene Petrenko
ERROR fs.Program - System.ArgumentException: URI formats are not supported.
at System.IO.Path.NormalizePathFast(String path, Boolean fullCheck)
at System.IO.Path.GetFullPathInternal(String path)
at System.IO.File.Delete(String path)
at JetBrains.TeamCity.Tfs.Program.Main(String[] args)
System.ArgumentException: URI formats are not supported.
at System.IO.Path.NormalizePathFast(String path, Boolean fullCheck)
at System.IO.Path.GetFullPathInternal(String path)
at System.IO.File.Delete(String path)
at JetBrains.TeamCity.Tfs.Program.Main(String[] args)

Are you referring to a more recent version?
Getting sources as I write this using tfs-native (plenty of tmp files being created) but no log i assume.


When I request patch from version 0 to c .. does that mean I am fetching all the changesets ever created? If that is the case why would I with a repository I consider quite bulky ever want to do that?

0

Please try the following:

tfs-native.exe /log http://source:8080 /u:joe /p:mypass test.txt BuildPatch $/project_to_fetch 0 c ]]>

It will create log file under %TEMP%/TeamCity.NET. Please clean that folder before staring TFS-NATIVE.

Thanks!

0

Thanks for your quick answers.

Same exception as above. If I leave out /log it works.

tfs-native.exe /log http://source:8080 /u:joe /p:mypass test.txt BuildPatch $/source 0 c c:\tfsnativeTest1 << Does not work
tfs-native.exe http://source:8080 /u:joe /p:mypass test.txt BuildPatch $/source 0 c c:\tfsnativeTest1 << Works

Is there a more recent version of tfs-native I should be using?

Repository contains over 150.000 objects and size is around 3Gb (current status of up to date files / not counting old versions).

Checkout (in temp folder) is at 350Mb 1hrs after start and contains 21.000 objects.

Doing BuildPatch from version 0 to c does that imply that all change sets should be fetched (we have over 10.000 changesets)? Judging from the logs I can only assume that that is the case. Could I somehow force start from current version and not from 0 when getting sources for the first time?

Is TeamCity a viable option for a repository such as this?

Is there no way around having to check everything out before applying the checkout rules - meaning only to check out exactly what I explicitly specify? To have to check out 3 Gb of sources to build a small project ~100kb seems to be a little awkward.

Regards,
Joe

Edited by: joejoe13 on Jul 3, 2008 4:58 PM

0

Ok tfs-native breaks down after about 4 hours from and 5.5Gb start with the following

ERROR TfsHelpers - Server item $/source/root/foo/bar.cpp d4204 was not found in source control at version 9493: System.IO.
IOException: There is not enough space on the disk.

at Microsoft.TeamFoundation.VersionControl.Client.Client.EndDownloadFile(IAsy
ncResult asyncResult)
at Microsoft.TeamFoundation.VersionControl.Client.Client.DownloadFile(String
downloadUrl, String localFileName)
at Microsoft.TeamFoundation.VersionControl.Client.Item.DownloadFile(String lo
calFileName)
at JetBrains.TeamCity.Tfs.TfsHelpers.DownloadFile(Item item, String localItem
)
System.IO.IOException: There is not enough space on the disk.

at Microsoft.TeamFoundation.VersionControl.Client.Client.EndDownloadFile(IAsy
ncResult asyncResult)
at Microsoft.TeamFoundation.VersionControl.Client.Client.DownloadFile(String
downloadUrl, String localFileName)
at Microsoft.TeamFoundation.VersionControl.Client.Item.DownloadFile(String lo
calFileName)
at JetBrains.TeamCity.Tfs.TfsHelpers.DownloadFile(Item item, String localItem
)

There are however 200Gb free on the disk I am trying to get the sources to.

Any ideas?

0

TeamCity needs to scan all history only for clean checkout. After that it will scan only new revisions.

I have created an issue for smarted application of checkout rules at http://www.jetbrains.net/tracker/issue/TW-5377 Please watch/vote/have a look to it. As workaround, please setup new VCS server pointing to the small project sources.

please make sure %TEMP% is accessible for the program. That folder is used to output debugging information.

0

What paths are written in the output xml file. Does those paths maps to the big disk?

Actually, you need to have about 3 time more than the size of file in you VCS on the server to perform checkout.
By default Tomcat uses /temp folder for that. Additionally, TeamCity created VCS caches under ]]>/system/caches

How much files was checked out?

0

It finished at 5.5Gb with exceptions like I mentioned above. The strange thing is that the files being downloaded from the source control are not in the current version - they had been deleted from source control. Is there a reason why these files are being downloaded? I noticed that the files are being checked out as part of a changeset but in that changeset they had been removed (deleted) - therefore I find it strange for tfs-native to be trying to download them.

And the path is on the large disk so the I think the error is coming from TFS.

Edited by: joejoe13 on Jul 4, 2008 5:09 PM

0

Thanks!
I have added the issue for that at http://www.jetbrains.net/tracker/issue/TW-5380 please watch/vote for it.

0

Can you confirm that this is a bug with TFS-Native? I am unable to access the issue you mention.

Regards

0

Please sign in to leave a comment.