Huge subversion repo which takes 6 hours to sync from scratch

Hi.

We have a repository which contains files (third party libraries etc) needed by other builds. It is huge, around 20GB or so, and a fresh sync takes approximately 6 hours to complete.

Right now I have set up the syncing of this repo as a dedicated build configuration which syncs to a fixed path so that other build configurations will know where the data is located. This works fine when running only one build agent.

Will this work when having more than one build agent or can it happen that one agent gets the sync job for the huge repo and the other (which depends on this data) gets started on another agent?

If this does not work then the only way I can think of to get this working is to sync this repo as part of the build configurations that depend on it. In this case I would like to be able to say that this particular repo should never be cleaned and should be shared among all build configurations which depend on it.

Is it somewhat clear what I am trying to achieve? Do you have any tips how to handle this in a way that would never cause 6-hour checkouts unless when initializing a new build agent for the first time.

Kind regards,
Thomas Sköldenborg

3 comments
Comment actions Permalink

How much time does it take to checkout using regular svn client? You can try to use checkout on agent mode (specified on the VCS settings page) as it works faster and speed should be comparable to regular svn client.

0
Comment actions Permalink

Hi Pavel,

Thanks for your suggestion. Currently TC is configured to check out on agent, so that is not the issue. Checking out via regular svn client is dead slow as well so that is not the issue either. The repo is huge (>20GB) and is located in another office and accessed over VPN so that is the main reason and the only thing I can think of which can speed this up is to have a svn-mirror in our office. Still, I think checking out from a svn-mirror in our office will still take more than one hour and that is not really ideal either. Setting up a mirror is also a one-way thing (i.e. the mirror will be read-only as far as I know) so labelling the repo etc will not work anymore against the mirror.

Kind regards,
Thomas Sköldenborg

0
Comment actions Permalink

Actually checkout is done once per agent. Full checkout can also be performed if VCS root settings are changed. After this only update is performed (if you haven't "clean all files" option turned on).

Another solution is to use checkout on server. In this case checkout is performed once and sources are stored in the server caches. With server side checkout agent will ask server for a patch and server will send full patch (if there were no sources on this agent) + delta. Chances are in your case with slow connection to repository checkout on server will work faster.

As for downsides of server side checkout: you will not have .svn folders on the agent.

0

Please sign in to leave a comment.