I'm just writing this to describe our experiences using TeamCity Server Side Checkout and the problems that have persuaded us to stop using it.
As a game developer, our repositories can get very large with middleware and assets taking the majority of the space. A repository of 80GB with 500,000 files is an expected size, although the build agents typically only need perhaps 5-8 GB of that. We are using Subversion.
Plan and intention:
As a priori filtering out of directories seems pretty difficult in Subversion, the agent side checkout has to check the entire repository out, which is slow. The supposed benefits of server side checkout are:
- Space saving: The server side checkout doesn't need a pristine copy halving the amount of space required on the agent. As we use SSDs to improve speed, space does come at a premium.
- Space saving: The checkout rules let us ignore directories we don't need saving more space.
- Source control load: Reduced load on source control as only one server needs to access it, at the cost of increased load and bandwidth required on the TeamCity server.
- Ability to cancel the checkout phase which doesn't seem to be possible on an agent side checkout.
- Increased speed: Sending patch blobs and extracting them inline is usually faster than a Subversion checkout (5 minutes vs. 1 hour).
Unfortunately we hit the following problems:
- Bug: Sometimes checkouts fail due to being unable to modify the file modification date (this is logged). It usually occurs in Boost which is composed of a great number of small files.
- Bug: Sub repositories without pegged revisions don't seem to be handled correctly. It often attempts to check out the sub repository on the revision of the parent repository (which is unrelated).
- Speed: We use sub repositories heavily and it appears that updating the revision the parent repository points to forces a large patch to be sent to the build agents. In contrast agent side svn updates in the same situation are very quick.
- Speed: Whenever an agent doesn't have a checkout, it appears that TeamCity tries to create a new full complete instead of using the last complete patch and sending an incremental update. This means the TeamCity server then spends the 1 hour it takes to perform a complete checkout.
- Speed: We have observed that TeamCity sometimes appears to "throttle" the sending of the sources to agents. Instead of using the network to its full potential (which usually occurs), an agent will perhaps only use a limited amount of bandwidth, sometimes right down to mere kb/s. We have tested performing file transfers at the same time and we have observed this to confirm neither the network, nor the agents are the limiting factor.
- Speed/Size: We suspect that the repository sources sent to the client include content that the checkout rules then removed. An example would be an 8 GB checkout that requires a 24 GB patch to be sent. Because of this transfers take much longer and the space requirements increase as the oversized patch needs to be stored.
- Going back in revisions appears to force a new checkout and full source transfer. We get this situation with chained dependencies as a particular machine may end up building a later revision, but then have to revert to an older one to perform the test which depended on a prior build. The is likely in situations where multiple commits are performed close together.
The end result of the issues means that server side checkout is unreliable and likely to fail. Secondly it creates a "lottery" situation where sometimes a checkout may take:
- Only take a few seconds as the agent is reasonably up to date and just needs a small incremental patch
- A few minutes if the agent doesn't have the sources but needs the full sources transferred
- Over an hour if TeamCity decides to perform a clean checkout from Subversion
- Many hours to over a day if the throttling issue strikes
Often simply running the build again will force another agent to perform the build and it will overtake the first run due to more favourable conditions.