TeamCity artifact dependency cache in multiple GEO scenario

Project:
Windows, C++/.NET based
Visual Studio, Installshield
TeamCity 8.1.4

Quick background:
My project has been ramping up on TeamCity for a few months now. I built our controller and agents out on a free version (proof of concept), but now I'm looking to migrate this to our internal team which manages the TeamCity server for the whole company. The TeamCity server is located in North America. However, our poject will utilize 20-30 agents initially (and will grow), split evenly between North America and another country. Obviously, the download/upload pipe between this 'other country' and our server in North America, isn't very fast.

We make heavy use of snapshot dependencies and artifact dependencies. I understand that TeamCity caches the artifacts on each agent. Howvever, it's not clear to me whether all agents in an agent pool will first query the agents to determine if they have the required artifacts before going to the server or if the artifact cache only works for the local agent.

Problem/Question:
I need to avoid at all costs any artifact dependency download from Agents in the 'other country' from the server in North America as this could take hours. These agents will have very slow download/upload speeds to the server in North America, and will defeat the purpose of CI.

How does the Agent artifact cache work? Will agents only look for artifacts in their local cache, or will/can they query other agents to determine if these agents have the artifact they require, prior to falling back to the server for the artifact download?

What is TeamCity's best recommendation for how to handle this scenario? Our intent is to have one managed teamcity server with multiple agents spread across different geographical locations all part of one 'big project' that retains all the great speed/efficiency that CI offers. So for instance, the team in North America might have 15 agents working on 50% of the project, while another country has 15 agents working on the other 50% of the project. All these agents can require artifacts/dependencies from the other 50% of the project - they may need to download/upload artifacts that are worked on in another country. How do I ensure the agents in the 'other country' do not get slowed down resolving artifacts from North America?

1.) I could build out 2 servers, one for each country, with the full stack on each side, but that is a waste of resources, is inefficient to manage, and doesn't provide a single location to view the current state of the entire project.

2.) I could require that all bulid chains get built on a single agent, but that defeats the scalable architecture of the agents.

3.) I could try some sort of DFS strategy to attempt to 'hide' this bandwidth issue from teamcity. Not sure if this would be feasible.

4.) I could schedule all agents in this 'other country' to build all their build configurations before their day starts thereby pulling down all the most recent code changes from North America. Same could be done in reverse. This would be hepful, but any changes that occur after the scheduled kick off time would have to be downloaded from the server. It also seems silly to have to force all agents to build prior to their day starting just to ensure that their artifact cache is updated locally on every agent.

5.) Some other configuration/option I'm not aware of? Or a new feature to better support multiple geo configurations in future teamcity versions?

5 comments
Comment actions Permalink

Hi Greg,

Thank you for detailed description of your environment.
I can suggest to use Torrent plugin. It uses BitTorrent protocol and helps to download artifacts faster, especially in a distributed environment. For more details please see a related blog post. Please make sure that both agents and server bind on the first available port in 6881-6889 interval (see comments in blog post).
If Torrent plugin does not help, the solution will be to #2, to configure build chains to build on a single agent. In this case, it is necessary to understand what requires more time: downloading of artifacts or build queue downtime, and select the better approach.

Regarding your question about agent cache. Agent looks for artifacts only locally and does not use caches of other agents.

0
Comment actions Permalink

Thanks Alina. This mostly answers my question.  I will explore the torrent plugin option, but I'm not sure this will be supported by our IT department.

Question on the torrent plugin: When an agent finishes a build and seeds the artifacts immediately, can other agents immediately download these artifacts?

There was a comment in the torrent plugin thread that made me doubt this is possible:

"We don’t use BitTorrent protocol to publish artifacts to server.
However, as of 8.1, all artifacts that agent produces during build execution are stored in the local (agent’s) artifacts cache. And agent seeds its artifacts directly from cache.
In two words, artifacts can be available before they are uploaded to TC if you have a corresponding torrent file. But TC cannot reuse these artifacts before they are completely uploaded to server."

Does the above bolded comment mean these torrents cannot be reused by the agents immediately from the agent cache torrent seed, or is this referring to the ability to download the artifact from the server directly (not the agent cache). My assumption is that other agents can immediately make use of artifacts seeded from the agent cache even if these artifacts haven't been uploaded o the server yet.




Are there plans for TeamCity to introduce a more native method of handling these types of multi-geo scenarios? For instance, the ability to keep track of which agents are 'local' to each other (whether done manually or through some bandwidth discovery algorithm, or even by using the existing agent pool feature), then keep a local cache tracking where each artifact can be downloaded from on 'local' agents (local = fast, remote = slow). I don't care so much about how quickly artifacts propagate back up to the server, but I do care that each agent is able to obtain the artifacts it needs as quickly as possible.

At the end of the day, I want a single teamcity server with many agents across the globe to be able to keep up the promise of CI but also give the full project transparency that the product owners and leadership will want.

Thanks!

0
Comment actions Permalink

One addiitonal question - does the Torrent plugin offer any security? For instance, there's Torrent over SSL (and HTPS for the tracker).

http://blog.libtorrent.org/2012/01/bittorrent-over-ssl/

0
Comment actions Permalink

Build finishes only when artifacts are uploaded to server. After it agent starts to seed artifacts from it's cache. So Torrent plugin speeds up artifacts downloading, not uploading to server.
To upload artifacts while build is running you can use option for TeamCity 8.0 based on Meta-runner: https://github.com/jetbrains/meta-runner-power-pack.

No, Torrent plugin does not offer any security. So you should configure your network to be secure.
We have related feature requests, please watch/vote for them: https://youtrack.jetbrains.com/issue/TW-4339, https://youtrack.jetbrains.com/issue/TW-22366

0
Comment actions Permalink

Thanks Alina.

I can't vote for any of these issues unfortunately because the jetbrains account I
have won't allow me to log in to youtrack.jetbrains.com. I even deleted my jetbrains account, created a new one fresh, and attempted to connect to the youtrack with no success. It just throws me back to the login with no messages about success or failure. I suppose I'll take this up in another thread.

0

Please sign in to leave a comment.