How to handle multiple build sites

For our project we currently have a set-up for two build sites. As the agents from one site can't connect to the server of the other build site (Am I correct that Teamcity doesn't handle SSH connections for builds?),
we are currently thinking of setting up two servers, each with a number of agents. Is there a way that these servers can communicate with each other and publish their build results to each other, so that our users have a single location that they can look at to
see the builds from all the sites (assuming that this system might grow in the future). Or that we can publish to a third public server?
I found the TCJSON plug-in that would allow to publish to a Jenkins/Hudson dashboard, but this doesn't seem the ideal solution. Is there another solution that I'm overlooking?
How do other members in the community handle this? Do you create private VPN connections for each of the agents to a single server. This would mean the server needs a private side on the VPN and a public side to display it's results.
Our agents on both sites are behind a firewall in a NAT domain, and can be accessed over SSH through a gateway, but direct HTTP connections aren't possible, which I understand Teamcity relies on.

5 comments
Comment actions Permalink

Hi Koen,

> Am I correct that Teamcity doesn't handle SSH connections for builds?

Server and agnets should be able to open HTTP connections to each other, so it might work only if you establish a tunnel which is transparenet for both.

> Is there a way that these servers can communicate with each other and  publish their build results to each other

No, at this time there is no integration between several TeamCity servers.

> Or that  we can publish to a third public server?

That can probably be handled out of TeamCity's scope: e.g. a build can upload it's artifacts to an external server (artifactory repository or a simple file share), which users will then access through a suitable interface.
Of course, this can be only appropriate for sharing artifacts. There can also be an artifact to point to the actual build on TeamCity server web UI.

If all your builds would need the ability, these uploading tasks can be implemented as a Java plugin for TeamCity or a service external to TeamCity server.

Another approach might be to provide a summarizing UI that will get information from the both server (e.g. via REST) and display some summary with links pointing to the actual servers.


> How do other members in the community handle this?

Can you please describe what specific issue do you want to solve and what is the ultimate goal? Why can't you use a single server? Do you want to build different or same projects? Why do you need to present results in the same place? etc...

0
Comment actions Permalink

>> Am I correct that Teamcity doesn't handle SSH connections for builds?

>Server and agnets should be able to open HTTP connections to each other, so it might work only if you establish a tunnel which is transparenet for both.

How would you handle this? Currently the best solution seems to be SSH or VPN. (but I haven't any experience with setting up SSH for this)

>> Or that  we can publish to a third public server?

>That  can probably be handled out of TeamCity's scope: e.g. a build can  upload it's artifacts to an external server (artifactory repository or a  simple file share), which users will then access through a suitable  interface.
>Of course, this can be only appropriate for sharing  artifacts. There can also be an artifact to point to the actual build on  TeamCity server web UI.

>If  all your builds would need the ability, these uploading tasks can be  implemented as a Java plugin for TeamCity or a service external to  TeamCity server.

>Another approach might be to provide a summarizing UI that will get information from the both server (e.g. via REST) and display some summary with links pointing to the actual servers.

Thanks I'll look into this

>> How do other members in the community handle this?

> Can  you please describe what specific issue do you want to solve and what  is the ultimate goal? Why can't you use a single server? Do you want to  build different or same projects? Why do you need to present results in  the same place? etc...

We have two build sites beeing configured at the moment, one we already had for a while and a second one we are currently setting up.  The two reasons are :
a.) we were sometimes experiencing unstablility of our first build site
b.) our second build site will have some different hardware that our first one currently can't test for. (for example: CUDA compatible hardware to include CUDA in our build tests, more agents with different OS's)

At both build sites the agents aren't on a public IP, they are all behind a firewall in a NAT configuration (because of security reasons).
Now for our first build site this wasn't a problem because the server was present within the same local network, but having a public IP as well to publish the results.
But as on our second build site all the agents are behind a firewall as well (and passing two gateways) (we do have the possibility here as well to have an public server, if we can push results), so the server from the first build site can't open an HTTP connection to the agents.
The agents can tunnel their http to the server over ssh, but as far as I'm getting this (don't have experience with this) this only works in a single direction, and it's the server from the first build site that needs to be able to open the connection to the second build site.

The reason why we want to present stuff on a single location is obvious, this is how our dev's know the system, and it's a single URL they know by heart, we don't want them to look at different locations to have an overview of all different builds.

0
Comment actions Permalink

Hi Koen,

> How would you handle this? Currently the  best solution seems to be SSH or VPN. (but I haven't any experience with  setting up SSH for this)

Yes, VPN seems to be an appropriate solution (might be a bit expensive in terms of overhead, though).


> We  have two build sites beeing configured at the moment, one we already  had for a while and a second one we are currently setting up.

So under "site" you mean different infrastructure/network and different set of agents.
Is your preferred option is to have only a single TeamCity server at the first site?

Do you need to build the same or different set projects at these sites?


Generally, the agents should be "close" to the server in the network sense. The recommended setup is where the connection between the agents and server is stable and performant enough.

With a single TeamCity server at the first site you will need to establish VPN from the second site to the first one to make them feel like networks wich are directly connected.

With two TeamCity servers installed, one at each location, you will then look into making second site server web UI accessible from outside and display results in some single place.
Seems this is only possible to some basic extent. BTW, I filed a feature request for the case: http://youtrack.jetbrains.com/issue/TW-22342 You are welcome to vote.

0
Comment actions Permalink

> So under "site" you mean different infrastructure/network and different set of agents.

Yes completely different, one in Europe, one in USA.

> Is your preferred option is to have only a single TeamCity server at the first site?

Preferred would be two servers that can sync, this gives some HA as one site can go down, commits are still beeing tested.

> Do you need to build the same or different set projects at these sites?

Both

> Generally, the agents should be "close" to the server in the network  sense. The recommended setup is where the connection between the agents  and server is stable and performant enough.

This is idd another reason to prefer two servers

> With two TeamCity servers installed, one at each location, you will  then look into making second site server web UI accessible from outside  and display results in some single place.

This would be great.

0
Comment actions Permalink

> Preferred would be two servers that can sync, this gives some HA as one site can go down, commits are still beeing tested.

There is no ability to sync two servers as I noted.
Just for a reference, can you please elaborate on what do you mean by syncing?
What specifically should be synchronized?

0

Please sign in to leave a comment.