Multiple docker build agents on same host. [SOLVED]

Hello Adrian!

Unfortunately, there is no plugin which would support a custom Cloud Profile for Proxmox; while it is possible to create one, it would be a considerably complex task. In theory, you could automate starting and stopping of agent containers via Proxmox API, and you could have a "launcher" build which would identify which container you need to start and process accordingly. Likewise, you could have "stopper" build which would check which agents are idle and stop them (using TeamCity REST API).

I can see some overhead involved with this approach, though, so you may want to consider using an external orchestrator to both start builds in TeamCity and control the Proxmox containers state. 

If you would like any details on the REST API usage, or if there is anything else I could assist with, please let me know.

Hi, it's ok, I can use a different machine which has docker installed to achieve what I want.

Although, I'm currenting getting this error when trying to build on docker:

[code]

[12:25:10]

[Step 1/3] Step Configure (Command Line) failed

[/code]

Hello Adrian,

Am I correct to assume that this build is started on an agent which itself runs in Docker container? If yes, you may want to consider the Running Builds Which Require Docker section in this article; long story short, underlying container should be started under the root user in this case. 

Yes.

Docker is running under root on the host, I'm using that docker image for the agent and I have appended -u 0 to the start up.

if I start a prompt for the agent docker instance using:

docker exec -it teamcity-agent /bin/bash

Executing "docker container ls" shows the docker containers, so the docker inside the container is able to control the docker instance running on the host.

But I have been unable to progress, I still keep getting the same error no matter what I do.  I can see the docker agent image has created the various folers and populated them (work, temp etc)

Hello Adrian,

Does the agent start with

-e DOCKER_IN_DOCKER=start

too? (as to enable Docker-in-Docker feature)

Yes it does, I just fixed it though!

You have to pass in the /opt/teamcity/work & /opt/teamcity/temp volume mappings to the container

Now it works, this should make things easier, I can now set up docker container images for the x86 & x86_64 linux distributions to build rather than having proxmox containers which consume ram.

Is it possible on the agent to limit the number build containers it spawns at any given time?  As I have 10+ different Linux builds for various distributions and running them all in parallel (which is currently what happens under proxmox) will result in high load.

Hello Adrian,

Sorry for the delay here. Glad to hear this works! Speaking of the containers that are started for the build in scope of Docker Wrapper feature, those should be removed automatically after the step is over. Docker Compose step will also despawn the containers after the build.

Hi Fedor,

My question (which I've already answered for myself) was to do with running parallel builds.  Having got most of my containers up and running, the docker agent will only spin 1 build at a time, so for parallel builds I simply need to set up as many agent containers as I need.

Thanks for the brilliant support (your support is fantastic, I've had suggestions I made to CLion recently make it into the latest release!), I'm an OSS licensee so I really appreciate you answering my queries as I'm not paying customer!

Hello Adrian,

I see, thank you for the clarification! If you are actually starting up agent containers, you could consider creating a simple local Kubernetes setup; it can be integrated with TeamCity to allow to automatically start and stop agents as required. The general details on the setup from TeamCity side are listed there, and here is a neat example of single host K8 installation

I will take a look, but it's probably overkill for what I need! 

Currently, I have the following docker images that are pre-configured to build my application:

Debian 10 (x86_64)

Ubuntu 18.04 (x86_64)

Ubuntu 20.04 (x86_64)

Ubuntu 20.10 (x86_64)

Arch (x86_64)

Fedora 33 (x86_64)

Fedora 32 (x86_64)

Previously these were all Proxmox containers, but now they're docker images which the agent spins up as required.  This dramatically reduces the memory used.  I am running docker on Unraid on a much more powerful PC than was being used by Proxmox.  I've created all the docker images and tested them and they work perfectly just like the containers did.

In addition to these, I also have the following agents:

MacOS Mac Mini

Windows 10 VM (x86 + x86_64)

Raspbian 10 (arm32)

Raspbian 9 (arm32)

The biggest problem for me was Linux, specifically, when using containers the memory requirements get quite large when targeting lots of distributions, now I've switched to docker I have solved that particular problem.

All I need to do is duplicate my teamcity-agent container (which is easy on Unraid) and spin up as many containers as I want for parallel builds.

My wishlist would be that seeing that the teamcity docker agent spawns the actual docker container for building, that it could handle spinning multiple containers up to a limit set by the user.  I'm probably guessing there are a lot of reasons why this isn't already done and spinning up additional agents isn't a problem.

Btw, my app is at https://www.pingnoo.com and the code is at https://github.com/nedrysoft/pingnoo

I am really grateful for the licenses and support I've received from you guys, it's awesome and first class.  I fell in love with CLion within a few minutes of trying it out.

One thing with TeamCity is that if you want to delete or clone a configuration, you end up having to dig down through many levels, the options to clone or delete are not at the top level of the configuration which I find slightly awkward, especially if you haven't been using it for a while and end up wondering where the option is in the menus.

Ok, reading the info on dockerhub, you cannot run multiple docker agents on a single host.

It then goes on to say that you can rebuild the image to use a different /opt/buildagent path.

Is this correct?  I can build my own teamcity agent images to support this?

Are there any instructions online on how to do this?

Hello Adrian,

Glad to hear you are loving our support, thank you! 

Speaking of the usability issue with delete/copy configuration; as of now those are available in edit context, so to simplify permission control (only the users able to edit configuration could remove it, for example). However, you can suggest a change at any moment via our issue tracker.

In regards to agent image customization - yes indeed, you could do this. There are two options:

1) you could start the agent container, enter it interactively and change the /opt/buildagent path, and then commit a new image from the container to be started later (the procedure is described in the Customization section of the agent image description on the Docker Hub), or 

2) you could create your own dockerfile based on the TeamCity agent image; here is the public repository containing configuration of the TeamCity Docker images which you could reference to amend the container as required. 

Hi,

I've read the information on dockerhub, but it says to change /opt/buildagent but doesn't tell you where you need to make the changes, I've looked at the Dockerfile and can't see anything obvious.

What files do I need to change to allow me to create new images that use /opt/buildagent-1, /opt/buildagent-2 etc.  I think I'm looking for a script or something where I can change these, but I cannot find anything.

The documentation on dockerhub says "change /opt/buildagent to something else" but doesn't tell you where you need to make the change.

I just want to create multiple personal copies of the buildagent docker where each uses it's own unique path.

To make several agents work with docker wrapper and docker.sock option, one have to build different teamcity-agent docker images with different paths of teamcity-agent installation inside those images (like /opt/buildagentN), and start those images with corresponding parameters like -v /opt/buildagent1/work:/opt/buildagent1/work etc.

I have tried this changing all the volumes to their own /opt/buildagent1/<folder> but it doesn't work.  You are setting a mapping of /opt/buildagent1/work inside the container, how does the agent know that it should be using that folder and not /opt/buildagent, surely there's a script or something that also needs modifying to tell the agent where the folders are.

Hello Adrian,

Yes, sure; as a first step, you could make a custom image where /opt/buildagent is renamed to /opt/buildagent1 (or any other different path). The path is referenced in the /run-agent.sh, and it is hardcoded there, so you would also have to amend the script accordingly (at a glance, replacing older path with newer should suffice for the goal). 

Here is the Dockerfile for the minimal agent version - https://github.com/JetBrains/teamcity-docker-images/blob/master/configs/linux/MinimalAgent/Ubuntu/Ubuntu.Dockerfile; it does illustrate how exactly the /opt/buildagent path is populated and how the scripts are copied into the container. The script itself is listed here: https://github.com/JetBrains/teamcity-docker-images/blob/master/context/run-agent.sh

That's what I was looking for, thank you!

I have to say, the documentation on this particular situation is poor (I'm trying to be constructive here), it just says change the /opt/buildagent path and mentions the volumes being passed into the docker run command, which if you do it won't work, because you need to change the value in run-agent.sh which is not mentioned at all and is absolutely crucial to this working.

This really needs more information/detail because I spent 30 minutes trying to find it in the github repo (and I knew what I was looking for but couldn't find it) and just gave up in the end. (there's a lot of files in the repo that are used to build the various images, so it's really hard to find exactly what you're looking for because there's so much stuff that isn't relevant to actually building the Dockerfile).

Now I know where it is, I can give it a go, and hopefully I can dump the virtual machines I spent a while installing with docker instances to solve this.

Hello Adrian,

I agree that the customization paragraph could be improved, and I will check with our documentation team on that. In the meantime, I have discussed the possible approaches to making an image with a custom agent path easier; one of the possibilities discussed was a parameterized Dockerfile which could be rebuilt to obtain a custom image more easily. Thank you for the feedback, it is appreciated!

Hi Fedor,

Thanks for taking that on board!  A parameterised container would be awesome and would solve all these issues and make it much easier than it currently is.

I haven't quite summoned the courage to checkout the docker repo and create modified agent images.  Instead, for the moment, I resorted to creating 2 debian VM's on one server, and 2 more on another server.  All the VM's have docker inside them and are running the standard buildagent container. 

This allows me to have 4 parallel Linux builds (macOS, Windows and Raspberry Pi are all special cases that don't use docker) which is probably the best use of the 2 servers resources, this obviously comes at a RAM cost, but both servers have 64GB of RAM, so I allocated 8GB to each VM so for the moment it's workable, but over the weekend I will build custom agents as this will allow me to have all the agents on one server without the allocation of chunks of memory.

TeamCity is a great product, I feel there could be some usability improvements, something that is a real time sink is for a configuration, having to set the docker image for each build step.  I think that maybe having an option that allows you to set the docker image at the configuration level (i.e all build steps execute in the given docker container), if it's blank then the docker image from the build step is used, if you need to execute a particular step in a different container then you simply just set the container on that step.

This would make it far easier to use and less error prone, especially in my case where I am building images for a lot of different Linux distributions/archs and going through each step is laborious.

Hi Fedor,

I don't think that is right.  I've been playing with the docker script and that variable (AGENT_DIST) is a location inside the container that the start up script uses to initialise the installation.  Changing AGENT_DIST results in it being very broken because it tries to initialise and none of the files are there.

Ok, I have it working.  I've put instructions on my GitHub repo, the instructions are more for my benefit so I can remember what I did at some point in the future when I want to update and have completely forgotten how to reproduce the container.

https://github.com/nedrysoft/pingnoo/tree/develop/docker

I'm amazed that customers haven't been knocking on your door asking for this to be configurable because it's something that is extremely useful when dealing with *many* Linux distribution builds (and there are potentially a lot of them).  I guess your corporate customer just has VM's spun up, but being able to configure an agent just by passing in a parameter into the docker run command would make this SUPER useful.

I really appreciate all your help on this and for pointing me in the right direction, we got there in the end.

I think I've just managed to change the title of the original post to reflect the direction it went and the solution that was obtained.

That was quite a bit of effort, mainly because it involved a bit of consolidation of bits and pieces, but it's now up and running and I can very easily add more docker agents if required.

alderaan, dagobah, endor and both are all docker agents running on the same host - it's a massive memory saver now that they're spawned from docker.

Hello Adrian,

Glad to hear this works for you! In the meantime, I have discussed the matter with the development team - as a result, an usability issue has been created: https://youtrack.jetbrains.com/issue/TW-71121. We should update the documentation shortly (thank you for sharing the repository, it is appreciated!), and when there are any updates about the Dockerfile parameterization, those will be reflected on the tracker. 

Glad to hear this works for you! In the meantime, I have discussed the matter with the development team - as a result, an usability issue has been created: https://youtrack.jetbrains.com/issue/TW-71121. We should update the documentation shortly (thank you for sharing the repository, it is appreciated!), and when there are any updates about the Dockerfile parameterization, those will be reflected on the tracker. 

This would be great, I'm not sure why this hasn't been fixed before given that it's something that most people would want to do - being able to only run one docker agent per host seems like a pointless exercise, you may as well just not bother with the container and install the agent directly! <insert shrug eomji!>

Thanks for your help and info which got me to multiple docker agents on a single host.

Hello Adrian,

This usability issue probably did not have much attention before because the case where one needs multiple Docker agent containers where all of them need to run Docker Wrapper is considerably rare, and if this functionality is not needed on all hosts, a single agent could be configured to run it whereas the rest could run anything but Docker-in-Docker builds. I am really sorry if I am bringing it up too late as I was under impression you need Docker Wrapper on all containerized agents, but if this is not the case, agents could be spun up without amending image - just by changing the path to agent config folder. Here is a line from agent image documentation:

>If you omit these options, you can run several build agents (you need to specify different <path to agent config folder> paths for them), but Docker Wrapper won't work on such agents.

I don't see how you can solve parallel Linux builds without either having multiple machines or the mechanism where you can run multiple docker agents on a single host for parallel builds?

This is specifically how I ended up here.  To build Linux images of my application, I have to build the application on the target Linux system, I can't for example build a version of my app that runs on Ubuntu 18.04 on a ubuntu system running 20.04, I have to build on 18.04 so that the dependencies are resolved.

Unless I've completely misunderstood something in the process, this is why I've ended up migrating from Proxmox containers to docker containers, the Proxmox containers require that I reserve ram which meant a lot of RAM was taken up because the containers had to be running all the time.

I could run a single docker agent on my server, and yes, it would work, but it would increase my build times considerably, I may as well use all the cores I have to build in parallel.

This to me just seems like something that anybody building Linux application binaries would want to do, so I'm surprised that you say it's a rare usage case!  If I had a jaw drop emoji right now I'd embed it! lol

Everything here is pretty much back up and running in terms of my builds and it all seems to work great and I'm looking forward to maybe at some point in the future the behaviour being "fixed" so that I don't have to create custom agent images.

I really appreciate all your help and pointers on this that have got me to this position, and of course, I'm super grateful for the open-source licenses that you've provided me with to help me on pingnoo.

The whole JetBrains team seem awesome and super helpful and accommodating.  I've made several suggestions on CLion which have made it into the latest versions of the product.  Being an OSS (aka free) license holder, I'm blown away with the help I've received considering that I'm not a paying customer and I hope that the suggestions that I've made is helpful and makes a small remuneration for your generosity.

Since trying CLion, I completely switched to it after about 5 minutes from QtCreator which I was originally using, I was paying for the bundle for 6 or so months before requesting the OSS license version.

@... I understand now.

Yesterday I got my docker raspbian stretch and buster images all set up and building my application.  It wasn't until I got to the point where I wanted to use the docker in docker build agent that I realised that it wasn't easily possible to create such a container as the files you have provided target specific versions of ubuntu and arm64.  I had a look around and decided that it was too much work.

It was at this point the lightbulb went on, I could just take the docker images I had created and downloaded the build agent zip file from my server and start the build agent in that container and then repeat the exercise again, now I have one standard build agent running on the pi in a docker container which targets buster and another standard build agent running in a another container which is based on stretch.

The only real downside to this is that the both builds will be triggered simultaneously on the pi, there may well be an option somewhere in teamcity to stop this from happening, possibly with pools or something?

I figured it out, create a shared resource with a count of 1 and then create a build feature on the 2 Pi builds, each using a write lock.

Only one pi build will run at any given time

Hello Adrian,

Yes indeed, this is what I have referred to - build agent does not require the path changes we have discussed before per se, unless you need to run multiple Docker-in-Docker capable agents on the same host (the above example with Pi hints this is not the case). 

A shared resource usage appears to be an elegant (and the only one) solution I can think of, speaking of ensuring a single build per host. Thanks for sharing that!

The Pi solution is limited by RAM, you can’t have many parallel compilations in process because the kernel will kill them when it needs RAM. Thankfully, the shared resources allows me to limit the number of agents that run at the same time, I looked over the weekend to find something obvious, but it wasn’t until I stepped away from the computer and came back hours later than I finally figured out how to do it. I had ordered a 8GB pi, and I thought I’d swapped to using it, but today I realised I hadn’t actually swapped over from my pi 4 with 4GB, I have a bit more RAM now to play with.

Hopefully anybody else wanting to do something similar and who is stuck will find this thread which shows 2 different ways to achieve the same result.

One thing (and it’s off topic here) is that I find the TeamCity GUI is overly terse, trying to navigate around it (especially when looking at a project) often involves redundant clicks because reversing out of a hierarchy gives a different path to when you went in, I find myself having to do a lot of clicks to get to where I want to be.

I mentioned it before with copying a configuration, you needlessly are required to go down to the bottom level before you get the options to copy a config, it could have been shown 1 or 2 levels earlier, it’s pretty hard to find stuff even when you know what you’re doing, I could see a new user sitting there for a long time trying to find out how to copy a build config from a project.

Other pages like the one that show a build summary could be improved by providing hyperlinks to a field for editing directly, currently you see the list, and then you have to click the edit button, and then do the same again.

It’s an incredible piece of software (I’m trying to give constructive feedback) that is hugely powerful, it just feels like I’m navigating around much more than I actually need to.

Hello Adrian,

Your feedback is really appreciated! Sorry for the delay - I was checking with the front-end team on some of the points you have raised. The navigation has become somewhat a focus recently - for instance, an option to quickly jump to a specific configuration option is a "fresh" one:

Personally, I do agree that some of the actions that do require you to enter the edit context could have been accessible from the "outside" (e.g. on the build configuration overview), but this would probably require some clever rework of the permission-related hide/show logic. I am still discussing this with the development team - will keep you posted.

Not sure if I fully understood this point:

> could be improved by providing hyperlinks to a field for editing directly, currently you see the list, and then you have to click the edit button

Could you please illustrate? 

I think the use of hyperlinks straight to field editing would be a huge time saver, this is a classic example. 

Here I can see all the fields, maybe there's a typo or a minor change to be made, I have to hit edit, then find the relevant section on the next page and then click to edit, this could be streamlined by making many of these entries shown here hyperlinks to editing that field on this first page.