Run tests in a Linux container (Docker or LXC)

Our environment is evolving into a large number of microservices, which is pretty cool, but poses a problem the testing perspective.  We deploy in LXCs so each service gets it own world where it has root, and it doesn't need to worry about stepping on any one else (or getting stepped on by anyone else).  This works great in production.  But in test we have a problem.  In order to make efficient use of agent hardware and licenses we want to use a shared pool of agents.  Some sevices may rev 20 times in a day while others may rev zero times.  A week later it could flip.

I would like a way to get TeamCity agents to spin up a specific LXC or Docker image for each project and execute the tests in there.  It is trivial to have a test step that calls "docker run" but I really would like to have a clean way to have the agent inject the custom scripts and code into the container.  In this mode the agent would run outside the LXC, but the tests would execute in the LXC.

It would also be OK to have the agent execute inside the LXC, but then the TeamCity server would need to be able to connect to a host and ask it to launch an LXC / Docker container.  This would be pretty slick actually.

In either case I want to make it super easy for developers who are used to using TeamCity to run tests to get them run inside the containers.

Any suggestions of how to make this work today, or ideas to add this capability in a future release?

4 comments
Comment actions Permalink

Hi Paul

Which build runners in TeamCity do you use to run your build configurations? Which test frameworks your projects use?

We are thinking on better integration with Docker and Vagrant, and have a basic build runner TeamCity.Virtual.
But test reporting is more complex task, and probably will requireadditional configuration.

0
Comment actions Permalink

For runner we use:
Runner type: Command Line (Simple command execution)

For test framework it is a mix, but mostly Python tests using regular unit test, twisted.trial or nosetest.

The more I think about it this is basically like trying to execute tests on a remote machine.  It happens to be inside your current machine, but effectively it is remote.  I can see how that is challenging to get the results properly.

Another option for us is to run the agent inside the docker container.  The down side here would be licensing.  The goal here is to have lots of different docker containers with pre-baked test environments, so to use this mode we would need lots of agent licenses.   If the server had the ability to spin up the docker containers (with agents inside them) on demand, and we paid only for simultaneos executing agents license wise, then this model could work.

Basically if I have 10 different container recipes for 10 different projects, but I only have 2 physical build machines, so I only want to run 2 tests in parralel, let us pay for 2 agents, not 10...

0
Comment actions Permalink

With Python tests it's a bit easier task.
Our teamcity-messages package enables test reporting by TeamCity's service messages.
This way you can have a build agent outside of a container and run your tests by docker run command. A build tool starts inside a container, test results are printed to STDOUT, and a TeamCity build agent is able to detect them.

This approach doesn't work with other build runners which do not use service messages. Please vote for TW-38023 feature request in our public issue tracker.

0
Comment actions Permalink

Hello,

We have come to the same need of executing build configurations inside containers:

  • This allows any one of of our agents to support multiple build environments.
  • It also means that we can upgrade or modify our building environments (containers) without affecting or independently of our agents.


To this end have implemented a Meta Runner that allows us to run builds inside a docker container.

The meta runner parameters allow to choose which one of a number of pre existing building containers to use and what parameter or commandline to use inside that container.
This so far works well and we use XML Report Processing for test reporting.

Attached is an example of how the meta runner is used.


Each time a build steps uses the "DinD Meta Runner" the build images recives via volume mounts the src code and a data area (the .m2 cache).
We have build containers now for different type of needs:

  • java/maven
  • python
  • rpm creation
  • application tests


This method however has these drawbacks:

  • we cannot use pre existing metarunners to execute the build
  • each step runs inside a new container and if the build requires several steps with some initialization then each steps need to re execute the initialization code


We would like to enlist jetbrains support to: Provide a new "Docker Build" feature, electable like "Free disk space" or "XML Report Processing". This way when in "Build Steps" that feature is selected  the build script will be executed inside a container. The container name should be provided via the build feature params or the build parameters.

This will allow to:

  • use any existing meta runners
  • run all steps inside the same container



thx,
Om



Attachment(s):
Screen Shot 2015-01-06 at 9.04.38 AM.png
0

Please sign in to leave a comment.