Poor performance with Docker in Docker builds

Answered

So, I've got a really weird issue where my tests run really slow when run by TeamCity, but not when I run them myself.

The setup is:

  • Linux host with TeamCity installed (it has an agent running on the host, but it's not the agent I'm using for this)
  • A TeamCity agent which is running in Docker (based on this image https://hub.docker.com/r/jetbrains/teamcity-agent/ )
  • The TeamCity builds done in the Docker agent use Docker In Docker using option 2 in the 'Running builds which require Docker' section of the image documentation above
  • The builds are done using `docker-compose run` which is run in the Docker TeamCity agent

If I let TeamCity run my PHP unit tests using the Docker TeamCity agent and `docker-compose`, they take 28 seconds.

If I `docker-compose exec` into the container that TeamCity has started to run the tests in and manually type the command to run `phpunit` the tests take about 8 seconds.

If I let TeamCity run the same tests using the agent on the host (i.e. the non-Docker one), it only takes about 6 seconds.

How can it be that TeamCity running the tests takes 8 seconds, but I can be in the very same container and run them manually in about 8 seconds?

I'm baffled as to why running tests from TeamCity is so much slower than running them in the container manually. I guess that when TeamCity runs the test, it's also getting line by line reporting going back to the server - could that be what's slowing it down? I'm not sure why it would be so different to running the tests on the host agent, which also will be reporting back to the TeamCity server.

Anyone got any idea at all why I'm seeing such bad performance when the Docker TeamCity agent runs the tests?

0
1 comment

Hi Iain,

Could you please upload mentioned build log examples, teamcity-agent.log files from the respective agents, and teamcity-server.log covering the same timeframes to https://uploads.jetbrains.com/ for us to check? Please also let me know the exact TeamCity version you are using.

0

Please sign in to leave a comment.