Team City Linux Agent Docker Build Failing (Docker in Docker) - error creating aufs mount to /var/lib/docker/aufs/mnt/…-init: invalid argument
I am trying to get a TeamCity Linux Agent setup within docker. The Agent needs to be able to build dotnet core 2 apps and then do a docker build on the app.
I have Docker running under Windows 10 atm but the plan is that the container will run under Linux in AWS.
I have gotten the TC Agent container up and running, dotnet stuff sorted but am stuck on the docker stuff.
The following is the TC agent log:
[08:33:47]Step 2/2: Docker Build (2m:16s)
[08:33:48][Step 2/2] Starting: docker build --pull -f Dockerfile .
[08:33:48][Step 2/2] in directory: /opt/buildagent/work/274731defed46d9f/think.ETL
[08:33:48][Step 2/2]
[08:33:48][Step 2/2] Step 1/9 : FROM microsoft/aspnetcore:2.0.4
[08:33:51][Step 2/2] 2.0.4: Pulling from microsoft/aspnetcore
[08:33:52][Step 2/2] 723254a2c089: Pulling fs layer
[08:33:52][Step 2/2] 499be8ca2075: Pulling fs layer
[08:33:52][Step 2/2] a6854c900e79: Pulling fs layer
[08:33:52][Step 2/2] 36653ffe7e35: Pulling fs layer
[08:33:52][Step 2/2] 2c2c279d1b1a: Pulling fs layer
[08:33:52][Step 2/2] 36653ffe7e35: Waiting
[08:33:52][Step 2/2] 2c2c279d1b1a: Waiting
[08:33:54][Step 2/2] a6854c900e79: Download complete
[08:35:02][Step 2/2] 36653ffe7e35: Verifying Checksum
[08:35:02][Step 2/2] 36653ffe7e35: Download complete
[08:35:30][Step 2/2] 499be8ca2075: Verifying Checksum
[08:35:30][Step 2/2] 499be8ca2075: Download complete
[08:35:41][Step 2/2] 2c2c279d1b1a: Verifying Checksum
[08:35:41][Step 2/2] 2c2c279d1b1a: Download complete
[08:35:54][Step 2/2] 723254a2c089: Verifying Checksum
[08:35:54][Step 2/2] 723254a2c089: Download complete
[08:35:58][Step 2/2] 723254a2c089: Pull complete
[08:35:59][Step 2/2] 499be8ca2075: Pull complete
[08:36:00][Step 2/2] a6854c900e79: Pull complete
[08:36:01][Step 2/2] 36653ffe7e35: Pull complete
[08:36:04][Step 2/2] 2c2c279d1b1a: Pull complete
[08:36:04][Step 2/2] Digest: sha256:0a54eac597bdb205f3023410741d7492dcb52cf9a5b1f917755dc755339e4002
[08:36:04][Step 2/2] Status: Downloaded newer image for microsoft/aspnetcore:2.0.4
[08:36:04][Step 2/2] ---> e49b04bf00d5
[08:36:04][Step 2/2] Step 2/9 : ARG source
[08:36:04][Step 2/2] error creating aufs mount to /var/lib/docker/aufs/mnt/ccc81e8c41476a5f72ea23ffc7222ba9b65c47b032a389db9acec26730d1a114-init: invalid argument
[08:36:04][Step 2/2] error creating aufs mount to /var/lib/docker/aufs/mnt/ccc81e8c41476a5f72ea23ffc7222ba9b65c47b032a389db9acec26730d1a114-init: invalid argument
[08:36:04][Step 2/2] Process exited with code 1
[08:36:04][Step 2/2] Process exited with code 1
[08:36:04][Step 2/2] Step Docker Build failed
These are my instructions to get where I am up to
-- Install TC Agent - https://hub.docker.com/r/jetbrains/teamcity-agent/
- docker run -it --name teamcity-agent-linux -e SERVER_URL="http://192.168.50.217:8111" -v D:\TeamCity\Agent\conf:/data/teamcity_agent/conf --privileged -e DOCKER_IN_DOCKER="start" jetbrains/teamcity-agent
-- Install .Net Core 2x - https://docs.microsoft.com/en-us/dotnet/core/linux-prerequisites?tabs=netcore2x
- curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.gpg
- sudo mv microsoft.gpg /etc/apt/trusted.gpg.d/microsoft.gpg
- sudo sh -c 'echo "deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-ubuntu-xenial-prod xenial main" > /etc/apt/sources.list.d/dotnetdev.list'
- sudo apt-get update
- sudo apt-get install dotnet-sdk-2.0.0
-- Fix for VS2017 Docker-Compose Project breaks build on command line -https://github.com/dotnet/cli/issues/6178
- mkdir /usr/share/dotnet/sdk/2.0.0/Sdks/Microsoft.Docker.Sdk
- copy C:\Program Files (x86)\Microsoft Visual Studio\2017{SKU}\MSBuild\Sdks\Microsoft.Docker.Sdk\Sdk to D:\TeamCity\Agent\conf (/usr/share/dotnet/sdk/2.0.0/Sdks)
- cp -a /data/teamcity_agent/conf/Microsoft.Docker.Sdk/ /usr/share/dotnet/sdk/2.0.0/Sdks/
- sudo apt install nuget
I have read and followed (unsuccessfully) Error: "error creating aufs mount to" when building dockerfile
So any ideas how I can fix the error "error creating aufs mount to /var/lib/docker/aufs/mnt/ccc81e8c41476a5f72ea23ffc7222ba9b65c47b032a389db9acec26730d1a114-init: invalid argument"
Please sign in to leave a comment.
The thing is, docker does not support aufs over aufs (aufs inside your docker over aufs used to create the container). I've encountered this on my CentOS dind containers. You have two options:
1. Use old docker (not docker-ce), like 1.13 (error in this case is more sensible BTW, not this mess - RedHat knows how to optimize software). Switch to devicemapper driver (set --storage-driver devicemapper in /etc/sysconfig/docker).
2. With new docker-ce this options does not work for me in CentOS, but vfs driver do (it's not the best option, but I have not managed to work out any other solution). So edit /etc/docker/daemon.json and set:
{
"storage-driver": "vfs"
}
Hi Andrew, and thanks for the tips, Andrey,
Docker themselves recommend staying away from docker in docker, particularly for CI: http://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/
Please check both Andrey's and this suggestions and check whether they can work for you.
Hi, there are to cases here:
1. In my case (infrastructure testing), dind is lesser of two evils - using docker.sock has other side effects, like managing networks or leaking containers, if something goes wrong during test...
2. In case of TC - yep, using dind is not the best solution, but TC requires mapping lots of volumes from host filesystem to containers it runs (and documentation on this matter is missing or incorrect: mapping only two volumes described in docs is not enough to run TC agent inside container).
Overall, either solutions aren't that great. I used dind TC agent for demo purposes only: in prod env I would not recommend installing agent inside container. TC has lot's of options to wrap execution modules inside docker and manages agent upgrades just fine, so you have little to no problem of setting up TC agents once.
To use solution described by jpetazzo in prod with TC, TC has to adapt its docker execution module to pass data inside containers some other way, not by mapping hosts directories (like gitlab does). But this may have some performance implications as well.