All docker images and containers deleted by TeamCity 2017.2.4 (build 51228)
I'm running a single agent on the same machine as the team city server, and have setup docker to run builds for different linux targets. This was all working really nicely for a while, then one day all the builds failed.
I thought Frank had made a bad checkin again, but on closer inspection all the docker images and containers on the machine had been deleted.
Looking at the agent log, because of low disk space, it decided to run the following command:
docker system prune -f -a
Without question this removed:
- all stopped containers
- all networks not used by at least one container
- all unused images
- all build cache
And so all the docker build images I built were gone.
Since we don't have our own docker registry, they are gone gone, needing manual setup again.
I don't think using docker without a registry for your own images would be particularly unusual for a small scale install with one build agent, so this feels like an anti-feature. Even if I do setup a registry it would probably run in a container on the same machine which would be at risk of being pruned if TeamCity ever cleaned up while the registry container was stopped.
So how can I stop TeamCity pruning everything so agressively? Or is there something in docker I can do to prevent these images being pruned (e.g. apply a --filter by default)?
What did I miss?
Please sign in to leave a comment.
Hello Mike,
I'm sorry for the problem. TeamCity agent tries to remove images as the last resort when other means of cleaning data did not help.
As a temporary workaround to disable this feature you can do the following:
Docker commands will be taken from /path/to/edited/commands.properties
Please note that after build agent upgrade, you may need to repeat the procedure again if docker plugin is changed to include other commands in the file.
Also, please vote and watch for https://youtrack.jetbrains.com/issue/TW-50478 issue - it is about a smarter way to clean images/containers from the build agent.
Again, sorry for the problem.
KIR