Docker compose not waiting for health check period
I've got a docker compose command specified within a step of a settings.kts file:
dockerCompose {
name = "Start some services"
file = ".teamcity/docker-compose.yml"
}
Within the docker-compose file is a service which defines a health check:
...
healthcheck:
test: [ "CMD", "curl", "-f", "http://server:8080/ping" ]
interval: 3s
timeout: 5s
start_period: 20s
retries: 60
...
When the build runs, docker compose is run and the services are started as expected, however the one with the health check fails immediately without retrying for the specified 5 minutes or waiting for the 20s start period.
When TeamCity verbose logging is enabled I see the following output:
10:16:14 Starting docker-compose for .teamcity/docker-compose.yml
10:16:14 Starting: /bin/sh -c docker-compose -f ".teamcity/docker-compose.yml" up -d
...
10:16:29 Creating teamcity_server_1 ...
10:16:29 Docker event: {"status":"create","id":"710950dedf865228789727f5e6582870dd7b1e803434a6ab9a1c4ebbb5e1f1b0" ...}
10:16:30 Docker event: {"status":"start","id":"710950dedf865228789727f5e6582870dd7b1e803434a6ab9a1c4ebbb5e1f1b0" ...}
10:16:30 Creating teamcity_server_1 ... done
10:16:30 Process exited with code 0
10:16:30 Docker event: {"status":"die","id":"710950dedf865228789727f5e6582870dd7b1e803434a6ab9a1c4ebbb5e1f1b0" ...}
...
10:16:31 Run: docker inspect 710950dedf865228789727f5e6582870dd7b1e803434a6ab9a1c4ebbb5e1f1b0 --format="{{ .State.Health.Status }} {{ .Name }}"
10:16:31 unhealthy /teamcity_server_1
10:16:31 Unhealthy status detected while waiting for the container teamcity_server_1/710950dedf865 to become healthy
10:16:31 Container logs for teamcity_server_1/710950dedf865:
10:16:31
10:16:31 Step Start some services (Docker Compose) failed
The same docker-compose file works fine locally, with everything starting up successfully.
I have the following questions:
- Why is the container is only inspected once instead of retrying?
- What is the process that is exiting just before the container dies?
- Why aren't there any container logs?
If anyone has any information that could help I would be grateful.
Please sign in to leave a comment.
I would recommend ensuring the TeamCity version is up-to-date. TeamCity should print the results of the `docker logs` command for the problematic container after detecting the unhealthy status since version 2018.1: https://youtrack.jetbrains.com/issue/TW-54723
Also, if you are using an outdated TeamCity version, the issue can be related to a bug like this: https://youtrack.jetbrains.com/issue/TW-59271