Docker-in-Docker docker-compose fails to start container: cannot enter cgroupv2 "/sys/fs/cgroup/docker" with domain controllers -- it is in an invalid state: unknown

Hi, 

I can't reproudce the issue from my side with you supplied compose file. 

But according to your description, it seems that error is due to incorrect permissions. Could you please have a try to run teamcity agent with root user?

docker run -d --name teamcity-agent --user 0 --network my-teamcity-network -e SERVER_URL=http://teamcity-server:8111 -v /opt/dockeragent/agent/conf:/data/teamcity_agent/conf -e DOCKER_IN_DOCKER=start -v /var/run/docker.sock:/var/run/docker.sock jetbrains/teamcity-agent

If it still doesn't work, for further investigation, please share the teamcity-build.log & teamcity-agent.log to https://uploads.jetbrains.com/.  And please let us know the exact Ids after the upload.

Thank you for reaching out. I appreciate your help. I am indeed elevating the privileges allowed in the docker agent container. Below is the full docker run command that I use. Please note that the docker image I'm using is a custom image based on (FROM) TK with minor additions of build tools meant to fit with my build process.

docker run -d --name teamcity-agent-1 -e SERVER_URL="https://host.docker.internal:5443" --add-host host.docker.internal:host-gateway --privileged -e DOCKER_IN_DOCKER=start -u 0 --memory=10g --cpus=6 teamcity-agent-php1

Here is the upload ID of the two requested files: 2023_11_21_W57JRFrWeZ8DbimUuSdbJL

In an effort to help with debugging, the following are the build configurations as code, with sensitive information redacted when appropriate.

Build Configuration Template:

package _Self.buildTypes

import jetbrains.buildServer.configs.kotlin.v2018_2.*
import jetbrains.buildServer.configs.kotlin.v2018_2.buildFeatures.PullRequests
import jetbrains.buildServer.configs.kotlin.v2018_2.buildFeatures.dockerSupport
import jetbrains.buildServer.configs.kotlin.v2018_2.buildFeatures.pullRequests
import jetbrains.buildServer.configs.kotlin.v2018_2.triggers.vcs

object template : Template({
    name = "Yeetnite API Template"
    description = "Base settings for the Yeetnite PHP API Test Suite"

    triggers {
        vcs {
            id = "TRIGGER_1"
            branchFilter = """
                +:rc\-*
                +:<default>
            """.trimIndent()
            perCheckinTriggering = true
            groupCheckinsByCommitter = true
            enableQueueOptimization = false

            enforceCleanCheckout = true
        }
    }

    features {
        pullRequests {
            id = "BUILD_EXT_1"
            vcsRootExtId = "${vcs_root.id}"
            provider = github {
                authType = token {
                    token = "******"
                }
                filterAuthorRole = PullRequests.GitHubRoleFilter.EVERYBODY
            }
        }
        dockerSupport {
            id = "DockerSupport"
            cleanupPushedImages = true
            loginToRegistry = on {
                dockerRegistryId = "PROJECT_EXT_3"
            }
        }
    }
})

Build Configuration

package _Self.buildTypes
import jetbrains.buildServer.configs.kotlin.v2018_2.*
import jetbrains.buildServer.configs.kotlin.v2018_2.buildFeatures.PullRequests
import jetbrains.buildServer.configs.kotlin.v2018_2.buildFeatures.dockerSupport
import jetbrains.buildServer.configs.kotlin.v2018_2.buildFeatures.pullRequests
object unit_testing_build_config : BuildType({
   templates(template)
   name = "Unit Testing"
   description = "Run the Yeetnite PHP API unit tests"
   vcs {
       checkoutMode = CheckoutMode.ON_AGENT
       checkoutDir = ""
   }
   features {
       dockerSupport {
           id = "DockerSupport"
           loginToRegistry = on {
               dockerRegistryId = "PROJECT_EXT_3"
           }
       }
       pullRequests {
           id = "BUILD_EXT_5"
           vcsRootExtId = "${vcs_root.id}"
           provider = github {
               authType = token {
                   token = "******"
               }
               filterAuthorRole = PullRequests.GitHubRoleFilter.EVERYBODY
           }
       }
   }
})

Thank you again for your help. Let me know if you need any more information. I'll be happy to provide it.

If it helps, I can run docker run hello-world and the container runs just fine and both of the containers in the docker-compose file using docker run separately, for that matter. docker-compose is the only command that is broken and can't seem to get the containers up and running. Docker containers that require cgroupv2 are the only ones that are broken. I also realize that the docker engine version is VERY old (20.10) and barely has a version that supports cgorupv2. Any suggestions you have? Thank you.

Thank you again for reaching out. I was under the impression that the teamcity-agent image starts its own dockerd inside of the container. passing in the dockerd socket of the host ends up working fine by the way, but that means that all of the containers and images end up “spilling” onto the host. For one, the host itself has limited storage space and I'm storing the agent data on a very large and fast network share, bypassing that limitation. Using the host's docker daemon doesn't allow for storage expansion of container images. Second, it's harder to maintain and separate out which images are genuinely used for running things on the host or the agent and which containers need to be kept during routine maintenance. With more than one agent (currently the plan) this will only become more difficult. I will try not passing in the host's dockerd socket with a manually-installed version of docker-compose soon once I have time, though, and report back results.

I just checked the Docker Hub README and indeed my assumption is correct. I'm trying to set up docker-in-docker. Please see the following quote:

You have two options if you need a docker daemon available inside your builds.

One is to use docker server from the host, which starts teamcity agent container (docker-out-of-docker). Another is to run a docker server inside teamcity agent container (docker-in-docker).

--privileged -u 0 -e DOCKER_IN_DOCKER=start should be all I need to make this work.

I'm also finding that any exposed ports inside of the agent are actually only opened on the host due to the docker socket being that of the host's inside of the agent container.

Hi ,

>I will try not passing in the host's dockerd socket with a manually-installed version of docker-compose soon once I have time, though, and report back results.

Have you had a chance to try it out yet? Take your time and let me know if there's anything I can do to assist you further.

Yes. I tried it and posted my results in the previous comments on this post. Host docker socket works fine, but the containers and images leak onto the host's Docker (as expected) and so do the port exposes (this is unworkable and and did not expect this). dockerd inside of the teamcity agent image works fine with docker run, but even the latest docker-compose fails to run with the original error being present. The images are downloaded just fine. Thanks for your help.

​Hi,

I have submitted a ticket TW-85440 to YouTrack on your behalf. Please follow this ticket for updates.

Thank you very much. You’ve been most helpful on this matter.