Docker Wrapper with Teamcity Agents running on Kubernetes

Hi all,

 

I'm experimenting Teamcity deployment running on k8s and facing a problem with the docker wrapper.

Our existing deployment runs in a physical machine so it works fine, but while doing the tests on k8s, I'm not able to find a good solution to make the docker wrapper work.

If I read the instructions on teamcity-agent docker hub, it says it's necessary to map the folders such as

 

docker run -it -e SERVER_URL="<url to TeamCity server>"  \
    -u 0 \
    -v <path to agent config folder>:/data/teamcity_agent/conf \
    -v /var/run/docker.sock:/var/run/docker.sock  \
    -v /opt/buildagent/work:/opt/buildagent/work \
    -v /opt/buildagent/temp:/opt/buildagent/temp \
    -v /opt/buildagent/tools:/opt/buildagent/tools \
    -v /opt/buildagent/plugins:/opt/buildagent/plugins \
    -v /opt/buildagent/system:/opt/buildagent/system \
    jetbrains/teamcity-agent

 

The problem I see is that the agent will have the PVCs for these volumes, but once the wrapper will try to map the corresponding folders to the container, the folder will be empty.

What is the best way to make this work? I would prefer to avoid having to use hostPath on the kubernetes hosts and creating the physical directories on each server for example, which would create a possible mess and also hard to manage/keep track.

Error that I get from the build log

 

10:56:36][Step 4/6] Running step within Docker container maven:3.6-jdk-11
[10:56:37][Step 4/6] Starting: /bin/sh -c ". /opt/buildagent/temp/agentTmp/docker-wrapper-1712215431782795103.sh && docker run --rm -w /opt/buildagent/work/acd2f6c64e53bf27 --label jetbrains.teamcity.buildId=13 -v "/opt/buildagent/lib:/opt/buildagent/lib:ro" -v "/opt/buildagent/tools:/opt/buildagent/tools:ro" -v "/opt/buildagent/plugins:/opt/buildagent/plugins:ro" -v "/opt/buildagent/work/acd2f6c64e53bf27:/opt/buildagent/work/acd2f6c64e53bf27" -v "/opt/buildagent/temp/agentTmp:/opt/buildagent/temp/agentTmp" -v "/opt/buildagent/temp/buildTmp:/opt/buildagent/temp/buildTmp" -v "/opt/buildagent/system:/opt/buildagent/system" --env-file /opt/buildagent/temp/agentTmp/docker-wrapper-7802146401024842252.envList --entrypoint /bin/sh "maven:3.6-jdk-11" /opt/buildagent/temp/agentTmp/docker-shell-script-3887750566398062691.sh"
[10:56:37][Step 4/6] in directory: /opt/buildagent/work/acd2f6c64e53bf27
[10:56:38][Step 4/6] /bin/sh: 0: Can't open /opt/buildagent/temp/agentTmp/docker-shell-script-3887750566398062691.sh

 

I can confirm on the agent itself the file does exist

# kubectl get pods -n teamcitytest
NAME READY STATUS RESTARTS AGE
teamcity-79cd5b77b-4j7qj 1/1 Running 1 25h
teamcity-agent-5fb597b57c-7rkh8 1/1 Running 0 5m33s
teamcity-mysql-5f44d8674-vlvg7 1/1 Running 0 25h

# kubectl exec -n teamcitytest -it teamcity-agent-5fb597b57c-7rkh8 sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.


# ls -l /opt/buildagent/temp/agentTmp/
total 20
-rw-r--r--. 1 root root 2913 Aug 17 11:56 build.finish.properties.gz
-rw-r--r--. 1 root root 2767 Aug 17 11:56 build.start.properties.gz
-rw-r--r--. 1 root root 2484 Aug 17 11:56 docker-shell-script-3887750566398062691.sh
-rw-r--r--. 1 root root 1155 Aug 17 11:56 docker-wrapper-1712215431782795103.sh
-rw-r--r--. 1 root root 362 Aug 17 11:56 docker-wrapper-7802146401024842252.envList

 

I did also one test to start a container in the build step and bound the /opt/buildagent/temp folder example and can confirm the folder is empty.

Any suggestions/recommendations ? It would be nice to be able to run the agents on k8s but still make use the wrapper so we don't need to rethink our pipelines and etc.

 

Thanks in advance.

 

Please sign in to leave a comment.