can't connect to unix:///var/run/docker.sock in kubernetes

Avatar
759587231
Created

I have read https://teamcity-support.jetbrains.com/hc/en-us/community/posts/360003164340-TeamCity-Kubernetes-Support-Plugin-Agents-that-can-run-Docker-daemon?input_string=can%27t%20connect%20to%20unix%3A%2F%2F%2Fvar%2Frun%2Fdocker.sock%20in%20kubernetes. the situatin is same to me .but I'm new to k8s either .I don't know after I choose "use pod template from deployment"where to set deploymetn?

Votes

0

Share

9 comments
Avatar
Dylan Munyard
Created

Hi - I recently set up K8s build agent. I had to use a custom pod spec and took some notes you might find helpful:

1. Create a new namespace to deploy them in:

---
apiVersion: v1
kind: Namespace
metadata:
  name: teamcity-agent
2. Enter teamcity-agent in the namespace field.
3. Create a new service account that has admin access, I found this step necessary otherwise TeamCity couldn't read and write into the teamcity-namespace:
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: teamcity-server-agent
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: teamcity-server-agent
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: teamcity-server-agent
  namespace: kube-system
4. Substitute <secret_name> with the name of the secret created above:
# Grab the certificate
kubectl get secret -n kube-system <secret_name> -o 'go-template={{range.items}}{{index .data "ca.crt"}} {{end}}' | base64 --decode
 
# Grab the token
kubectl get secret -n kube-system <secret-name> -o 'go-template={{range.items}}{{index .data "token"}} {{end}}' | base64 --decode
5. Paste the values into the Certificate Authority and Authentication Strategy token fields respectively.
For the agent image:
1. I created a pod spec like this:
apiVersion: v1
kind: Pod
metadata:
  name: jetbrains-teamcity-agent
  namespace: teamcity-agent
spec:
  containers:
  - name: jetbrains-teamcity-agent
    image: jetbrains/teamcity-agent:2019.2.3
    env:
      - name: "DOCKER_IN_DOCKER"
        value: "start"
    securityContext:
      privileged: true
  nodeSelector:
    kubernetes.io/os: linux
    kubernetes.io/arch: "amd64"
0
Avatar
759587231
Created

thanks @Dylan Munyard I have solved that problem by add

securityContext:
fsGroup: 1000
runAsUser: 0

 

I have a another problem : how to avoid pulling dockerhub images again and again when pods restart.

0
Avatar
Dylan Munyard
Created

That depends on the imagePullPolicy, you could add it to the pod spec and set it to ifexists (I'm going on memory here check doco). In term of start up time, TC agents have to download all the plug ins, from 2020.1 they're going to bundle the entire agent, I haven't confirmed it this means containers too.

0
Avatar
759587231
Created

@Dylan Munyard thanks very much,I don't  know how your ci deploy, my teamcity isn't in k8s, It's on other host, agent is in k8s.I used your 

 

apiVersion: v1
kind: Pod
metadata:
  name: jetbrains-teamcity-agent
  namespace: teamcity-agent
spec:
  containers:
  - name: jetbrains-teamcity-agent
    image: jetbrains/teamcity-agent:2019.2.3
    env:
      - name: "DOCKER_IN_DOCKER"
        value: "start"
    securityContext:
      privileged: true
  nodeSelector:
    kubernetes.io/os: linux
    kubernetes.io/arch: "amd64"

I'm trying to get result .......

0
Avatar
759587231
Created

by th way I also choose use custom pod template

0
Avatar
759587231
Created

the same question : can't find USER. the env is abnormal~~@Dylan Munyard

0
Avatar
759587231
Created

A special situation that : I can use docker pull when come into pods as root (set

securityContext:
fsGroup: 1000
runAsUser: 0),

but our CI project depond on non-root user to build.How to solve the yaml?thanks@Dylan Munyard

0
Avatar
Dylan Munyard
Created

I think we should return to your original issue "can't connect to unix:///var/run/docker.sock". This kind of sounds like something is wrong with Docker, especially because it worked after you set your user to run as root. How are you running Kubernetes? I'm using EKS (Amazon) so a lot of this is set up automatically and I didn't have problems, I don't think you should have to run as root to use Docker, that reminds me of an error like this https://www.digitalocean.com/community/questions/how-to-fix-docker-got-permission-denied-while-trying-to-connect-to-the-docker-daemon-socket.

0
Avatar
759587231
Created

thanks a lot 。@Dylan Munyard。when I enter into pods as a non-root, execute

sudo usermod -aG docker

I need enter buildagent or root password, I don't know? Do you have good idea?@Sergey Pak

0

Please sign in to leave a comment.