Kubernetes Cloud Plugin: Using Custom Pod Template

Hi,

I am trying to run TeamCity build agents (2018.2.4) with the latest Kubernetes cloud plugin. If I run the agents using the single agent pod from an image, the agents start up correctly, but I need to specify environment variables and other settings to allow Docker build functionality to work.

I have tried specifying a custom pod template as follows inside the TeamCity UI:
apiVersion: v1
kind: Pod
metadata:
name: base-mono
namespace: teamcity-agent
spec:
containers:
- name: base-mono
image: private.registry/tcbuildagents-test/base_mono:latest
securityContext:
privileged: True
env:
- name: DOCKER_IN_DOCKER
value: start

I am unable to get any agent to start and cannot see that the TeamCity host is connecting to the Kubernetes cluster at all. I am getting the following in the cloud log:

[2019-10-28 16:41:36,199] WARN [6 Flush Queue 1] - e.impl.FlushQueueVirtualAction - Failed to start agent. null
jetbrains.buildServer.clouds.server.instances.FailedToStartInstanceException
at jetbrains.buildServer.clouds.server.impl.instances.StartInstanceAction.startInstance(StartInstanceAction.java:101)
at jetbrains.buildServer.clouds.server.impl.CloudManagerFacade.startInstance(CloudManagerFacade.java:281)
at jetbrains.buildServer.clouds.server.impl.virtual.StartableAgentTypeImpl.startAgent(StartableAgentTypeImpl.java:77)
at jetbrains.buildServer.clouds.server.impl.virtual.StartableAgentImpl.startAgent(StartableAgentImpl.java:21)
at jetbrains.buildServer.serverSide.impl.FlushQueueVirtualAction.startAgent(FlushQueueVirtualAction.java:34)
at jetbrains.buildServer.serverSide.impl.FlushQueueVirtualAction.startBuildOnAgent(FlushQueueVirtualAction.java:22)
at jetbrains.buildServer.serverSide.impl.FlushQueueVirtualAction.startBuildOnAgent(FlushQueueVirtualAction.java:64)
at jetbrains.buildServer.serverSide.impl.FlushQueueTemplate$1.buildDistributed(FlushQueueTemplate.java:15)
at jetbrains.buildServer.serverSide.impl.buildDistribution.AbstractDistributor$DelegatingDistributorCallback.buildDistributed(AbstractDistributor.java:4)
at jetbrains.buildServer.serverSide.impl.buildDistribution.AbstractDistributor.doDistribute(AbstractDistributor.java:32)
at jetbrains.buildServer.serverSide.impl.buildDistribution.DefaultBuildDistributor.assignBuildsAndStartNewAgents(DefaultBuildDistributor.java:7)
at jetbrains.buildServer.serverSide.impl.buildDistribution.BuildDistributorProxy.assignBuildsAndStartNewAgents(BuildDistributorProxy.java:20)
at jetbrains.buildServer.serverSide.impl.FlushQueueVirtualAction.distribute(FlushQueueVirtualAction.java:53)
at jetbrains.buildServer.serverSide.impl.FlushQueueTemplate.flushQueue(FlushQueueTemplate.java:3)
at jetbrains.buildServer.serverSide.impl.FlushQueueAction$4.run(FlushQueueAction.java:11)
at jetbrains.buildServer.serverSide.impl.FlushQueueAction$5.run(FlushQueueAction.java:2)
at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupState.runUnderLock(ServerCleanupState.java:51)
at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupState.runReadAction(ServerCleanupState.java:61)
at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl.executeWithInactiveCleanup(ServerCleanupManagerImpl.java:115)
at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl$$FastClassBySpringCGLIB$$ba2c8525.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:738)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at jetbrains.buildServer.serverSide.impl.auth.TeamCityMethodSecurityInterceptor.invoke(TeamCityMethodSecurityInterceptor.java:10)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.CglibAopProxy$FixedChainStaticTargetInterceptor.intercept(CglibAopProxy.java:622)
at jetbrains.buildServer.serverSide.impl.cleanup.ServerCleanupManagerImpl$$EnhancerBySpringCGLIB$$321f6a26.executeWithInactiveCleanup(<generated>)
at jetbrains.buildServer.serverSide.impl.FlushQueueAction.flushQueue(FlushQueueAction.java:21)
at jetbrains.buildServer.serverSide.impl.BuildServerImpl.flushQueue(BuildServerImpl.java:209)
at jetbrains.buildServer.serverSide.impl.BuildServerRunner$2.doSomething(BuildServerRunner.java:1)
at jetbrains.buildServer.serverSide.impl.BuildServerRunner$BuildServerWorker.lambda$runAction$0(BuildServerRunner.java:42)
at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadNameFuncThrow(NamedThreadFactory.java:107)
at jetbrains.buildServer.serverSide.impl.BuildServerRunner$BuildServerWorker.runAction(BuildServerRunner.java:8)
at jetbrains.buildServer.serverSide.impl.BuildServerRunner$BuildServerWorker.run(BuildServerRunner.java:11)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NullPointerException
at java.util.HashMap.putMapEntries(Unknown Source)
at java.util.HashMap.putAll(Unknown Source)
at jetbrains.buildServer.clouds.kubernetes.podSpec.CustomTemplatePodTemplateProvider.getPodTemplate(CustomTemplatePodTemplateProvider.java:69)
at jetbrains.buildServer.clouds.kubernetes.KubeCloudImageImpl.startNewInstance(KubeCloudImageImpl.java:93)
at jetbrains.buildServer.clouds.kubernetes.KubeCloudClient.startNewInstance(KubeCloudClient.java:61)
at jetbrains.buildServer.clouds.server.impl.instances.StartInstanceAction.startInstance(StartInstanceAction.java:92)
... 34 more

Any suggestions?
Thanks.

0
1 comment

You've no doubt solved the problem, but I had the same error because my pod spec wasn't valid YAML. You can lint it through kustomize first, but I found the source code on GitHub has unit tests with sample data: https://github.com/JetBrains/teamcity-kubernetes-plugin/blob/bccb09da6105b6f09710d8ed30884b5008eb576e/teamcity-kubernetes-plugin-server/src/test/java/jetbrains/buildServer/clouds/kubernetes/podSpec/CustomTemplatePodTemplateProviderTest.kt

So a valid pod spec for me was as follows:

apiVersion: v1
kind: Pod
metadata:
  name: teamcity-agent
  namespace: teamcity-agent
spec:
  containers:
  - name: jetbrains-teamcity-agent
    image: jetbrains/teamcity-agent
    imagePullPolicy: IfNotPresent
  nodeSelector:
    kubernetes.io/os: linux
    kubernetes.io/arch: "amd64"

The YAML is patched so you only need to specify what you want to override.

0

Please sign in to leave a comment.