TeamCity agent cannot upgrade because of exception occurred
I have teamcity server running on Windows and I try to start agent on Ubuntu (version 22.04) in docker by instruction: https://hub.docker.com/r/jetbrains/teamcity-agent/.
I used the simple command:
docker run -e SERVER_URL="..." -v /home/buildAgent:/data/teamcity_agent/conf jetbrains/teamcity-agentbut I received error: JAVA not found. I installed JAVA. Now I use the next command:
docker run -e JAVA_HOME=/opt/java/openjdk/bin/java -e TEAMCITY_JRE=/opt/java/openjdk/bin/java -e SERVER_URL="..." -v /home/buildAgent:/data/teamcity_agent/conf -d --name agent jetbrains/teamcity-agentand agent cannot upgrade. Agent logs (last lines):
[2024-03-06 11:14:20,358] INFO - .ProcessTreeTerminatorImplBase - Collecting processes from PIDs current process
[2024-03-06 11:14:20,359] INFO - .ProcessTreeTerminatorImplBase - Will use command 'sh -c echo $$ && ps awwxo pid,ppid,command | tee'.
[2024-03-06 11:14:20,389] INFO - .ProcessTreeTerminatorImplBase - Will use command 'sh -c echo $$ && ps awwxo pid,ppid,command | tee'.
[2024-03-06 11:14:20,420] INFO - ses.ProcessTreeTerminatorLinux - Second thread id is 321
[2024-03-06 11:14:20,421] INFO - ses.ProcessTreeTerminatorLinux - Thread is Process thread model: false
[2024-03-06 11:14:20,421] INFO - .ProcessTreeTerminatorImplBase - Current process PID 321
[2024-03-06 11:14:20,422] INFO - .ProcessTreeTerminatorImplBase - No processes to kill
[2024-03-06 11:14:20,422] INFO - jetbrains.buildServer.AGENT - Exit for upgrade
[2024-03-06 11:14:20,423] INFO - ent.impl.upgrade.AgentExitCode - Agent exited. Upgrade processLogs of upgrade.log file with exception (last lines):
[2024-03-06 11:14:25,509] INFO - ver.agent.upgrade.AgentUpgrade - Finish upgrade at 2024-03-06 11:14:25.509
[2024-03-06 11:14:25,509] INFO - r.agent.upgrade.UpgradeRunBase - Run command line (arguments separated with '|'): /bin/sh|./agent.sh|start| (at /opt/buildagent/bin)
[2024-03-06 11:14:25,514] INFO - .agent.run.ProcessOutputLogger - Starting TeamCity build agent...
[2024-03-06 11:14:25,970] INFO - .agent.run.ProcessOutputLogger - Java not found. Cannot start TeamCity agent. Please ensure JDK or JRE is installed and JAVA_HOME environment variable points to it.
[2024-03-06 11:14:25,970] INFO - r.agent.upgrade.UpgradeRunBase - Process exited with code: 1
[2024-03-06 11:14:25,971] ERROR - r.agent.upgrade.UpgradeRunBase - Failed to start upgrade process. Return code was 1 is not equal to zero.
[2024-03-06 11:14:25,970] WARN - .agent.run.ProcessOutputLogger -
[2024-03-06 11:14:25,972] ERROR - r.agent.upgrade.UpgradeRunBase - Failed to start agent after upgrade: java.io.IOException: Failed to start upgrade process. Return code was 1 is not equal to zero.
java.io.IOException: Failed to start upgrade process. Return code was 1 is not equal to zero.
at jetbrains.buildServer.agent.upgrade.UpgradeRunBase.runCmdLine(UpgradeRunBase.java:104) ~[launcher.jar:117072]
at jetbrains.buildServer.agent.upgrade.RunShUpgrade.startAgain(RunShUpgrade.java:36) ~[launcher.jar:117072]
at jetbrains.buildServer.agent.upgrade.UpgradeRunBase.run(UpgradeRunBase.java:51) ~[launcher.jar:117072]
at jetbrains.buildServer.agent.upgrade.UpgradeMode$5.run(UpgradeMode.java:133) ~[launcher.jar:117072]
at jetbrains.buildServer.agent.upgrade.Upgrade2$1.lockHeld(Upgrade2.java:57) ~[launcher.jar:117072]
at jetbrains.buildServer.agent.upgrade.Upgrade2$1.lockHeld(Upgrade2.java:55) ~[launcher.jar:117072]
at jetbrains.buildServer.agent.lock.impl.misc.MockLock.assumeLocked(MockLock.java:22) ~[launcher.jar:117072]
at jetbrains.buildServer.agent.lock.impl.misc.MockLock.tryLock(MockLock.java:14) ~[launcher.jar:117072]
at jetbrains.buildServer.agent.upgrade.Upgrade2.main3(Upgrade2.java:54) ~[launcher.jar:117072]
at jetbrains.buildServer.agent.upgrade.Upgrade2.main2(Upgrade2.java:27) ~[launcher.jar:117072]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
at jetbrains.buildServer.agent.ClazzLoader.call(ClazzLoader.java:59) ~[launcher.jar:117072]
at jetbrains.buildServer.agent.ClazzLoader.callMain2(ClazzLoader.java:19) ~[launcher.jar:117072]
at jetbrains.buildServer.agent.upgrade.Upgrade.main(Upgrade.java:16) ~[launcher.jar:117072]I don't understand why Java not found because I set JAVA_HOME variable. Could anybody help?
Please sign in to leave a comment.
There is no need to set environment variables for the Docker container.
Could you provide the full output when you start the container with a command from the documentation? This command should be enough to start the container and connect the build agent with the server, so I need to check at what point of the startup the error is encountered.
Best regards,
Anton
Ok, I have started container with documentation command.
Full output:
Full upgrade.log from /opt/buildagent/logs:
I assume that this is not a full upgrade.log, as it ends in the middle of the line. Could you attach the complete file or send the remaining part of the logs?
Best regards,
Anton
No, it's full upgrade.log, I rechecked.
The last messages:
2 points are unclear to me:
1. Why does the TeamCity agent Docker image try to upgrade itself? I tested it in my environment, and no upgrade was required after pulling the latest image from the Docker repository.
2. Why did it lose Java, as it was clearly found during the initial startup (as per the command output)?
As I can't reproduce it (the upgrade is not required on my side), I assume there may be something wrong with the local Docker image. Please try `docker images | grep jetbrains/teamcity-agent` to see what versions you have and then clean them up. Or, you can remove all of the images named jetbrains/teamcity-agent by running `docker rmi -f $(docker images -q jetbrains/teamcity-agent)`. After cleaning up the local images, try running the container again and see if the upgrade process will be started.
Best regards,
Anton
I run command ‘docker images | grep jetbrains/teamcity-agent’:
Looks like it's latest version.
I removed this images by using your command and restart container. Upgrade also failed. Logs are the same.
1. After the upgrade failed, run docker inspect for this container and share the output.
2. After the upgrade failed, check that the /opt/java/openjdk/bin/java file exists in the container.
3. As a possible cause of the upgrade, please check if you have any custom plugins added to the TeamCity server (if there are custom plugins which are not available in the agent by default, the agent will start upgrading to install the plugin).
4. Please check that you are using all the recent versions of the plugins. If there is an old version of the plugin used in the TeamCity server, the agent will start upgrading to install this version of the plugin.
Best regards,
Anton
I checked the plugins - we have only JetBrains plugins (latest versions), no custom.
File /opt/java/openjdk/bin/java exists in running container.
docker inspect output:
Best regards,
Anton
I was in contact with the Docker images maintainer, and we ran multiple tests but still couldn't reproduce the issue on our side. Since the new version of TeamCity is out, I would like to ask you to upgrade the TeamCity server to the latest version before proceeding. Regarding the tools, it seems that an upgrade is forced on the agent due to non-bundled tools.
As indicated by the `upgrade.log`, the upgrade fails because it cannot find the Java version. We have JDK 17 (`Corretto-17.0.7.7.1`) pre-installed within our Docker images, and the upgrade should not affect its presence. As noted, there is no need to set the `TEAMCITY_JRE` environment variable unless a custom image is created.
At the same time, looking at the attached log, we could see the `JAVA_HOME` variable was not found:
[2024-03-19 06:22:23,028] INFO - .agent.run.ProcessOutputLogger - Java not found. Cannot start TeamCity agent. Please ensure JDK or JRE is installed and JAVA_HOME environment variable points to it.
To conduct a thorough investigation and fully understand the issue, may I kindly request that you share the following details?
1. For investigation purposes, please use a specific tag instead of the `latest` manifest:
`jetbrains/teamcity-agent:2024.03-linux` - Linux AMD64;
`jetbrains/teamcity-agent:2024.03-linux-arm64` - Linux ARM64;
I've added 2024.03 Linux-based agent images as an example. Please note more images (versions & OS) can be found in our Dockerhub (https://hub.docker.com/r/jetbrains/teamcity-agent/tags).
2. Configuration directory & property files. In theory, environment variables, including `JAVA_HOME`, can be overwritten by agent configuration. To exclude this cause, please (1) ensure `/home/buildagent` (volume directory on the host) is empty before the container starts, (2) the property file inside the container does not have any JDK-related custom `env.`-prefix property:
# Inside TeamCity Agent Container
cat /opt/buildagent/conf/buildAgent.properties | grep "env."
3. Environment variables - Docker. Please ensure the `JAVA_HOME` environment variable is set correctly upon the container's start-up and after the upgrade.
# Inside TeamCity Agent Container
echo $JAVA_HOME
4. Agent start-up after the upgrade. The logs show that the agent cannot start. It seems that the problem is with finding the Java installation, as indicated in the `upgrade.log`. Please try running `agent.sh start` manually within the container and, if possible, share the output with us - this would help us understand the issue better.
# Inside TeamCity Agent Container
$ cd /opt/buildagent && ./bin/agent.sh start
Best regards,
Anton