Build Agent keeps restarting after upgrade request from server - 2022.10.2

Created February 23, 2023 16:29

I have recently completed an upgrade from Teamcity 2020.2.3 to 2022.04.4 with no issues. A few days later I decided to take it to 2022.10.2 using the same upgrade method used before. The server upgrades just and I can access the console but the agents appear to be stuck in a constant cycle of upgrading. The agent goes through the process of downloading the new build agent, installing, register and then gets an update command sent to it from the server and the cycle repeats again. This continues indefinitely if not interrupted. This behaviour was not seen at all with 2022.04.4.

[2023-02-23 16:26:48,928] INFO - ldServer.AGENT.PollingProtocol - New command is received from server "upgrade" {id = 82853} (enable debug to see command body)

11 comments

Paul Jennings

Created February 23, 2023 16:34

[2023-02-23 16:27:21,114]   INFO -    jetbrains.buildServer.AGENT - Build agent started
[2023-02-23 16:27:21,287]   INFO - buildServer.AGENT.registration - Server supports the following communication protocols: [polling]
[2023-02-23 16:27:21,288]   INFO - buildServer.AGENT.registration - Trying to register on server using 'polling' protocol.
[2023-02-23 16:27:21,375]   INFO - ldServer.AGENT.PollingProtocol - Start polling server for commands
[2023-02-23 16:27:21,378]   INFO - buildServer.AGENT.registration - Registered on server with id 4 and authorization token '#############'
[2023-02-23 16:27:21,378]   INFO - buildServer.AGENT.registration - If this is the first time this agent registered on the server make sure it is authorized by administrator in the server web UI.
[2023-02-23 16:27:21,387]   INFO - ldServer.AGENT.PollingProtocol - New command is received from server "testLocal" {id = 82852} (enable debug to see command body)
[2023-02-23 16:27:21,557]   INFO - r.artifacts.impl.HttpDiskCache - Cleaning up items with life time in cache greater than 172800 seconds
[2023-02-23 16:27:21,560]   INFO - r.artifacts.impl.HttpDiskCache - Finished cleaning up expired items, 0 items removed
[2023-02-23 16:27:21,610]   INFO - dDirectoryBasedCleanupRegistry - Removing files from /opt/teamcity/agent_1/work/.old
[2023-02-23 16:27:21,638]   INFO - ies.DirectoryMapUnknownCleaner - Checking not listed in directory.map folder /opt/teamcity/agent_1/work/temp.
[2023-02-23 16:27:21,900]   INFO - ldServer.AGENT.PollingProtocol - New command is received from server "upgrade" {id = 82853} (enable debug to see command body)
[2023-02-23 16:27:21,903]   INFO -    jetbrains.buildServer.AGENT - Upgrade call received from the build server
[2023-02-23 16:27:21,904]   INFO -    jetbrains.buildServer.AGENT - Will upgrade when become idle
[2023-02-23 16:27:21,904]   INFO -    jetbrains.buildServer.AGENT - Wait for build agent registration to finish
[2023-02-23 16:27:21,904]   INFO -    jetbrains.buildServer.AGENT - Registration of build agent is finished.
[2023-02-23 16:27:21,905]   INFO -    jetbrains.buildServer.AGENT - Stop command was not performed. No build to stop with reason: agent shutdown
[2023-02-23 16:27:21,905]   INFO -    jetbrains.buildServer.AGENT - Starting agent shutdown sequence, reason: Restart agent, failed to download upgrade from server
[2023-02-23 16:27:21,909]   INFO - gent.impl.DiskSpaceCleanerImpl - Try to ensure 3 GB free space under directory /opt/teamcity/agent_1 before upgrade. Currently available 22.68 GB
[2023-02-23 16:27:21,913]   INFO -    jetbrains.buildServer.AGENT - Downloading http://x.x.x.x:8111/update/teamcity-agent.xml ==> /opt/teamcity/agent_1/work/temp/fiH2qxUHSzbmAN0pbSlR0mAWH2jNwui4
[2023-02-23 16:27:21,919]   INFO - agent.impl.AgentPortFileWriter - Delete agent runtime file from /opt/teamcity/agent_1/logs/buildAgent.xmlRpcPort
[2023-02-23 16:27:21,921]   INFO -    jetbrains.buildServer.AGENT - Upgrade mode: jetbrains.buildServer.agent.impl.upgrade.modes.FullUpgradeMode
[2023-02-23 16:27:21,921]   INFO -    jetbrains.buildServer.AGENT - Downloading http://x.x.x.x:8111/update/plugins/agentSystemInfo.zip ==> /opt/teamcity/agent_1/update/plugins/agentSystemInfo.zip
[2023-02-23 16:27:21,921]   INFO - ldServer.AGENT.PollingProtocol - Stop polling server for commands
[2023-02-23 16:27:21,923]   INFO -    jetbrains.buildServer.AGENT - Unregistering from the server. Agent id: 4
[2023-02-23 16:27:21,923]   INFO -    jetbrains.buildServer.AGENT - Before unregistering from the server.

Anatoly Cherenkov

Created February 23, 2023 17:23

Hi! Typically it happens if the upgrade is unsuccessful. Please provide the teamcity-agent.log, upgrade.log, buildAgent.properties files from the agent, and the teamcity-server.log file from the server. Feel free to use https://uploads.jetbrains.com/ to upload the files privately.
Also specify how is the agent started (automatically or manually, which command and the arguments).

Paul Jennings

Created February 24, 2023 10:10

Hi Anatoly,

I have uploaded the files - the upload ID is 2023_02_24_edDUhfZSBPbGTwQrWspKRK.

upgrade.log
teamcity-agent.log
buildAgent.properties
teamcity-server.log
teamcity_agent_1

The agent was started manually using service teamcity_agent_1 start - an init.d wrapper to the usual agent.sh command. I have included that in the upload.

Thanks

Paul

Anatoly Cherenkov

Created February 27, 2023 13:58

The only thing that strikes the eye is that the tempDir property value is non-default in the in the buildAgent.properties file. Even though the upgrade should work even with the non-default tempDir, I'd recommend trying to change it from `tempDir=/opt/teamcity/agent_1/work/temp` to `tempDir=/opt/teamcity/agent_1/temp` or just `tempDir=../temp`. Please post the update about the results.

Paul Jennings

Created February 27, 2023 15:22

Afternoon Anatoly,

No joy unfortunately. Config amended as follows

# grep temp /opt/teamcity/agent_1/conf/buildAgent.properties
## Container directory for the temporary directories.
tempDir=/opt/teamcity/agent_1/temp

Fresh set of logs uploaded as 2023_02_27_zCHAM7qRNzoj8uiBabpBH7.

Agent remains stcuk in an 'upgrade' cycle.

Anatoly Cherenkov

Created February 27, 2023 15:32

Hi Paul. The last entry in the logs is dated three days ago, and the tempDir is still non default there. Please verify if both `tempDir=/opt/teamcity/agent_1/temp` and `tempDir=../temp` values yield the same result and upload the evidence.

Paul Jennings

Created February 27, 2023 16:57

Ah yes, spectacular failure on my part there selecting the old logs.

Try again with 2023_02_27_29JiRGbq8tHehFvY3cGouU.

Two sets here - the first from 15:15 with the correct logs and config pathed to /opt/teamcity/agent_1/temp and the 2nd from after 16:15 with config set to ../temp.

Many thanks

Paul

Anatoly Cherenkov

Created March 03, 2023 15:14

Sorry about the delay. I couldn't pinpoint the issue, but I would recommend trying it with the following values in the buildAgent.properties files. Please provide the new teamcity-agent.log and upgrade.log files if it still doesn't work.

```
workDir=../work
tempDir=../temp
systemDir=../system
```

Paul Jennings

Created March 08, 2023 16:41

Afternoon Anatoly,

What magic is this that you have proposed! I made the amendment and the agent registered straight away. I did the same for agent 2 and again it registered ok. I tried agent 3 with the original values and it failed to register.

Logs uploaded to 2023_03_08_21prBWperFne5M81642ndr

So it would seem it did not like a full path for the workDir, tempDir and systemDir. We previously tried amending tempDir so I presume the change to systemDir or workDir did the trick.

How very strange but thank you.

Kind regards

Paul

Anatoly Cherenkov

Created March 15, 2023 18:20

Sorry about the delay. It looks like there is a problem with loading the plugins (see log the snippet below). Please check if the files /opt/teamcity/agent_1/plugins/agentSystemInfo.zip or /opt/teamcity/agent_1/plugins/agentSystemInfo/lib/agentSystemInfo.jar exist on the agent.

[2023-03-08 16:26:30,421]   INFO - rver.plugins.PluginManagerImpl - Plugins initialization completed (56 plugins loaded): [agentSystemInfo, amazonEC2, ant, ant-net-tasks, antPlugin, aws-core-agent, cloud-vmware-agent, commandLineRunner, coveragePlugin, crashDetector, deploy-runner-agent, docker-support, dotnet, dotnetPlugin, dotNetRunners, duplicatePlugin, environment-fetcher, file-content-replacer, fxcop, gant, gantPlugin, golang-agent, gradle-runner, idea-runner, inspectionPlugin, java-dowser, jetbrains.git, jps, junitPlugin, jvm-update, kotlin-script-runner-agent-117025, mavenPlugin, mercurial, meta-runner, nodejs-agent-117025, nuget-agent, perfmon-agent, perforce-agent, plugin-agent, python-agent-117025, qodana, rake-runner, remoteAccess, s3-artifact-storage-agent, sbt-runner-agent, ssh-manager, stacktracesPlugin, svnAgent, swabra, teamcity-kubernetes-plugin-agent, teamcity-parallel-tests-agent, testNGPlugin, tfs-agent, visualstudiotest, xcode-runner, xml-report-plugin]
[2023-03-08 16:26:30,423]   INFO - rver.plugins.PluginManagerImpl - ===========================================================
[2023-03-08 16:26:30,428]   WARN - ent.plugins.AgentPluginsLoader - Plugin agentSystemInfo.zip was not found 
...
<all agent-side plugins are listed here>
... 
[2023-03-08 16:26:30,433]   WARN - ent.plugins.AgentPluginsLoader - Plugin xml-report-plugin.zip was not found 
[2023-03-08 16:26:30,433]   WARN - ent.plugins.AgentPluginsLoader - Some plugins are missing. Agent will upgrade again

Paul Jennings

Created March 21, 2023 14:42

Afternoon Anatoly

Following up on your last comment the jar was in the expected place but the zip is in a slightly different place - see below.

[root@teamcityserver-clone teamcity]# locate agentSystemInfo.zip
/opt/teamcity/TeamCity/webapps/ROOT/WEB-INF/plugins/agentSystemInfo.zip
/opt/teamcity/TeamCity/webapps/ROOT/WEB-INF/plugins/.unpacked/agentSystemInfo/agent/agentSystemInfo.zip

[root@teamcityserver-clone teamcity]# locate agentSystemInfo.jar
/opt/teamcity/agent_1/plugins/agentSystemInfo/lib/agentSystemInfo.jar
/opt/teamcity/agent_2/plugins/agentSystemInfo/lib/agentSystemInfo.jar
/opt/teamcity/agent_3/plugins/agentSystemInfo/lib/agentSystemInfo.jar

As I mentioned in my previous comment however the upgrade is successful now following the use of relative paths in buildAgent.properties.

workDir=../work
tempDir=../temp
systemDir=../system

Kind regards

Paul

Please sign in to leave a comment.