Failed to download upgrade from server on agents after adding handle.exe tool

Hello,

I just recently added the handle.exe tool via the TeamCity UI. The upload to the server and packaging of the tool to a zip seems to work fine. The server also sends the agent upgrade notification to all the agents.

The problem is when the agents try to install the upgrade. I get a "failed to download upgrade from server" error in the agent logs so the upgrade does not proceed therefore the handle.exe tool is not installed.

I do see an error in the launcher.log when the upgrade process stops the agent. Not sure if this is the problem.

There is no logging in the upgrade.log. Seems like it never gets that far to log in that log.

We have added custom tools before in the past and the agents were successfully upgraded.

We are on 2020.2 (build 85487).

Any help would be appreciated!

launcher.log:
[2021-04-30 17:37:09,102]   INFO - dServer.agent.run.AgentProcess - Agent process finished 
[2021-04-30 17:37:09,102]   INFO - ldServer.agent.run.AgentRunner - Agent has exited with code: 23 
[2021-04-30 17:37:09,102]  ERROR - ldServer.agent.run.AgentRunner - Too long run time, unexpected exit. Restarting. Run time was : 4620.139 sec. 
[2021-04-30 17:37:09,102]   INFO - ldServer.agent.run.AgentRunner - Launcher is restarting agent process. 
[2021-04-30 17:37:09,102]   INFO - dServer.agent.run.AgentProcess - Fri Apr 30 17:37:09 UTC 2021 ==> Start launch 
[2021-04-30 17:37:09,102]  DEBUG - .buildServer.agent.JavaChooser - Agent Java runtime version is 1.8 
[2021-04-30 17:37:09,102]   INFO - dServer.agent.run.AgentProcess - Launching agent with command:  C:\Program Files\Amazon Corretto\jdk1.8.0_242\jre\bin\java.exe -ea -Xmx1024m -Dlog4j.configuration=file:../conf/teamcity-agent-log4j.xml -Dteamcity_logs=../logs/ -classpath C:\BuildAgent\lib\agent-configurator.jar;C:\BuildAgent\lib\agent-installer-ui.jar;C:\BuildAgent\lib\agent-launcher.jar;C:\BuildAgent\lib\agent-openapi.jar;C:\BuildAgent\lib\agent-upgrade.jar;C:\BuildAgent\lib\agent.jar;C:\BuildAgent\lib\annotations.jar;C:\BuildAgent\lib\app-wrapper.jar;C:\BuildAgent\lib\buildAgent-updates-applying.jar;C:\BuildAgent\lib\cloud-shared.jar;C:\BuildAgent\lib\common-impl.jar;C:\BuildAgent\lib\common-runtime.jar;C:\BuildAgent\lib\common.jar;C:\BuildAgent\lib\commons-beanutils-core.jar;C:\BuildAgent\lib\commons-codec.jar;C:\BuildAgent\lib\commons-collections-3.2.2.jar;C:\BuildAgent\lib\commons-compress-1.9.jar;C:\BuildAgent\lib\commons-httpclient-3.1.jar;C:\BuildAgent\lib\commons-io-1.3.2.jar;C:\BuildAgent\lib\commons-logging.jar;C:\BuildAgent\lib\coverage-agent-common.jar;C:\BuildAgent\lib\coverage-report.jar;C:\BuildAgent\lib\duplicator-util.jar;C:\BuildAgent\lib\ehcache-1.6.0-patch.jar;C:\BuildAgent\lib\ehcache-1.7.2.jar;C:\BuildAgent\lib\freemarker.jar;C:\BuildAgent\lib\gson.jar;C:\BuildAgent\lib\idea-settings.jar;C:\BuildAgent\lib\inspections-util.jar;C:\BuildAgent\lib\jaxen-1.1.1.jar;C:\BuildAgent\lib\jdk-searcher.jar;C:\BuildAgent\lib\jdom.jar;C:\BuildAgent\lib\joda-time.jar;C:\BuildAgent\lib\launcher-api.jar;C:\BuildAgent\lib\launcher.jar;C:\BuildAgent\lib\log4j-1.2.12-json-layout.jar;C:\BuildAgent\lib\log4j-1.2.12.jar;C:\BuildAgent\lib\messages.jar;C:\BuildAgent\lib\nuget-utils.jar;C:\BuildAgent\lib\openapi.jar;C:\BuildAgent\lib\patches-impl.jar;C:\BuildAgent\lib\patches.jar;C:\BuildAgent\lib\processesTerminator.jar;C:\BuildAgent\lib\resources_en.jar;C:\BuildAgent\lib\runtime-util.jar;C:\BuildAgent\lib\server-logging.jar;C:\BuildAgent\lib\serviceMessages.jar;C:\BuildAgent\lib\slf4j-api-1.7.5.jar;C:\BuildAgent\lib\slf4j-log4j12-1.7.5.jar;C:\BuildAgent\lib\spring-scripting\spring-scripting-bsh.jar;C:\BuildAgent\lib\spring-scripting\spring-scripting-groovy.jar;C:\BuildAgent\lib\spring-scripting\spring-scripting-jruby.jar;C:\BuildAgent\lib\spring.jar;C:\BuildAgent\lib\trove-3.0.3.jar;C:\BuildAgent\lib\trove4j.jar;C:\BuildAgent\lib\util.jar;C:\BuildAgent\lib\xercesImpl.jar;C:\BuildAgent\lib\xml-rpc-wrapper.jar;C:\BuildAgent\lib\xmlrpc-2.0.1.jar;C:\BuildAgent\lib\xpp3-1.1.4c.jar;C:\BuildAgent\lib\xstream-1.4.11.1-custom.jar;C:\BuildAgent\lib\xz-1.5.jar jetbrains.buildServer.agent.AgentMain -file ..\conf\buildAgent.properties -launcher.version 85487 

teamcity-agent.log:
[2021-04-30 12:22:59,974]   INFO - ldServer.AGENT.PollingProtocol - New command is received from server "upgrade" {id = 12} (enable debug to see command body)
[2021-04-30 12:22:59,974]   INFO -    jetbrains.buildServer.AGENT - Upgrade call received from the build server
[2021-04-30 12:22:59,974]   INFO -    jetbrains.buildServer.AGENT - Will upgrade when become idle
[2021-04-30 12:22:59,974]   INFO -    jetbrains.buildServer.AGENT - Wait for build agent registration to finish
[2021-04-30 12:22:59,974]   INFO -    jetbrains.buildServer.AGENT - Registration of build agent is finished.
[2021-04-30 12:22:59,974]   INFO -    jetbrains.buildServer.AGENT - Stop command was not performed. No build to stop with reason: agent shutdown
[2021-04-30 12:22:59,974]   INFO -    jetbrains.buildServer.AGENT - Starting agent shutdown sequence, reason: Restart agent, failed to download upgrade from server
[2021-04-30 12:22:59,990]   INFO - gent.impl.DiskSpaceCleanerImpl - Try to ensure 3 GB free space under directory C:\BuildAgent before upgrade. Currently available 59.53 GB
[2021-04-30 12:22:59,990]   INFO - agent.impl.AgentPortFileWriter - Delete agent runtime file from C:\BuildAgent\logs\buildAgent.xmlRpcPort
[2021-04-30 12:22:59,990]   INFO -    jetbrains.buildServer.AGENT - Unregistering from the server. Agent id: 992
[2021-04-30 12:22:59,990]   INFO -    jetbrains.buildServer.AGENT - Downloading https://teamcity-XXX/update/teamcity-agent.xml ==> D:\TC\temp\HAy87gOlK5N8rBE4E11vKrgmfAKyWsfm
[2021-04-30 12:23:00,021]   INFO -    jetbrains.buildServer.AGENT - Upgrade mode: jetbrains.buildServer.agent.impl.upgrade.modes.PluginsOnlyUpgradeMode
[2021-04-30 12:23:00,021]   INFO - rocesses.ProcessTreeTerminator - Using jetbrains.buildServer.processes.ProcessTreeTerminatorWindows
[2021-04-30 12:23:00,021]   INFO - .ProcessTreeTerminatorImplBase - Collecting processes from PIDs current process
[2021-04-30 12:23:00,037]   INFO - .ProcessTreeTerminatorImplBase - Will use command 'D:\TC\temp\globalTmp\TeamCity8592383162386991471ps\JetBrains.TeamCity.ps.x64.exe ps'.
[2021-04-30 12:23:00,193]   INFO - .ProcessTreeTerminatorImplBase - Current process PID 516
[2021-04-30 12:23:00,193]   INFO - .ProcessTreeTerminatorImplBase - No processes to kill
[2021-04-30 12:23:00,193]   INFO -    jetbrains.buildServer.AGENT - Exit for upgrade
[2021-04-30 12:23:00,193]   INFO - ent.impl.upgrade.AgentExitCode - Agent exited. Restart agent, some plugins needs to be updated
[2021-04-30 12:23:00,193]   INFO - buildServer.agent.AgentMain2$3 - Closing jetbrains.buildServer.agent.AgentMain2$3@53f65459: startup date [Fri Apr 30 04:04:28 UTC 2021]; root of context hierarchy
[2021-04-30 12:23:02,043]   INFO -   jetbrains.buildServer.SERVER - Starting TeamCity agent