After agent jre upgrade, agent fails downloading teamcity-agent.xml and can't upgrade itself.
I upgraded a TeamCity agent in order to resolve the "outdated jre" error we'd been getting. I did it by downloading an agent installer exe from our server instance, and running that installer (I'm on windows). However since then, the agent is unable to transition to "connected" status.
The agent's log shows 403 errors. First when downloading teamcity-agent.xml, then it tries to download buildAgent.zip. In both cases it results in a 403. However it can reach the server. I know that because the logs shows `Registering on server via URL https://dev.fpweb.net...` and it seems to be successful. Also the agent's page on the server shows this agent's last communication as just a few minutes ago. The agent tries to get those files I mentioned, a number of times, and then ultimately restarts itself. This effectively has it in an infinite loop, which explains why the last communication date is always very recent.
I'm not sure how to troubleshoot this. I used an unauthenticated browser, on that machine, to download the xml file with no problem.
Here's the relevant sections of the log.
```
[2017-06-23 18:37:52,334] INFO - buildServer.AGENT.registration - Registering on server via URL https://dev.fpweb.net: AgentDetails{Name='fpwebwfe01', AgentId=null, BuildId=null, AgentOwnAddress='null', AlternativeAddresses=[172.27.0.214, 172.29.9.214, 172.29.9.243, 172.29.9.244, 172.29.9.245], Port=9090, Version='46654', PluginsVersion='NA', AvailableRunners=[Ant, cargo-deploy-runner, dotnet-tools-dupfinder, dotnet-tools-inspectcode, Duplicator, ftp-deploy-runner, FxCop, gradle-runner, Inspection, jb.nuget.installer, jb.nuget.pack, jb.nuget.publish, jetbrains.dotNetGenericRunner, jetbrains.mspec, jetbrains_powershell, JPS, Maven2, MSBuild, NAnt, NUnit, rake-runner, SBT, simpleRunner, sln2003, smb-deploy-runner, ssh-deploy-runner, ssh-exec-runner, VisualStudioTest, VS.Solution], AvailableVcs=[tfs, cvs, jetbrains.git, mercurial, svn, perforce], AuthorizationToken='c7278eb23198bcec8a56313bb791b0ae', PingCode='rSPtmEBWdkLfZSK3G0M67H0HEuUB4lyH'}
[2017-06-23 18:37:53,833] INFO - jetbrains.buildServer.AGENT - Shutdown agent WebServer finish
[2017-06-23 18:37:53,833] INFO - jetbrains.buildServer.AGENT - Shutdown agent finish
[2017-06-23 18:37:54,016] INFO - jetbrains.buildServer.AGENT - Downloading https://dev.fpweb.net/update/teamcity-agent.xml ==> C:\BuildAgent\temp\USBPJYbCRkDUFqziiMH79KJyWJCJQKR2
[2017-06-23 18:37:54,522] INFO - jetbrains.buildServer.AGENT - Downloading https://dev.fpweb.net/update/teamcity-agent.xml ==> C:\BuildAgent\temp\USBPJYbCRkDUFqziiMH79KJyWJCJQKR2
[2017-06-23 18:37:55,029] WARN - ade.UpgradeFilesDownloaderImpl - Failed to download AgentUpdateInfo from server. Server returned [403] Forbidden for https://dev.fpweb.net/update/teamcity-agent.xml
jetbrains.buildServer.agent.impl.upgrade.UpgradeFailedException: Server returned [403] Forbidden for https://dev.fpweb.net/update/teamcity-agent.xml
at jetbrains.buildServer.agent.impl.upgrade.HttpFilesDownloaderImpl$DownloaderImpl.downloadFile(HttpFilesDownloaderImpl.java:76)
at jetbrains.buildServer.agent.impl.upgrade.HttpFilesDownloaderImpl.downloadFile(HttpFilesDownloaderImpl.java:33)
at jetbrains.buildServer.agent.impl.upgrade.RecoverableFilesDownloaderImpl.downloadFile(RecoverableFilesDownloaderImpl.java:33)
at jetbrains.buildServer.agent.impl.upgrade.UpgradeFilesDownloaderImpl.downloadContentsXmlFile(UpgradeFilesDownloaderImpl.java:60)
at jetbrains.buildServer.agent.impl.upgrade.UpgradesProcessor.downloadUpdates(UpgradesProcessor.java:42)
at jetbrains.buildServer.agent.impl.upgrade.UpgradeThread.downloadUpdates(UpgradeThread.java:104)
at jetbrains.buildServer.agent.impl.upgrade.UpgradeThread.prepareUpgrade(UpgradeThread.java:72)
at jetbrains.buildServer.agent.impl.upgrade.UpgradeThread.run(UpgradeThread.java:57)
at java.lang.Thread.run(Thread.java:745)
[2017-06-23 18:37:55,032] INFO - ade.modes.OldServerUpgradeMode - Server does not provide teamcity-agent.xml. Will download only buildAgent.zip.
[2017-06-23 18:37:55,032] INFO - jetbrains.buildServer.AGENT - Downloading https://dev.fpweb.net/update/buildAgent.zip ==> C:\BuildAgent\update\buildAgent.zip
[2017-06-23 18:37:55,539] INFO - jetbrains.buildServer.AGENT - Downloading https://dev.fpweb.net/update/buildAgent.zip ==> C:\BuildAgent\update\buildAgent.zip
[2017-06-23 18:37:56,044] INFO - jetbrains.buildServer.AGENT - Downloading https://dev.fpweb.net/update/buildAgent.zip ==> C:\BuildAgent\update\buildAgent.zip
[2017-06-23 18:37:56,550] INFO - jetbrains.buildServer.AGENT - Failed to download and prepare upgrade. Server returned [403] Forbidden for https://dev.fpweb.net/update/buildAgent.zip
[2017-06-23 18:37:56,551] INFO - jetbrains.buildServer.AGENT - Downloading https://dev.fpweb.net/update/teamcity-agent.xml ==> C:\BuildAgent\temp\8RfPiUQNTCM1CR6nGCcivZfgI4SIMggm
```
Please sign in to leave a comment.
Hi, do you use a reverse proxy or smth like this in front of the TeamCity server?
IIS receives the http requests and does a redirect to https. The https traffic goes directly to the TeamCity Server.
Are there any traces of these requests in catalina.yyyy-MM-dd.log, localhost.yyyy-MM-dd.log, teamcity-auth.log.* and teamcity-server.log.*?
No traces that I can see. I downloaded all the logs, and did a full text search for `teamcity-agent.xml` and `buildAgent.zip`.
Can I manually place one of these files on the agent perhaps?
Unfortunately, no. I still think there's an issue with proxying. Did you configure TeamCity tomcat to work with SSL or traffic is proxied by IIS anyway?
What if you setup an additional (non-SSL) connector in Tomcat and point agent to it directly?
I configured TeamCity tomcat to work with ssl (see an earlier message I posted about the server.xml getting blown away on me during upgrades).
I specifically disabled non-ssl traffic as this is a public facing server. I'll try enabling it long enough for the agent to get this update I suppose.
Can you hit the url? https://dev.fpweb.net/update/teamcity-agent.xml I can from this pc I am on now, and I used an unauthenticated browser. I would think that fact would mean any sort of proxying issue is off the table. Additionally, the other https requests from agent to server get through.
Thanks, please try the following command. :
curl -i -H "User-Agent: Jakarta Commons-HttpClient/3.1" -H "Host: dev.fpweb.net" https://dev.fpweb.net/update/teamcity-agent.xml
I get: The owner of this website (dev.fpweb.net) has banned your access based on your browser's signature (377267cfc85b764e-ua19).
Looks like we might have a smoking gun right there. I need to figure out how to map that call to the `curl` that comes with powershell in order to duplicate, but this is good. Thanks, I'll let you know what I find.
Hi Sergey,
I'm back working on this problem now that I've returned from holiday. I was able to duplicate the result you saw (that is, a 403) by running a curl equivalent in my machine. I confirmed that this error is being served up by TeamCity server (tomcat?) by shutting down IIS and running curl again. I continued to get the 403.
I hope you agree that whatever fix it is, it needs to happen in the TeamCity web server. Or perhaps the agent needs to use a different user-agent string. Any thoughts on this?
Thanks,
Josh
Hi, there's actually nothing to do with TeamCity on this. You just need to configure your firewall properly. This looks similar: https://support.cloudflare.com/hc/en-us/articles/200171806-Error-1010-The-owner-of-this-website-has-banned-your-access-based-on-your-browser-s-signature
Ah I see. This would explain why the local agent (running on the same box as the server) doesn't experience this problem. I'll run this lead down and let you know how it goes.
Thanks again!