Agents not using buildAgent.properties config params
We recently changed the serverUrl for our teamcity server from teamcity.oldDomain.com to teamcity.newDomain.com. We had a redirect in place for a while, but shut it off now that all our users are used to the new url. However, the agents don't seem to agree. They continue to log this in teamcity-agent.log:
[2018-04-18 17:24:07,060] INFO - .agent.AmazonPropertiesUpdater - TeamCity Build Agent was started by the TeamCity Server in Amazon EC2.
Server URL: http://teamcity.olddomain.com.
Proposed agent name: <will be set on connection to the server>
Cloud profile: profile 'TeamCity AWS Agent Cloud'{id=amazon-1, projectId=_Root}
Idle time to shutdown: 100 minutes
CustomParameters: {
cloud.amazon.agent-name-prefix = EC2
system.cloud.profile_id = amazon-1
teamcity.cloud.agent.remove.policy = unauthorize
teamcity.cloud.instance.hash = 1234569sdjflkdf
}
And then:
[2018-04-18 17:24:09,778] INFO - buildServer.AGENT.registration - Registering on server via URL http://teamcity.oldDomain.com: AgentDetails{Name='EC2-i-1asdfadsfasdf', AgentId=null, BuildId=null, AgentOwnAddress='null', AlternativeAddresses=[(AGENT IP ADDRESS)], Port=9090, Version='51047', PluginsVersion='51047-md5-asdfadfadsfasd', AvailableRunners=[Ant, cargo-deploy-runner, CompressFiles, dotnet-tools-dupfinder, dotnet-tools-inspectcode, dotnet.cli, Duplicator, ftp-deploy-runner, FxCop, gradle-runner, Inspection, jb.nuget.installer, jb.nuget.pack, jb.nuget.publish, jetbrains.dotNetGenericRunner, jetbrains.mspec, jetbrains_powershell, jonnyzzz.grunt, jonnyzzz.gulp, jonnyzzz.node, jonnyzzz.npm, jonnyzzz.nvm, jonnyzzz.phantom, JPS, Maven2, MSBuild, NAnt, NUnit, octopus.create.release, octopus.deploy.release, octopus.promote.release, octopus.push.package, rake-runner, SBT, simpleRunner, sln2003, smb-deploy-runner, sonar-plugin, ssh-deploy-runner, ssh-exec-runner, VisualStudioTest, VS.Solution, xUnitRunner], AvailableVcs=[tfs, cvs, jetbrains.git, mercurial, svn, perforce], AuthorizationToken='asdfasdfdsfa', PingCode='adfafadfadsf'}
The buildAgent.properties file:
serverUrl=teamcity.newDomain.io
name=EC2-i-12314tdsgfdsgdg
workDir=../work
tempDir=../temp
systemDir=../system
ownPort=9090
authorizationToken=asdfj124390udsflkjf
ownAddress=(AGENT IP ADDRESS)
Teamcity Global Settings has the serverUrl overridden with the "newDomain" url as a secondary effort to force the correct behaviour. The cloud config sections have no mention of the old url.
We have tried reinstalling the agent and reconfiguring, still sees the old url. The Teamcity config XML has zero mentions of the old domain.
Kinda at our wits end, anywhere else to look? Our next steps are really to blow away all the agents and agent config and recreate. Which is very unfortunate.
Please sign in to leave a comment.
Hi, if you are using TC EC2 integration, then agents started using this integration receive server address from TC server.
So, you need to go to Administration->Global Settings and change Server URL here.
I played around, did two experiments:
So we'll see how that goes.
Nope, both of these, no change. Sometimes works, sometimes doesn't. Maybe the config value is in the database or something?
We're on the right track now. I realized you could enable Debug logging on the agent and found this line:
I decoded the instance's userdata, and sure enough, that's where the URL is stored!
So the issue is, why is it not self-updating the userdata?
After manually updating (stop agent EC2 instance, copy old userdata, decode, fix URL, encode, update userdata, start agent EC2 instance), I think that has fixed it.
Could you kindly let me know:
Thanks.
Update on this. The issue has returned, guess the userdata stuff is being cached on the agent then repopulated on shut down.
Really need some kinda answer on this please...
Any guidance at all?
Is this the wrong place to be logging issues like this? My understanding with the paid license packs is that the forum is how we get support. We're now on the third business day with no reply.
Hi Chris,
sorry for the delay, we've been having quite a bit of workload lately and we're catching up still with all requests. We are investigating this a bit deeper and will report back as soon as we have something.
Thanks for the response, appreciate it. If you want me to open a YouTrack issue, let me know.
Regarding Sergey's comment, we have already done this, it does not impact the EC2 userdata. Neither does configuring the serverUrl in the buildAgent.properties file. Hence the problem.
User data is injected into VM when TC starts it. If you are using instance that was started by TC at a certain point, then you need to remove the user-data from instance manually to prevent agent from using this data (and serverUrl from it) to connect to server.
Sergey, when I attempt to modify the user-data, it just gets repopulated with the old user-data (with bad url).
So, you have already set Server URL under Administration->Global Settings and TC still puts the wrong value into user-data?
BTW, user-data is Base64 encoded XML, so you can decode it and see what server injects into VM
Yes. Both the global settings and buildAgent.properties have the correct URL, but TC keeps resetting the user-data back to the incorrect URL.
Here is a test I did:
But later...
Another test:
Driving me insane.
Please also check for the presence of teamcity.ec2.server.internal.address in the internal.properties file
There is only teamcity.ec2.instance.types set in that file.
If you are on 2017.2, then Cloud Profile can also contain the server URL. Can you also check that?
Yes, it contains the correct URL (actually typed in, not just using the default). Originally this was blank, as it purports to use the default URL.
that's very weird. The old URL must be somewhere in configs. Can you do a full-text search against TC data directory for presence of <teamcity.olddomain.com>?
I ran a grep of the entire config directory, nothing. e.g. grep -R teamcity.olddomain.com ./config
The data directory is rather large and full of non-text objects, but I think I did the same thing there and got nothing. Did you have other specific directories for me to query?
can you try to enable debug-clouds logging under Server Administration->Diagnostics and then start an instance once again.
It should produce logging like this:
[2018-04-23 19:23:17,446] DEBUG [nio-8080-exec-5] - .server.impl.CloudEventsLogger - BeforeStart instance: profile 'Amazon-zwei'{id=amazon-1, projectId=_Root}, AmazonImageInstance{id=i-012345, amazonId=null}[2018-04-23 19:23:17,446] INFO [nio-8080-exec-5] - .instances.StartInstanceAction - Starting cloud instance: profile 'Amazon-zwei'{id=amazon-1, projectId=_Root}, AmazonImageInstance{id=i-012345, amazonId=null}, hash=5LypW1lC1OaQlgFA6B6FYyDHgMeVkM5O, reason=User s started agent from web UI, CloudInstanceUserData{myAgentName=<none>, myAuthToken='', myServerAddress='http://localhost:8080/bs', myProfileDescription='profile 'Amazon-zwei'{id=amazon-1, projectId=_Root}', myIdleTimeout=40 min, myCustomAgentConfigurationParameters={system.cloud.profile_id=amazon-1, teamcity.cloud.instance.hash=5LypW1lC1OaQlgFA6B6FYyDHgMeVkM5O}}
[2018-04-23 19:23:17,448] DEBUG [nio-8080-exec-5] - async.AmazonThreadPoolExecutor - Adding operation start EBS instance to the queue
[2018-04-23 19:23:17,449] INFO [nio-8080-exec-5] - .instances.StoppedInstanceTask - Instance has changed status from stopped to Scheduled to start: Amazon Instance{instanceId=i-012345, imageId=i-012345, amazonImageId=null, status: Scheduled to start}, profile 'Amazon-zwei'{id=amazon-1, projectId=_Root}
[2018-04-23 19:23:17,449] INFO [nio-8080-exec-5] - .server.impl.CloudEventsLogger - Cloud instance start succeeded: profile 'Amazon-zwei'{id=amazon-1, projectId=_Root}, Amazon Instance{instanceId=i-012345, imageId=i-012345, amazonImageId=null, status: Scheduled to start}
I do expect to see teamcity.olddomain.com here as well, but maybe there would be something else that will help to shed some light on the issue
UPD:
debug-cloud is not necessary here, actually, to see the server URL, but still maybe there would be smth else
Can you also put some different value to the server URL in profile settings (like teamcity.anotherdomain.com) and see what will appear in logs and injected into user-data
Will get you some logging.
We have already overridden the value in serverUrl and I have not seen it show up consistently in logging. Sometimes the agent config applies, but usually not.
Added the debug, not seeing that URL anywhere (or even the new one).
Feels like there should be a tmp file somewhere I can delete to prevent user-data from being recreated. Perhaps I should delete all cloud config, re-create? (although I've already tried recreating cloud profiles & pools and moving agents around)
FWIW, I grep'd the entire TC dir tree, cannot find the bad URL anywhere.
>We have already overridden the value in serverUrl and I have not seen it show up consistently in logging.
please try searching for this text in teamcity-clouds.log
There was only one entry - our Linux agent starting. It has not experienced the same issues as our Windows agents. Otherwise cannot find that.
Do you have a backup server running somewhere else?
We did have an old server running elsewhere, but I have no idea how the agents would know about it. We shut it off an hour ago. I'll see if I can reproduce the issue now...