EC2 Cloud Agent Startup - Always upgrading - even after installing full plugins

We are building TC build agents into AMI's then hosted in AWS EC2. We use packer.io to build the image, then setup TeamCity to spin up those AMI's as agents when needed. This is working well and does what we need - except the startup of these agents is over 10 minutes - not AWS being slow, it's the initial startup of the agent service itself and it's need to upgrade itself. 

We install the agent using the "full" agent zip with packer and when it's first start during that packer build everything is fine with no upgrade. But when we shut the machine down, create the AMI, then start up an EC2 based off that AMI the agent thinks everything needs to be upgraded (even minutes between the ami being created and started). Here is an example snip from the agent log on the ec2 startup: 

[2021-05-14 17:58:57,098]   INFO - .vmware.VMWarePropertiesReader - VSphere plugin initializing...
[2021-05-14 17:58:57,098]   INFO - .vmware.VMWarePropertiesReader - Unable to locate vmware-rpctool. Looks like not a VMWare VM or VWWare tools are not installed
[2021-05-14 17:58:59,381]   INFO - gers.vcs.git.MirrorManagerImpl - No mapping file found at Z:\system\git\map, create a new one
[2021-05-14 17:58:59,381]   INFO - gers.vcs.git.MirrorManagerImpl - Restore mapping from existing repositories
[2021-05-14 17:58:59,381]   INFO - gers.vcs.git.MirrorManagerImpl - No existing repositories found
[2021-05-14 17:59:09,165]   INFO - rver.plugins.PluginManagerImpl - Plugins initialization completed (59 plugins loaded): [agentPlugin, agentSystemInfo, amazonEC2, ant, ant-net-tasks, antPlugin, aws-codedeploy-plugin-agent, awsS3Plugin-agent, cloud-vmware-agent, commandLineRunner, coveragePlugin, crashDetector, cvsAgent, deploy-runner-agent, docker-support, dotnet, dotnet-tools, dotnetPlugin, dotNetRunners, duplicatePlugin, environment-fetcher, file-content-replacer, fxcop, gant, gantPlugin, golang-agent, gradle-runner, idea-runner, inspectionPlugin, java-dowser, jetbrains.git, jonnyzzz.node, jps, junitPlugin, jvm-update, mavenPlugin, mercurial, meta-runner, nuget-agent, Octopus.TeamCity, perfmon-agent, perforce-agent, powershell-agent, python-agent-86002, rake-runner, remoteAccess, s3-artifact-storage-agent, sbt-runner-agent, sonar-plugin-agent, ssh-manager, stacktracesPlugin, svnAgent, swabra, teamcity-kubernetes-plugin-agent, testNGPlugin, tfs-agent, visualstudiotest, xcode-runner, xml-report-plugin]
[2021-05-14 17:59:09,165]   INFO - rver.plugins.PluginManagerImpl - ===========================================================
[2021-05-14 17:59:09,259]   WARN - ent.plugins.AgentPluginsLoader - Plugin Octopus.TeamCity.zip was not found 
[2021-05-14 17:59:09,259]   WARN - ent.plugins.AgentPluginsLoader - Plugin agentPlugin.zip was not found 
[2021-05-14 17:59:09,259]   WARN - ent.plugins.AgentPluginsLoader - Plugin agentSystemInfo.zip was not found 
[2021-05-14 17:59:09,259]   WARN - ent.plugins.AgentPluginsLoader - Plugin amazonEC2.zip was not found 
[2021-05-14 17:59:09,259]   WARN - ent.plugins.AgentPluginsLoader - Plugin ant-net-tasks.zip was not found 
[2021-05-14 17:59:09,259]   WARN - ent.plugins.AgentPluginsLoader - Plugin ant.zip was not found 
[2021-05-14 17:59:09,259]   WARN - ent.plugins.AgentPluginsLoader - Plugin antPlugin.zip was not found 
[2021-05-14 17:59:09,259]   WARN - ent.plugins.AgentPluginsLoader - Plugin aws-codedeploy-plugin-agent.zip was not found 

Notice it says the plugin was not found. They were all there during the packer build install. We've played around with permissions, even setting the "c:\buildagent\" directory to be fully readwrite by everyone. I've uploaded a full agent log from this AMI (first part is the packer install, then second half is actually it starting up). What am I missing? 

Upload id: 2021_05_14_WYF5nHkocvbsvdyc

Please sign in to leave a comment.