Canceled :Agent unauthorized: ASG agent unregistered

Answered

We use the Agent autoscaling feature (https://wecode.wepay.com/posts/cicd-autoscale-two) where agents are created on-demand and terminated when they are idle.

Recently we have started observing that the agents are getting unregistered in the middle of the build with an error

"Canceled :Agent unauthorized: ASG agent unregistered".

There are exceptions at the agent side

"java.net.SocketException: Connection reset" (teamcity-xmlrpc.log).

The agent tries to re-register back with a polling protocol but fails with the below error


"Failed to perform remote command 'pingAndReregister' for build with id 483403, error: java.lang.Exception: Unable to register on server " (teamcity-agent.log).

The server throws the below error and does not seem to let the agent register back
"Agent "teamcity-agent-poc-monolith-ssd-1" {id=26027} tries to register while it is already registered as "teamcity-agent-poc-monolith-ssd-1" {id=26027}"

The autoscaling plugin eventually considered this agent as dead and issues a command to clean up the instance.

Also, some of the SQLs are timing out with the error(thread dump on the server). Is this something to be bothered about?

"Execute query 'select agent.id, name, host_addr, port, status, registration_timestamp, last_binding_timestamp, unregistered_reason, authorized, authorization_token, agent_type_id from agent where (registered <> 1 or registered is null) and id = ?' | PARAMETERS: 20918; 18:42:57 GET '/agents.html?tab=agentPools', from client 35.236.42.130 (127.0.0.1:49052), authenticated as '' () {id=3}; http-nio-8111-exec-504"

I appreciate it if someone suggests a way to avoid this problem. Look forward to your help.

1
1 comment

The issue has been investigated outside of the forums; as a quick community update, it has turned out to be related to the agent-server communication issue. If you would see similar symptoms, please reach out via "Submit a request" form with agent-side logs and teamcity-server.log files from the server log folder.

0

Please sign in to leave a comment.