Cause the build hung up

Hi,

This issue can show up if there is issues with connectivity with the agents. They might lose connection to the server and report back for registration when the server hasn't still removed the registration. After re-registration, the connection should work fine.

If the issue keeps happening consistently, you can try manually removing the agents from the web UI, then adding them back. If it still reproduces afterwards, send us the teamcity-server.log and the teamcity-agent.log of one of the agents via the Submit a request button on top of the page.

Thank you!

Please let me further questions.

In order to investigate the cause of the connection issues,

If you check 'teamcity-agent.log'

The following log was output every time WARN occurred.

What kind of cause can be considered?

teamcity-agent.log:

[2018-01-25 18:30:33,530] INFO - r.agent.impl.AgentLogProxyImpl - Failed to perform remote command '100 log messages' for build with id 400044, error: jetbrains.buildServer.xmlrpc.RemoteCallException: Call 'https://172.20.149.92/teamcity/RPC2', method 'buildServer.log' failed: org.apache.xmlrpc.XmlRpcClientException: Server returned incorrect status code: 500 Internal Server Error: jetbrains.buildServer.xmlrpc.RemoteCallException: Call 'https://172.20.149.92/teamcity/RPC2', method 'buildServer.log' failed: org.apache.xmlrpc.XmlRpcClientException: Server returned incorrect status code: 500 Internal Server Error (enable debug to see stacktrace) 
[2018-01-25 18:30:33,530] INFO - r.agent.impl.AgentLogProxyImpl - Adding ping and re-register command on top of the remote command queue 
[2018-01-25 18:30:33,531] INFO - buildServer.AGENT.registration - Registering on server via URL https://172.20.149.92/teamcity: AgentDetails{Name='BuildAgent-03', AgentId=64, BuildId=400044, AgentOwnAddress='52.197.71.210', AlternativeAddresses=[172.20.149.83, 52.197.71.210], Port=9090, Version='47070', PluginsVersion='47070-md5-fb171cd396a9ef0d429e43d44a7dba9e', AvailableRunners=[Ant, cargo-deploy-runner, Duplicator, ftp-deploy-runner, gradle-runner, Inspection, jonnyzzz.grunt, jonnyzzz.gulp, jonnyzzz.node, jonnyzzz.npm, jonnyzzz.nvm, jonnyzzz.phantom, JPS, Maven2, rake-runner, SBT, simpleRunner, smb-deploy-runner, ssh-deploy-runner, ssh-exec-runner], AvailableVcs=[tfs, cvs, jetbrains.git, mercurial, svn, perforce], AuthorizationToken='1d1ba17b1c38f911080a425dec4e2c0a', PingCode='nlijZ1c2rBEm6FlGjlJNUsuT6Z7iakQq'} 
[2018-01-25 18:30:33,533] INFO - buildServer.AGENT.registration - Server supports the following communication protocols: [polling] 
[2018-01-25 18:30:33,533] INFO - buildServer.AGENT.registration - Trying to register on server using 'polling' protocol. 
[2018-01-25 18:30:33,537] INFO - ldServer.AGENT.PollingProtocol - Stop polling server for commands 

teamcity-server.log:

[2018-01-25 18:30:33,537] WARN - jetbrains.buildServer.AGENT - Agent "BuildAgent-03" {id=64} tries to register while it is already registered. 

Thanks for adding the extra information. The "Failed to perform remote command" is probably happening for quite a while before that action is triggered. This is usually due to connectivity issues, then as the log states, it tries to re-register because there are too many errors trying to connect. As mentioned, please give it a restart, and tell us if it keeps failing.

Thank you!

At that time succeeded in rerunning the build,

After that, this kind of situation has not occurred so far.

(Agent re-registration has not been performed.)

There was one thing to tell.

At that time, the disk usage of the Teamcity server was 100%.

Filesystem Size Used Avail Use% Mounted on
udev       7.9G  12K 7.9G 1%   /dev
tmpfs      1.6G 388K 1.6G 1%   /run
/dev/xvda1 7.8G 7.4G    0 100% /
none       4.0K    0 4.0K 0%   /sys/fs/cgroup
none       5.0M    0 5.0M 0%   /run/lock
none       7.9G    0 7.9G 0%   /run/shm
none       100M    0 100M 0%   /run/user

For this reason, we had deleted logs of nginx that had accumulated before re-running the build.

Is there a possibility that it may be caused by insufficient capacity of the disk?

Glad to read it's working now. It's likely that having 0 space on the agent created the issue, but it's hard to ensure that it's the certain root of the issue, so please keep an eye for further issues. Unfortunately, having no disk space is sometimes very hard to track down, as only a subset of the problems can be directly detected. If the agent runs out of space often, I would suggest using some of the build features to clean up space.

Since then the issues has not occurred, urgent response seems unnecessary, so I'd like to see a state for a while.

Thank you for your response.

Hi Denis,

By removing the build agent, do you mean to Unauthorize the agent, remove the authroizationToken's value in the conf/buildAgent.properties, and then re-authorizing the agent so it has a fresh token?

Or other steps need to be taken about it?

Thanks,

Waqas

Hi Syed,

for that point in the issue, there shouldn't be a need to remove the auth token from the build agent properties, they will be overwritten after the next authorization. The recommendation would include unauthorizing the agent, then fully stopping it and restarting it and reauthorizing. The idea is that if due to some background issue it has gotten to an inconsistent state, that should refresh its state to avoid that (which is what happens usually automatically during the re-registration process), so it isn't guaranteed to work, as the issue has many different possible causes, as you can see if you follow the rest of the thread.

If you are having specific issues, please detail them, although it might be better to do so in its own thread.