Agent has unregistered (will upgrade)

I've just installed a TeamCity server and one build agent. When I browse to the server's webpage for build agents, I see that the agent is disconnected with the following message:

"Agent has unregistered (will upgrade) "

I've attached all the agent logs. Can someone please tell me what's wrong?

Some possibilities that crossed my mind:

* The server has no access to the internet, i.e. you can't browse to www.google.com from there. However, it does have access to the intranet.

* The server has no access to network shares, though it can share it's own folders

* The agent runs as Windows service under the local system account - perhaps insufficient permissions?

Any help would be much appreciated. Thanks.



Attachment(s):
logs.zip
7 comments
Comment actions Permalink

Any response on this please?

I noticed this in the upgrade log:

[2010-12-20 22:59:35,896]   WARN - buildServer.agent.LauncherUtil - Error deleting file: E:\BuildAgents\PBTBuildAgent1\lib\xstream-1.3.1.jar
[2010-12-20 22:59:35,896]  ERROR - ver.agent.upgrade.AgentUpgrade - Upgrade failed :Problems deleting E:\BuildAgents\PBTBuildAgent1\lib
java.io.IOException: Problems deleting E:\BuildAgents\PBTBuildAgent1\lib
 at jetbrains.buildServer.agent.upgrade.UpgradeFolder.doUpgrade(UpgradeFolder.java:52)
 at jetbrains.buildServer.agent.upgrade.UpgradeFolder.upgrade(UpgradeFolder.java:38)
 at jetbrains.buildServer.agent.upgrade.AgentUpgrade.doUpgrade(AgentUpgrade.java:84)
 at jetbrains.buildServer.agent.upgrade.AgentUpgrade.applyUpdates(AgentUpgrade.java:56)
 at jetbrains.buildServer.agent.upgrade.UpgradeRunBase.run(UpgradeRunBase.java:37)
 at jetbrains.buildServer.agent.upgrade.UpgradeMode$2.run(UpgradeMode.java:28)
 at jetbrains.buildServer.agent.upgrade.Upgrade2.main2(Upgrade2.java:33)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at jetbrains.buildServer.agent.ClazzLoader.call(ClazzLoader.java:59)
 at jetbrains.buildServer.agent.ClazzLoader.callMain2(ClazzLoader.java:19)
 at jetbrains.buildServer.agent.upgrade.Upgrade.main(Upgrade.java:16)
[2010-12-20 22:59:35,896]   INFO - ver.agent.upgrade.AgentUpgrade - AGENT UPDATE FAILED, exiting ...


However, when I used 'Unlocker' to see what locks it, I see it's java.exe from Teamcity itself.

I've also compared the TeamCity\webapps\ROOT\update\plugins folder versus the BuildAgent\update\plugins folder. All non-zero kB files seem to match up.

Lastly, when I restarted the agent today, I noticed that it ran up 4 java.exe processes from the build agent and they were maxing out the CPU. There are a lot of these in the logs:

[2010-12-21 11:44:49,773]   WARN -    jetbrains.buildServer.AGENT - Registration attempt for buildId: null was ignored. RegistrationInProgress lock was held
[2010-12-21 11:44:57,836]   WARN -    jetbrains.buildServer.AGENT - Registration attempt for buildId: null was ignored. RegistrationInProgress lock was held
[2010-12-21 11:45:39,415]   WARN -    jetbrains.buildServer.AGENT - Registration attempt for buildId: null was ignored. RegistrationInProgress lock was held

0
Comment actions Permalink

It might also be worth noting that the server and agent run on the same VM.

0
Comment actions Permalink

Chien,

Yes, as you noted the upgrade fails beacause it cannot delete E:\BuildAgents\PBTBuildAgent1\lib directory as the files in it seems to be locked. During the upgrade, agent runs a separate upgrade process, the main process exits and the upgrade process replaces all the necessary files. During this time agent service is not started and no manual intervention is recommended.

In your case it seems that at some point there were two agent processes launched.

Can you please stop TeamCity build agent and check there is no other/duplicate process running?
If not, then try to run TeamCity agent to see if the issue reproduces. If it does, then it would be helpful to try get the locking process at the same time as the agent tries to delete the files.

0
Comment actions Permalink

Before your response, I uninstalled the agent, ensured no TeamCity java.exe processes were running, re-installed the agent then started it.

I now see 2 java.exe processes from BuildAgent\jre\bin. One of them is holding the lock on the lib directory. I've attached all the new logs from the fresh reinstall.

I'm not sure what other information I can give you. Please let me know what else you need.



Attachment(s):
logs-1.zip
0
Comment actions Permalink

It seems to work now, though I don't know what has changed!

I stopped the service, killed a lingering java.exe process from the build agent, and restarted. Now it works. ?:|

Thanks anyway. :^O

0
Comment actions Permalink

In my case the "will upgrade" bug turned out to be caused by a bug in networking performance in a VMWare virtual machine.

  http://communities.vmware.com/thread/34497

I applied Microsoft's fix to the VMWare host....

http://support.microsoft.com/kb/888750

... and it then worked for me.  I suspect this indicates a race condition bug in the agent upgrade process which could cause random failures.

0
Comment actions Permalink

Tom,

If you believe there is a bug, please post the description of the case accompanied by the all agent logs into our issue tracker.

0

Please sign in to leave a comment.