Teamcity Maven and MSBuilds hang when upgraded to version 6.0.1

Arun, try to take thread dump from the agent Java process (or from all of the Java processes on the agent). The simplest possible way to do this is to launch agent using Java 1.6 and then use Java jstack utility to take thread dump.

Pavel,

Please could give me the steps required to  launch agent using Java 1.6?

Given that you have installed Java 1.6 you can do the following:
- stop the agent service
- edit <agent inst dir>/launcher/conf/wrapper.conf file - specify there path to Java 1.6 java exe (wrapper.java.command property)
- start the agent service

When agent connects to the server you can check whether it is using the right version of Java on Agent Parameters tab, you need to look for: teamcity.agent.jvm.version parameter

I have tired running the jstack tool against the build agent as suggested but I am having problems obtaining the thread dump from the agent. I have tired the various methods as well suggested on : http://confluence.jetbrains.net/display/TCD6/Reporting+Issues#ReportingIssues-Agentthreaddump

• if JDK 1.6 is used, use jstack <pid_of_java_process> command (jstack is located in the "bin" directory of JDK installation).
• C:\Program Files\Java\jdk1.6.0_20\bin>psexec jstack 5592 > threaddumps.log
  
  PsExec v1.94 - Execute processes remotely
  Copyright (C) 2001-2008 Mark Russinovich
  Sysinternals - www.sysinternals.com
  
  PsExec could not start C:\Program:
  The system cannot find the file specified.

     I get the above error.

When I tried the second method

• you can also use TeamCity-bundled thread dump tool (can be found in agent's plugins).
• I get the below error

     C:\TeamCity\buildAgent\plugins\stacktracesPlugin\bin\x86\JetBrains.TeamCity.Injector.exe exited with error code -6.

Please advise.

Have you tried to use jstack without psexec?

Hi Pavel,

I have attached a thread dump of the agent finally got it to work, it was because of the account I was using to run the build agent. Attached are two agent thread dumps one is when the agent is running our dotnet build using msbuild and the other is using the maven build.

Thanks

Arun, please try to take thread dumps from all of the Java processes on the agent. These thread dumps are from agent launcher process, while we need thread dump from the agent process itself.

Attached are all the thread dumps from all of the Java processes on the agent.

Any update on this please?

Please can you give me an update on this issue as its causing major problems to our builds

Hello, Arun,

What OS is running your build agent?

Hi Victory,

The operating system that running the build agent is Microsoft Windows Server 2003 R2     

Thanks

Arun

Arun,

Such a hanging is a known issue with xml-report-plugin and Windows Server 2003 (http://youtrack.jetbrains.net/issue/TW-12798) which is very like to be an internal Java issue.
We are now on a vacation. Will prepare a custom xml-report-plugin build for you to try as soon as back to office (in several days).

I appreciate the response Victory. However, we are working to extremely tight timeframes so please can you coordinate resolution with a colleague/team whilst you are on vacation. If you have any questions or if this is an issue please can you escalate asap.

Hello, Arun,

Here is a custom plugin pack which hopefully avoids hanging in your case.

Please, put the xml-report-plugin.zip to .BuildServer/plugins folder and restart TeamCity server.

Let me know if facing any problems.

Hi Victory,

Thank you for the quick response, going to test it out now and will let you know of the outcome. Just have a quick question can i just remove the previous plugin that is there and replace it with this one?

Thanks

Hi Victory,

I have put the plugin in and my agent is showing "Agent has unregistered (will upgrade)"? and has been like that for a couple of minutes

Please advise

The agent upgrade process should start shortly.
If it doesn't, please attach the teamcity-server.log and teamcity-agent.log

The agent is still upgrading, attached are the logs as requested..

justed updated log files

No, you can't.
To install the new plugin version you should put it to .BuildServer/plugins and restart the server.

Hello,

I do not see any errors in the build agent log file you attached. Build agent update process may take 5-10 minutes. While the upgrade build agent windows service will be reported as stopped. To enforce build agent to re-update for the build agent, please do the following:
- stop build agent windows service
- remove <agent>/plugins folder
- start build agent
- wait till build agent is connected back to TeamCity.

Let me know if the process failed. If so, please attach all files from <agent>/logs folder.
Thanks!

Thanks, will let you know of the outcome

I have waited 15 minutes and it remains in the "Agent has unregistered (will upgrade)" state, attached are the logs file as requsted.

I have uninstalled and re-installed the agent, this seems to have worked, its now connected, will now test the builds...will let you know of the outcome

The plugin has fixed the issue, the builds no longer hang..thank you all for your help!! :)

Will this issue be part of the next release of teamcity as a fix, as this has happened to us twice now ?

This fix will be included into the next release. We apologize for not doing it last time.

Thank you.

As for publishing an unrecognized artifact.

This is normal since 6.0. "maven-build-info.xml.gz" is an internal TeamCity artifact used for showing Maven build info in the corresponding tab on the build page.

Can someone confirm that this issue has been fixed in the latest teamcity build? As I am thinking of updating