Teamcity Maven and MSBuilds hang when upgraded to version 6.0.1

I have recently updated to teamcity 6 and appear to be having the same problems with our maven build. At the end of the build teamcity produces an error "Process exited with code 0" and then tries to publish artifacts that I have not even specified. This hangs the build and I am not able to stop via the UI.

[22:48:36]: [INFO] ------------------------------------------------------------------------
[22:48:36]: [INFO] BUILD SUCCESSFUL
[22:48:36]: [INFO] ------------------------------------------------------------------------
[22:48:36]: [INFO] Total time: 5 minutes 12 seconds
[22:48:36]: [INFO] Finished at: Wed Jan 05 22:48:36 GMT 2011
[22:48:36]: [INFO] Final Memory: 32M/78M
[22:48:36]: [INFO] ------------------------------------------------------------------------
[22:48:37]: Process exited with code 0
[22:48:37]: Publishing artifacts (18s)
[22:48:37]: [Publishing artifacts] Paths to  publish:  [C:\TeamCity\buildAgent\temp\buildTmp\.tc-maven-bi\maven-build-info.xml.gz  => .teamcity]
[22:48:37]: [Publishing artifacts] Sending files


This relates to the http://devnet.jetbrains.net/message/5282468#5282468


Please advise


Attachment(s):
teamcity-build.log.zip
teamcity-agent.log.zip
teamcity-server.log.zip
threadDump-2011-01-06_09.29.05.txt.zip
32 comments
Comment actions Permalink

Arun, try to take thread dump from the agent Java process (or from all of the Java processes on the agent). The simplest possible way to do this is to launch agent using Java 1.6 and then use Java jstack utility to take thread dump.

0
Comment actions Permalink

Pavel,

Please could give me the steps required to  launch agent using Java 1.6?

0
Comment actions Permalink

Given that you have installed Java 1.6 you can do the following:
- stop the agent service
- edit <agent inst dir>/launcher/conf/wrapper.conf file - specify there path to Java 1.6 java exe (wrapper.java.command property)
- start the agent service

When agent connects to the server you can check whether it is using the right version of Java on Agent Parameters tab, you need to look for: teamcity.agent.jvm.version parameter

0
Comment actions Permalink

I have tired running the jstack tool against the build agent as suggested but I am having problems obtaining the thread dump from the agent. I have tired the various methods as well suggested on : http://confluence.jetbrains.net/display/TCD6/Reporting+Issues#ReportingIssues-Agentthreaddump

  • if JDK 1.6 is used, use jstack <pid_of_java_process> command (jstack is located in the "bin" directory of JDK installation).
  • C:\Program Files\Java\jdk1.6.0_20\bin>psexec jstack 5592 > threaddumps.log

    PsExec v1.94 - Execute processes remotely
    Copyright (C) 2001-2008 Mark Russinovich
    Sysinternals - www.sysinternals.com

    PsExec could not start C:\Program:
    The system cannot find the file specified.



     I get the above error.

When I tried the second method

  • you can also use TeamCity-bundled thread dump tool (can be found in agent's plugins).
  • I get the below error

     C:\TeamCity\buildAgent\plugins\stacktracesPlugin\bin\x86\JetBrains.TeamCity.Injector.exe exited with error code -6.

Please advise.

0
Comment actions Permalink

Have you tried to use jstack without psexec?

0
Comment actions Permalink

Hi Pavel,

I have attached a thread dump of the agent finally got it to work, it was because of the account I was using to run the build agent. Attached are two agent thread dumps one is when the agent is running our dotnet build using msbuild and the other is using the maven build.

Thanks



Attachment(s):
agentThreadDumpMaven.txt.zip
agentThreadDumpMSBuild.txt.zip
0
Comment actions Permalink

Arun, please try to take thread dumps from all of the Java processes on the agent. These thread dumps are from agent launcher process, while we need thread dump from the agent process itself.

0
Comment actions Permalink

Any update on this please?

0
Comment actions Permalink

Please can you give me an update on this issue as its causing major problems to our builds

0
Comment actions Permalink

Hello, Arun,

What OS is running your build agent?

0
Comment actions Permalink

Hi Victory,

The operating system that running the build agent is Microsoft Windows Server 2003 R2     

Thanks

Arun

0
Comment actions Permalink

Arun,

Such a hanging is a known issue with xml-report-plugin and Windows Server 2003 (http://youtrack.jetbrains.net/issue/TW-12798) which is very like to be an internal Java issue.
We are now on a vacation. Will prepare a custom xml-report-plugin build for you to try as soon as back to office (in several days).

0
Comment actions Permalink

I appreciate the response Victory. However, we are working to extremely tight timeframes so please can you coordinate resolution with a colleague/team whilst you are on vacation. If you have any questions or if this is an issue please can you escalate asap.

0
Comment actions Permalink

Hello, Arun,

Here is a custom plugin pack which hopefully avoids hanging in your case.

Please, put the xml-report-plugin.zip to .BuildServer/plugins folder and restart TeamCity server.

Let me know if facing any problems.



Attachment(s):
xml-report-plugin.zip
0
Comment actions Permalink

Hi Victory,

Thank you for the quick response, going to test it out now and will let you know of the outcome. Just have a quick question can i just remove the previous plugin that is there and replace it with this one?

Thanks

0
Comment actions Permalink

Hi Victory,

I have put the plugin in and my agent is showing "Agent has unregistered (will upgrade)"? and has been like that for a couple of minutes

Please advise

0
Comment actions Permalink

The agent upgrade process should start shortly.
If it doesn't, please attach the teamcity-server.log and teamcity-agent.log

0
Comment actions Permalink

The agent is still upgrading, attached are the logs as requested..

justed updated log files



Attachment(s):
teamcity-server.log.zip
teamcity-agent.log.zip
0
Comment actions Permalink

No, you can't.
To install the new plugin version you should put it to .BuildServer/plugins and restart the server.

0
Comment actions Permalink

Hello,

I do not see any errors in the build agent log file you attached. Build agent update process may take 5-10 minutes. While the upgrade build agent windows service will be reported as stopped. To enforce build agent to re-update for the build agent, please do the following:
- stop build agent windows service
- remove <agent>/plugins folder
- start build agent
- wait till build agent is connected back to TeamCity.

Let me know if the process failed. If so, please attach all files from <agent>/logs folder.
Thanks!

0
Comment actions Permalink

Thanks, will let you know of the outcome

0
Comment actions Permalink

I have waited 15 minutes and it remains in the "Agent has unregistered (will upgrade)" state, attached are the logs file as requsted.



Attachment(s):
logs.zip
0
Comment actions Permalink

I have uninstalled and re-installed the agent, this seems to have worked, its now connected, will now test the builds...will let you know of the outcome

0
Comment actions Permalink

The plugin has fixed the issue, the builds no longer hang..thank you all for your help!! :)

0
Comment actions Permalink

Will this issue be part of the next release of teamcity as a fix, as this has happened to us twice now ?

0
Comment actions Permalink

This fix will be included into the next release. We apologize for not doing it last time.

0
Comment actions Permalink

As for publishing an unrecognized artifact.

This is normal since 6.0. "maven-build-info.xml.gz" is an internal TeamCity artifact used for showing Maven build info in the corresponding tab on the build page.

0
Comment actions Permalink

Can someone confirm that this issue has been fixed in the latest teamcity build? As I am thinking of updating

0

Please sign in to leave a comment.