TeamCity web server frequent crash

Hi,

we are experiencing several crashes of the web server with no apparent details in the logs and Windows event log.
Is there a way to collect more information about the crashes?

40 comments
Comment actions Permalink

Take a look at the working directory (ususally TeamCity/bin). There can be JVM crash logs - files with names like: hs_err_pid*.log
Also please check you are using the latest JVM available.

0
Comment actions Permalink

There are not such files there. Any other hints?     


0
Comment actions Permalink

Can you search the whole disk for these files? JVM process working directory can be different.

0
Comment actions Permalink

I found several of them under c:\windows\system32. Is there anything specific I should be looking for?

0
Comment actions Permalink

Please attach the latest one to this thread.

0
Comment actions Permalink

Hi Pavel, I am not allowed to post this kind of stuff unfortunately. Is there a chance to have it sent to you privately?

0
Comment actions Permalink

Yep, you can use this emal address: teamcity-feedback [at] etbrains.com
BTW crash log does not contain any sensitive data.

0
Comment actions Permalink

Ok, I'm sending it to that address. Thanks a lot.

0
Comment actions Permalink

Please install latest JVM 1.5 or 1.6. Currently you use 1.5.0_12 which is pretty old (latest 1.5 JVM is 1.5.0_22, but 1.6 would be even better). The crash occurs inside JVM code (during socket read), we can't do anything with it - this is a JVM bug.

0
Comment actions Permalink

Thank you, I will try that. Is there anything I need to do to make TC use that JVM then or is it going to use it automatically?

0
Comment actions Permalink

I suppose you are using service to start TeamCity. In this case once you install new Java, launch service editor:

//ES//TeamCity

and specify there path to new JVM.
0
Comment actions Permalink

I have the similar problem after update to 5.0.1.

the problem is when you navigate to some tabs like "project", it show "loading" for a long time. other tabs like "Agents" works fine.

I can't see any error at the event viewer. I check all log files under the Teancity/logs. everything seems fine.


I have to restart the windows service. it show me that: "Windows could not stop the teamcity werver service on local computer, Error 1053: The service did not response to the start or control request in a timely fashion." and the service stoped. I can start it again without any problem.

ater I restart the windows service I see some java exceptions at the stdout_20100112.txt. tkae a look at the attachment below.


So I have to restart Teamcity  couple times a day. any idea?



Attachment(s):
stdout_20100112.log.zip
0
Comment actions Permalink

I don't think it is the same issue. In my case the UI worked fine until suddenly the server stopped working and the Windows Service actually stopped entirely. I have upgraded the JVM and will update the thread as soon as I'm sure that fixed the problem.

0
Comment actions Permalink

If it happens again. please try to navigate to the agent page directly( http://yourserver:yourport/agents.html.) to see what happens.

Thanks

0
Comment actions Permalink

sunny, when the server crashes there's no way you can navigate to a web page served by it

0
Comment actions Permalink

I know. I just doubt that the server is entirely crash. looks like to me that that server is still alive. I can still edit the configuration. but the screen freeze if I navigate to the "project" or "my changes" tab.

when you go to the service, the teamcity server is stoped? it show me that the service is still started. but when I try to stop or restart it. it throw exceptions.

0
Comment actions Permalink

That's why I said that you're in a different situation. In my case the server really crashes.

0
Comment actions Permalink

Sunny, please take a thread dump from the server when it hangs, read more on thread dumps here: http://www.jetbrains.net/confluence/display/TCD5/Reporting+Issues#ReportingIssues-HangsandThreadDumps

0
Comment actions Permalink

I run Teamcity as windows service. So

If I use  AdaptJ StackTrace Utility. it only allow me select the "StackTrace"  process itself. So I guess I have to use "psexec".

for the 'the process id of the hanging process". How can I get this ID?  I saw two processes at  the windows task manager

1. tomcat6.exe *32   service runner (pid=2384)

2. tomcat6.exe *32   service runner (pid=2488)

which one is the process id of the hanging process?

Thanks

0
Comment actions Permalink

Do you see any Java processes? Try to take thread dumps from all of them and attach here dump with "buildserver" string in it.

0
Comment actions Permalink

Here is the thread dump.

I  confirm that  I am using JRE:1.6.0_17-b04 from the stdout_20100113.log  And I confirm that it point to the 1.6.0_17 jvm from the " TeamCity Web Server properties" by running "tomcat6w.exe //ES//TeamCity".

but the dump file show me java running at 1.5.0_12-b04 mixed mode.

any idea?



Attachment(s):
buildserver2408.out.zip
0
Comment actions Permalink

The dump is from the agent process (do you have agent running on the same server)? Are there other processes?

0
Comment actions Permalink

Hi Pavel,

two days after upgrading the JVM the crash occurred agan. I'm sending you the error log via email again.

0
Comment actions Permalink

Could you please try to add -server argument to JVM?


0
Comment actions Permalink

How?

I tried adding it to Java tab -> Java Options

but the service won't start anymore.

0
Comment actions Permalink

This means that your JRE does not have server version. Please install JDK (not JRE) and try again.

0
Comment actions Permalink

I couldn't make it work, I just pointed it to:

jdk1.6.0_18\jre\bin\server\jvm.dll

I hope it is the same

0
Comment actions Permalink

Yes. I do have one agent running at the same machine ( currently disabled).

I did saw two java process. here is another dump file for another java process.



Attachment(s):
buildserver2888.out.zip
0
Comment actions Permalink

The dump is from the agent process  show the jre is 1.5. It is the build-in jre for teamcity (from the TeamCity/jre folder). But in the Java tab. I point it to my JDK. and I can see the jre 1.6 is using at the log file.How could that happen?

0
Comment actions Permalink

Still no luck. Let's do the following. Start TeamCity server from the console, using "teamcity-server.bat start" command. To take thread dump just press Ctrl-Break in the console. Try to take several thread dumps with some interval when server starts to hang.

0

Please sign in to leave a comment.