TeamCity web server frequent crash

Take a look at the working directory (ususally TeamCity/bin). There can be JVM crash logs - files with names like: hs_err_pid*.log
Also please check you are using the latest JVM available.

There are not such files there. Any other hints?     

Can you search the whole disk for these files? JVM process working directory can be different.

I found several of them under c:\windows\system32. Is there anything specific I should be looking for?

Please attach the latest one to this thread.

Hi Pavel, I am not allowed to post this kind of stuff unfortunately. Is there a chance to have it sent to you privately?

Yep, you can use this emal address: teamcity-feedback [at] etbrains.com
BTW crash log does not contain any sensitive data.

Ok, I'm sending it to that address. Thanks a lot.

Please install latest JVM 1.5 or 1.6. Currently you use 1.5.0_12 which is pretty old (latest 1.5 JVM is 1.5.0_22, but 1.6 would be even better). The crash occurs inside JVM code (during socket read), we can't do anything with it - this is a JVM bug.

Thank you, I will try that. Is there anything I need to do to make TC use that JVM then or is it going to use it automatically?

I suppose you are using service to start TeamCity. In this case once you install new Java, launch service editor:

//ES//TeamCity

and specify there path to new JVM.

I have the similar problem after update to 5.0.1.

the problem is when you navigate to some tabs like "project", it show "loading" for a long time. other tabs like "Agents" works fine.

I can't see any error at the event viewer. I check all log files under the Teancity/logs. everything seems fine.


I have to restart the windows service. it show me that: "Windows could not stop the teamcity werver service on local computer, Error 1053: The service did not response to the start or control request in a timely fashion." and the service stoped. I can start it again without any problem.

ater I restart the windows service I see some java exceptions at the stdout_20100112.txt. tkae a look at the attachment below.


So I have to restart Teamcity  couple times a day. any idea?

I don't think it is the same issue. In my case the UI worked fine until suddenly the server stopped working and the Windows Service actually stopped entirely. I have upgraded the JVM and will update the thread as soon as I'm sure that fixed the problem.

If it happens again. please try to navigate to the agent page directly( http://yourserver:yourport/agents.html.) to see what happens.

Thanks

sunny, when the server crashes there's no way you can navigate to a web page served by it

I know. I just doubt that the server is entirely crash. looks like to me that that server is still alive. I can still edit the configuration. but the screen freeze if I navigate to the "project" or "my changes" tab.

when you go to the service, the teamcity server is stoped? it show me that the service is still started. but when I try to stop or restart it. it throw exceptions.

That's why I said that you're in a different situation. In my case the server really crashes.

Sunny, please take a thread dump from the server when it hangs, read more on thread dumps here: http://www.jetbrains.net/confluence/display/TCD5/Reporting+Issues#ReportingIssues-HangsandThreadDumps

I run Teamcity as windows service. So

If I use  AdaptJ StackTrace Utility. it only allow me select the "StackTrace"  process itself. So I guess I have to use "psexec".

for the 'the process id of the hanging process". How can I get this ID?  I saw two processes at  the windows task manager

1. tomcat6.exe *32   service runner (pid=2384)

2. tomcat6.exe *32   service runner (pid=2488)

which one is the process id of the hanging process?

Thanks

Do you see any Java processes? Try to take thread dumps from all of them and attach here dump with "buildserver" string in it.

Here is the thread dump.

I  confirm that  I am using JRE:1.6.0_17-b04 from the stdout_20100113.log  And I confirm that it point to the 1.6.0_17 jvm from the " TeamCity Web Server properties" by running "tomcat6w.exe //ES//TeamCity".

but the dump file show me java running at 1.5.0_12-b04 mixed mode.

any idea?

The dump is from the agent process (do you have agent running on the same server)? Are there other processes?

Hi Pavel,

two days after upgrading the JVM the crash occurred agan. I'm sending you the error log via email again.

Could you please try to add -server argument to JVM?

How?

I tried adding it to Java tab -> Java Options

but the service won't start anymore.

This means that your JRE does not have server version. Please install JDK (not JRE) and try again.

I couldn't make it work, I just pointed it to:

jdk1.6.0_18\jre\bin\server\jvm.dll

I hope it is the same

Yes. I do have one agent running at the same machine ( currently disabled).

I did saw two java process. here is another dump file for another java process.

The dump is from the agent process  show the jre is 1.5. It is the build-in jre for teamcity (from the TeamCity/jre folder). But in the Java tab. I point it to my JDK. and I can see the jre 1.6 is using at the log file.How could that happen?

Still no luck. Let's do the following. Start TeamCity server from the console, using "teamcity-server.bat start" command. To take thread dump just press Ctrl-Break in the console. Try to take several thread dumps with some interval when server starts to hang.