TeamCity freezes after -Server configuration loaded- message in logs

Guys
I'm running into the following issue:

I had a machine restart over the weekend, when I'm trying to start TeamCity server now (using runAll.sh script) after a few seconds all I get is TeamCity maintenance page.
The log file stopps at:

[2013-09-02 11:03:42,062]   INFO - rver.plugins.PluginManagerImpl - ===========================================================
[2013-09-02 11:03:42,062]   INFO -   jetbrains.buildServer.SERVER - TeamCity plugins loaded in: 15070 ms
[2013-09-02 11:03:42,112]   INFO -   jetbrains.buildServer.SERVER - Start loading server configuration...
[2013-09-02 11:03:42,124]   INFO -  jetbrains.buildServer.CLEANUP - Max cleanup duration is set to to 0 seconds
[2013-09-02 11:03:42,125]   INFO -  jetbrains.buildServer.CLEANUP - Cleanup time has been set to 03:00:00 daily
[2013-09-02 11:03:42,126]   INFO -  jetbrains.buildServer.CLEANUP - Cleanup is enabled
[2013-09-02 11:03:42,681]   INFO -   jetbrains.buildServer.SERVER - Server configuration loaded

The Catalina out shows me this:

02-Sep-2013 10:45:23 org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /opt/jdk1.6.0_13/jre/lib/amd64/server:/opt/jdk1.6.0_13/jre/lib/amd64:/opt/jdk1.6.0_13/jre/../lib/amd64:/usr/java/packages/lib/amd64:/lib:/usr/lib
02-Sep-2013 10:45:24 org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-nio-8111"]
02-Sep-2013 10:45:24 org.apache.tomcat.util.net.NioSelectorPool getSharedSelector
INFO: Using a shared selector for servlet write/read
02-Sep-2013 10:45:24 org.apache.catalina.startup.Catalina load

The javaLoggin file shows this:

02-Sep-2013 11:03:05 org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring FrameworkServlet 'buildServer'
02-Sep-2013 11:03:31 com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.11 12/09/2011 10:27 AM'

And nothing happens.

I need some gelp here. All was working fine until today.

My setup is:

  • running on 32 Bit Linux box with 32 bit JDK 1.6_13
  • HSQL DB that comes with TeamCity
  • TeamCity version 8.0.2 (build 27482)

I need assistence in finding the issue.
Cheers, Greg
6 comments
Comment actions Permalink

After I left TC in this state for some time, it finally managed to start. (A bit odd)

[2013-09-02 11:03:42,681]   INFO -   jetbrains.buildServer.SERVER - Server configuration loaded
[2013-09-02 11:24:58,521]   INFO -   jetbrains.buildServer.SERVER - Start prefetching changes for 6 build configurations...

It took it 20 minutes though. Any ideas what might have happened?
I only have 6 projects with 10 build configurations all together.

Cheers, Greg

0
Comment actions Permalink

One more thing, I have a farly large list of users (15000)  and groups (6000).
Thought I need to mention that.
Greg

0
Comment actions Permalink

Hello Greg,

please send the server thred dump according to this guide http://confluence.jetbrains.com/display/TCD8/Reporting+Issues section "Slowness, Hangings and Low Performance".

regards

Zdenek

0
Comment actions Permalink

Hi Zdenek

Please find the Thread Dumps attached to this post.

The first three Dumps were taken few seconds apart once the problem described by me in the original post occured.

The last thread dump was taken when the TeamCity server started to respond to requests with login page not the Maintenance page.

Greg



Attachment(s):
thread-dump4.txt.zip
thread-dump3.txt.zip
thread-dump2.txt.zip
thread-dump.txt.zip
0
Comment actions Permalink

Hi Pavel
The issue seems to be fixed with the nightly build. http://youtrack.jetbrains.com/issue/TW-31852#comment=27-581894
Thanks, Greg

0

Please sign in to leave a comment.