Teamcity website is not loading...

Hi,

When we try to go to the TeamCity website on our build server, the webpage just hangs.  We can get the login window, but once we enter login credentials, the next page never comes up.

I have tried this in Chrome, Firefox and IE10 - no joy.

We were on TeamCity 8.0.5, but I have upgraded to 8.1.3 to see if that helps, but it doesnt seem to.

We use MySQL for the DB and I can query that ok via its tools.

I can see a successful login in the teamcity-server.log, but nothing else is in the log.

Looking in Chrome dev tools/inspector - it seems to be hanging on the GET for overview.html.

Any suggestions on where I should look to investigate this?

Could it be an out of memory issue - is that logged anywhere?

Its running on old/slow Virtual Machine, but the machine is generally quiet.  I have restarted the VM, but no joy.

I have even turned off the local build agent, to ensure there is no contention.

Thanks in advance,
Chris

8 comments
Comment actions Permalink

I see this error in the teamcity-winservice.log, not sure if its relevant:

[2014-06-30 07:49:45,100] console                           [Info] INFO: Server startup in 37314 ms

[2014-06-30 07:51:45,477] console                           [Info] [2014-06-30 07:51:45,383]  ERROR - gins.spring.SpringPluginLoader - Failed to initialize spring context for plugin autoincrementer

[2014-06-30 07:51:45,477] console                           [Info] java.lang.NoClassDefFoundError: jetbrains/buildServer/web/plugins/bean/ServerPluginInfo

[2014-06-30 07:51:45,477] console                           [Info]  at java.lang.Class.getDeclaredConstructors0(Native Method)

[2014-06-30 07:51:45,477] console                           [Info]  at java.lang.Class.privateGetDeclaredConstructors(Class.java:2493)

[2014-06-30 07:51:45,477] console                           [Info]  at java.lang.Class.getDeclaredConstructors(Class.java:1901)

[2014-06-30 07:51:45,477] console                           [Info]  at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.determineCandidateConstructors(AutowiredAnnotationBeanPostProcessor.java:227)

[2014-06-30 07:51:45,477] console                           [Info]  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.determineConstructorsFromBeanPostProcessors(AbstractAutowireCapableBeanFactory.java:930)

[2014-06-30 07:51:45,477] console                           [Info]  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:903)

[2014-06-30 07:51:45,477] console                           [Info]  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)

[2014-06-30 07:51:45,477] console                           [Info]  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)

[2014-06-30 07:51:45,477] console                           [Info]  at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)

[2014-06-30 07:51:45,477] console                           [Info]  at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)

[2014-06-30 07:51:45,477] console                           [Info]  at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)

[2014-06-30 07:51:45,477] console                           [Info]  at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)

[2014-06-30 07:51:45,477] console                           [Info]  at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:580)

[2014-06-30 07:51:45,477] console                           [Info]  at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)

[2014-06-30 07:51:45,477] console                           [Info]  at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)

[2014-06-30 07:51:45,477] console                           [Info]  at jetbrains.buildServer.plugins.spring.SpringPluginLoader.pluginClassesLoaded(SpringPluginLoader.java:72)

[2014-06-30 07:51:45,477] console                           [Info]  at sun.reflect.GeneratedMethodAccessor58.invoke(Unknown Source)

[2014-06-30 07:51:45,477] console                           [Info]  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

[2014-06-30 07:51:45,477] console                           [Info]  at java.lang.reflect.Method.invoke(Method.java:606)

[2014-06-30 07:51:45,477] console                           [Info]  at jetbrains.buildServer.util.EventDispatcher.dispatch(EventDispatcher.java:121)

[2014-06-30 07:51:45,477] console                           [Info]  at jetbrains.buildServer.util.EventDispatcher$2.invoke(EventDispatcher.java:71)

[2014-06-30 07:51:45,477] console                           [Info]  at com.sun.proxy.$Proxy8.pluginClassesLoaded(Unknown Source)

[2014-06-30 07:51:45,477] console                           [Info]  at jetbrains.buildServer.plugins.PluginManagerImpl$2.visitPlugin(PluginManagerImpl.java:122)

[2014-06-30 07:51:45,477] console                           [Info]  at jetbrains.buildServer.plugins.PluginsCollection.foreachLoadedPlugins(PluginsCollection.java:107)

[2014-06-30 07:51:45,477] console                           [Info]  at jetbrains.buildServer.plugins.PluginManagerImpl.firePluginClassesLoaded(PluginManagerImpl.java:120)

[2014-06-30 07:51:45,477] console                           [Info]  at jetbrains.buildServer.plugins.PluginManagerImpl.loadPlugins(PluginManagerImpl.java:80)

[2014-06-30 07:51:45,477] console                           [Info]  at jetbrains.buildServer.web.plugins.PluginManagerConfigurator.initializePlugins(PluginManagerConfigurator.java:0)

[2014-06-30 07:51:45,477] console                           [Info]  at jetbrains.buildServer.web.impl.BuildServerConfigurator.loadConfiguration(BuildServerConfigurator.java:19)

[2014-06-30 07:51:45,477] console                           [Info]  at jetbrains.buildServer.maintenance.TeamCityDispatcherServlet$WebApplicationCreatorAndDestroyer.createApplication(TeamCityDispatcherServlet.java:16)

[2014-06-30 07:51:45,477] console                           [Info]  at jetbrains.buildServer.maintenance.StartupProcessor.doApplicationStarting(StartupProcessor.java:319)

[2014-06-30 07:51:45,477] console                           [Info]  at jetbrains.buildServer.maintenance.StartupProcessor.processConcreteStage(StartupProcessor.java:439)

[2014-06-30 07:51:45,477] console                           [Info]  at jetbrains.buildServer.maintenance.StartupProcessor.processConcreteStageSafe(StartupProcessor.java:70)

[2014-06-30 07:51:45,477] console                           [Info]  at jetbrains.buildServer.maintenance.StartupProcessor.processTeamCityLifecycle(StartupProcessor.java:118)

[2014-06-30 07:51:45,477] console                           [Info]  at jetbrains.buildServer.maintenance.StartupProcessor.access$000(StartupProcessor.java:9)

[2014-06-30 07:51:45,477] console                           [Info]  at jetbrains.buildServer.maintenance.StartupProcessor$1.run(StartupProcessor.java)

[2014-06-30 07:51:45,477] console                           [Info]  at java.lang.Thread.run(Thread.java:744)

[2014-06-30 07:51:45,477] console                           [Info] Caused by: java.lang.ClassNotFoundException: Class 'jetbrains.buildServer.web.plugins.bean.ServerPluginInfo' was not found

[2014-06-30 07:51:45,477] console                           [Info]  at jetbrains.buildServer.plugins.classLoaders.TeamCityClassLoader.doLoadClass(TeamCityClassLoader.java:83)

[2014-06-30 07:51:45,477] console                           [Info]  at jetbrains.buildServer.plugins.classLoaders.TeamCityClassLoader.loadClass(TeamCityClassLoader.java:40)

[2014-06-30 07:51:45,477] console                           [Info]  at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

[2014-06-30 07:51:45,477] console                           [Info]  ... 36 more

[2014-06-30 07:52:01,213] console                           [Info] =======================================================================

[2014-06-30 07:52:01,213] console                           [Info] TeamCity initialized, server UUID: 8e0af842-505c-4060-8856-b946dc08babb

[2014-06-30 07:52:01,213] console                           [Info] TeamCity is running in professional mode

[2014-06-30 07:52:01,431] console                           [Info] [2014-06-30 07:52:01,369]   WARN -   jetbrains.buildServer.SERVER - Removing 1 broken build promotions, probably left from the incorrect server shutdown

0
Comment actions Permalink

I have removed the autoincrementer plugin and config.  Thats stopped error on startup.

However, the webpage still does not load - I can get the login page, but when you click "Log in", it just hangs.

I have copied the mysql db to my local machine and with a bit of config copying, I can run the TC server there ok and bring up the webpage.

I have changed the memory settings to give it more memory, that didnt help either.

Also removed the _trash config.

I think I will see if I can make my local server work properly for now...

0
Comment actions Permalink

Also tried re-installing into a new directory on the server - no joy.

Currently running a disk check.

0
Comment actions Permalink

Hi,

Can you please collect a few server thread dumps while the problem is reproduced?

0
Comment actions Permalink

The thread dump url works ok - attached is a dump after just starting and then another after clicking the login button, which then hangs.



Attachment(s):
threaddump-after-click-login.log.zip
threaddump-just-after-starting.log.zip
0
Comment actions Permalink

Issue raised - http://youtrack.jetbrains.com/issue/TW-37110 - debug logs attached to it.

0
Comment actions Permalink

According to thread dump SQL requests hang. Can you please check which threads are running in MySQL using show processlist.

0
Comment actions Permalink

Thanks for the reply Alina.

I don't see anything but sleeping connections in the processlist - see attached.

I also had the processlist running in the workbench, with a .5 second refresh - I see sql queries briefly coming and going, but nothing hanging around.



Attachment(s):
Temp.zip
0

Please sign in to leave a comment.