Windows Service starts two instances of TeamCity

TeamCity version: 8.0.4 (build 27616), data format version 628

After upgrading, I know find that starting the service attempts to start 2 instances of TeamCity, one right after another. This of course leads to a database lock error for the second instance and the frequent Database in Use error.

Does anyone have an idea what sort of error in what configuration file might be causing this? I see no previous mention and have searched through config files myself in vain.

Much thanks.

6 comments
Comment actions Permalink

Hi Jon,

Could you please investigate why the server is starting two times?
Some related questions:
Is there only a single service configured i the system?
Is the service the only way the server can be started?
What are the relevant entries in the Windows even log?
What are the command lines and starting data of the server processes started?
If that does not reveal the root cause, could you please attach full set of logs under "<TeamCity server home>\logs" ?

0
Comment actions Permalink

Thank you for replying, Yegor.

I have investigated and so far am unable to see a reason for it to start twice. At the end of this message, I document a solution that seems to cure the UI error if not the actual double-start behavior I see in the logs.

In response to your questions:

1) Is there only a single service configured i the system?

    There is only a single service for TeamCity and a single service for TeamCity BuildAgent configured on the system.
    In Windows Task Manager, I can see that there is only a single process for each after starting.

2) Is the service the only way the server can be started?

    I imagine the server could be started from one of the included batch files in the TC bin directory, but in practice we only start the TC Server using a batch file that starts the service.

3) What are the relevant entries in the Windows even log?

    I see no entries in the Windows Event Log or System Log originating from TeamCity.

4) What are the command lines and starting data of the server processes started?

    TeamCity Server service command:
    "D:\PROG\TeamCity\bin\TeamCityService.exe" jetservice "/settings=D:\PROG\TeamCity\conf\teamcity-server-service.xml" "/LogFile=D:\PROG\TeamCity\logs\teamcity-winservice.log"

    TeamCity Build Agent service command:
    D:\PROG\TeamCity\buildAgent\launcher\bin\TeamCityAgentService-windows-x86-32.exe -s D:\PROG\TeamCity\buildAgent\launcher\conf\wrapper.conf


I am attaching a zip of the logs directory.

A little more information:

We have a batch file that we run each night. It does the following:
  1) Stops all services that use our MS SQL databases, including TeamCity;
     Relevant lines in bat file:
          net stop TCBuildAgent
          net stop TeamCity
  2) Stops MS SQL
  3) Backs up the server and databases
  4) Starts MS SQL
  5) Starts The services that use the databases, including TeamCity.
     Relevant lines in bat file:
          net start TeamCity
          net start TCBuildAgent

This process almost universally results in the "Database In Use" error. If I then manually stop and start the services from the Services Management Console, the error goes away. However, in both cases the logs show that TC tries to start twice, and the second time cannot acquire a lock. I've attached the file TC_TypicalStart.txt which contains the last lines in the teamcity-server.log demonstrating this.

While experimenting with this, I noted that it takes the TeamCity service a very long time to shutdown. So I tried the following: after issuing the net stop command in the batch file, and before calling the command to stop MS SQL, I added a line to add a 2 minute pause between shutting down TeamCity and shutting down the database. This seems so far to remove the "Database in Use" error, the 2 times I have tried it so far.

But the logs still show TeamCity starting twice, and failing to get a db lock the second time. That doesn't seem right. I'd greatly appreciate any thoughts you have on the matter.

Thank you,
Jon



Attachment(s):
TC_TypicalStart.txt.zip
logs.zip
0
Comment actions Permalink

Hi Jon,

Could you please check the list of processes when the eror occurs: how many TeamCity server-related java.exe processes it has? What are the parents of the processes, start time and command line parameters?

From what I see from the logs it seems like there is only a single Tomcat process, but two attempts to start TeamCity web application.
Do you have any customizations of the TeamCity configuration files besides those mentioned in our documentaiton?

Could you please aslo attach all the files in of D:\PROG\TeamCity\conf directory (most importantly, server.xml) ?

0
Comment actions Permalink

Thank you again for your assistance, Yegor.


http://devnet.jetbrains.com/people/yaegor
1) The attached file Debug_TC_double-start.txt documents the TC/java process while the behavior is exhibiting. This is after a server reboot, when the TC services start automatically. This did result in the "Database in Use" error and two TC starts.

2) The only customization I can think of is this in server.xml, to adjust the UI paths for our proxy:

      <Host name="localhost"  appBase="webapps"
            unpackWARs="true" autoDeploy="true">

       <!-- JONATHAN EDIT: MAKE TC ROOT IN SUBDIR /TEAMCITY FOR PROXY PURPOSES: -->
        <Context path="/teamcity"
                 docBase="D:\PROG\TeamCity\webapps\ROOT"
                 debug="1"
                 reloadable="true" >
        </Context>

[ content not shown ]

      </Host>

3) Conf.zip, attached, contains the conf files.

Thank you, I really appreciate your assistance.

Jon



Attachment(s):
conf.zip
Debug_TC_double-start.txt.zip
0
Comment actions Permalink

Jon,

I think the additional context in server.xml is exactly what is causing the double start.

I'd recommend removing the added "Context" from server.xml and renaming webapps\ROOT into webapps\teamcity
Or you can find another Tomcat guide on changing web application context properly.

0
Comment actions Permalink

That was precisely it. Thank you, Yegor.

Jon

0

Please sign in to leave a comment.