Service process exited without service stop request

Hi,

We are using TeamCity since several years and we are running the latest release at all times (we typically upgrade within 5 days of any patch or new version release). This means that we currently run 9.1 (build 36973) which we upgraded about July 16th.

Already the day after the upgrade, and then every 3-5 days the TeamCity server stops. Or actually its server service stops. Without any valid reason, as far as I can tell. When I check in windows services the TeamCity Server service is clearly stopped. In the server.log file there is no explanation. And in Windows event viewer I get the following:

The description for Event ID 404 from source TeamCity (see below) cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.


If the event originated on another computer, the display information had to be saved with the event.


The following information was included with the event:


===============================================================

TeamCity JetBrains JetService v1.1.730.777

c:\TeamCity\bin\TeamCityService.exe

Service process exited without service stop request

===============================================================

This does not provide much lead on what is going on. Some installation details:
* OS is Win 2008 R2 SP1 64-bit
* We use SQL server as DB
* The windows service run with a domain user account (which also acts as local admin on the server)
* No configuration changes have been applied since quite many months
* some java config details:

Java version: 1.8.0_45

Java VM info: Java HotSpot(TM) Server VM (32 bit)

Java Home path: c:\TeamCity\jre

Server: Apache Tomcat/7.0.59

JVM arguments:

-Djava.util.logging.config.file=c:\TeamCity\bin\..\conf\logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dteamcity.git.fetch.separate.process=false -Dteamcity.git.fetch.process.max.memory=1024M -Dteamcity.git.stream.file.threshold.mb=1024M -Xrs -Xmx1300m -XX:MaxPermSize=270m -Dteamcity.configuration.path=../conf/teamcity-startup.properties -Dlog4j.configuration=file:../conf/teamcity-server-log4j.xml -Dteamcity_logs=../logs/ -Djsse.enableSNIExtension=false -Djava.endorsed.dirs=c:\TeamCity\bin\..\endorsed -Dcatalina.base=c:\TeamCity\bin\.. -Dcatalina.home=c:\TeamCity\bin\.. -Djava.io.tmpdir=c:\TeamCity\bin\..\temp

* There is no build agent installed on server (however we have 12 agents in total, running in enterprise license)
* The server stop time is random and ometimes in morning, sometimes in afternoon and sometimes in evening (so far not during night time)
* We are running build tasks almost around the clock
* The solution is to start the service again and wait ten minutes for everything to initialize. However this requires that an admin user is monitoring and available.

I am not sure how to trace this issue further and what information that needs to be collected. If anyone have any ideas of what to test or suggestions on how to trouble shoot I would be quite happy. Currently the issue is mostly annoying since it is not very often, but if the error happens when no-one is monitoring it slows down our teams quite much.


Thanks,
Johan

9 comments
Comment actions Permalink

Hi Johan,

Please attach zipped TeamCity server log folder and also search the disk for the recent files with "hs_err_pid" in the name (for more details please see https://confluence.jetbrains.com/display/TCD9/Reporting+Issues#ReportingIssues-JVMCrashes).
You can create an issue in our tracker and attach logs visible for JetBrains team only.

0
Comment actions Permalink

Great!

The hs_err_pid*.log helped quite a lot. For some reason our JVM runs in 32-bit mode on a 64-bit computer and in the log file it clearly states that it runs out of memory, so I will try to change to 64-bit mode instead and add more memory to JVM (there is quite much available on server).

Thanks for pointing in this direction. Annoying though that the upgrade cause this problem without addressing it.


Regards,
Johan

0
Comment actions Permalink

Glad that the issue is resolved!
We have the related feature request https://youtrack.jetbrains.com/issue/TW-11686 to notify admins about memory problems. Please watch/vote for it.

0
Comment actions Permalink

Thank you for filing the FR.

I would like to add that I do not yet confirm that problem is solved. You pointed me in the direction that memory and bitness was part of the problem, and this have been reconfigured now. But wether this was the only (full) issue, is yet to be determined. Since problem occured every 3-5 days, I need to await a fews days of processing until I can close my case.

And as stated before, my problem started with latest release so in some sense it is a kind of bug that is introduced causing the system to consume more memory suddenly. Also quite annoying is that on a 64-bit environment that default settings is running in 32-bit mode and with very limited memory options. There is a reason we have a powerul sever to run this application, and it seem to be kind of waste if it is not used properly.


Regards,
Johan

0
Comment actions Permalink

Sorry, I misunderstood your previous message. Let's wait for a few days. Please report if the issue is reproduced.
32bit JVM is bundled and used by default because the recommended approach is to start with initial settings and monitor for the percentage of used memory at the Administration | Diagnostics page. If the server uses more then 80% of memory consistently without drops for tens of minutes, that is probably a sign to increase the memory values by another 20%. However if the server load is not that high then there is no need to switch to 64bit JVM. Please find more details here and here.

0
Comment actions Permalink

Hi,

This was new information to me:
"the recommended approach is to start with initial settings and monitor for the percentage of used memory"

But if this is the recommendation, then I suggest that you add the possibility to see the data for several days in this view. Right now the view shows data for a few minutes only. So this recommendation leads to that we should allocate a person to have as its only job to look at this graph constantly. That is way too expensive usage of our resources.


Regards,
Johan

0
Comment actions Permalink

Johan,


Thank you for your feedback. It would be great if you fill a feature request in our tracker, please use this link.

0
Comment actions Permalink

It have now been a few days of running in 64-bit mode without any issues. I'll keep monitoring until beginning of next week before I am happy to close this issue.

0
Comment actions Permalink

Our TeamCity installation have been runnin gsmooth on 64-bit java for some time now so I close this thread.

0

Please sign in to leave a comment.