Clean-up stuck on Executing global cleaner MetadataStorageBean

Hello,

I'm having an issue with my TeamCity install, where the clean-up job is stuck on "Executing global cleaner (11 of 20): MetadataStorageBean". The previous job was hindering the server performance so much, that I had to manually stop the TeamCity Service (it had been running for 215h:54m).

For ference - this TeamCity instance is installed on Windows Server 2012 R2 running on AWS t2.small (1 vCPU, 2 GB vRAM). I have also attached a screenshot showing the Clean-up Settings page.

If there are any logs I can provide, please point me in the right direction and I will post those.

Any help would be greatly appreciated.

Thanks.



Attachment(s):
TeamCityCleanupSettings.png
10 comments

Hello,

What TeamCity version is used? If not one of the latest one, the please consider an upgrade.
If the issue is reproduced on one of the recent versions, than please attach teamcity-cleanup.log and teamcity-server.log.

0

Hi Alina,

I am actually on the current latest version - 9.1.4.

I have attached the following: teamcity-server.log and teamcity-cleanup.log

I am also noticing that the TeamCity CPU usage is stuck at 100%, even though the clean-up isn't running. I've attached "Diagnostics Screenshot.png" for reference.

Thanks,
Jesse



Attachment(s):
Diagnostics Screenshot.png
teamcity-server.log.zip
teamcity-cleanup.log.zip
0

It seems that there is not enough memory. Please try to increase memory settings as suggested in this section.
If it does not help then please attach several server thread dump (you have lots of them auto generated in C:\TeamCity\logs\threadDumps-* directory).

0

Hi Alina,

Increasing to a t2.medium (2 vCPU, 4 GB vRAM) at first made it worse - the TeamCity process was still stuck at 100% CPU usage (both vCPUs maxed) and would not even respond to web requests (or maybe it did, but I just didn't wait past a few minutes). Per the documentation linked, I increased the Java heap memory to 1024mb (-Xmx1024m -XX:MaxPermSize=270m) via the environment variable TEAMCITY_SERVER_MEM_OPTS, with no change in result.

I uninstalled TeamCity and Agent completely, then I manually deleted all artifacts (C:\ProgramData\JetBrains\TeamCity\system\artifacts), a bunch of old logs (C:\TeamCity\logs, leaving only the last 4 months), and old backups (C:\ProgramData\JetBrains\TeamCity\backup). Then, I reinstalled and fired it off - it's now "working" but stuck at 50% CPU usage all of the time. The 1024mb Java heap memory setting is still set.

I haven't seen any OutOfMemoryError, with the exception of this one:

[2015-11-29 20:37:42,377]   WARN - .index.BuildIndexer (metadata) - Index is corrupted, history will be re-indexed: jetbrains.buildServer.serverSide.build.index.BuildIndexException: jetbrains.buildServer.serverSide.db.UnexpectedDBException: Unexpected exception SQLException/HsqlException: SQL error when doing: Querying for list of values
SQL query: select distinct build_id from indexed_build
SQL exception: java.lang.OutOfMemoryError: Java heap space 


I hope the above is just a reference to a previous error and not that it is actually throwing an exception. I have attached a new teamcity-server.log and, per your last message, I have attached a few thread dumps.

Please help.

Thanks,
Jesse



Attachment(s):
ThreadDumps.zip
teamcity-server.log.zip
0

Thank you for provided logs. I was able to anachive only one thread dump (2015-11-10). The hanging thread is:
"TC: 15:54:17 Executing build type trigger: Fresh Content :: Fusion :: Fusion.Robot.Package :: [RC] robot-stable-2015-04-20: Linux, Release {id=Fusion_FusionTools_RcRobotStable20150420LinuxRelease, internal id=bt1411}; 15:54:16 Build Triggers Monitor 1" daemon group="main" prio=5 tid=117 nid=117 runnable
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:152)
    at java.net.SocketInputStream.read(SocketInputStream.java:122)
    at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
    at sun.security.ssl.InputRecord.read(InputRecord.java:480)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
    at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
    at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)
    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
    at org.apache.xmlrpc.DefaultXmlRpcTransport.sendXmlRpc(DefaultXmlRpcTransport.java:87)
    at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:72)
    at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
    at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
    at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
    at com.yandex.teamcity.sandbox.trigger.SandBoxApiHelper.executeXmlRpcMethod(SandBoxApiHelper.java:79)
    at com.yandex.teamcity.sandbox.trigger.SandBoxApiHelper.getLastReleaseId(SandBoxApiHelper.java:99)
    at com.yandex.teamcity.sandbox.trigger.SandBoxMutiResourcesBuildTriggerPolicy.getLatestResourceId(SandBoxMutiResourcesBuildTriggerPolicy.java:260)
    at com.yandex.teamcity.sandbox.trigger.SandBoxMutiResourcesBuildTriggerPolicy.updateResourceChanges(SandBoxMutiResourcesBuildTriggerPolicy.java:191)
    at com.yandex.teamcity.sandbox.trigger.SandBoxMutiResourcesBuildTriggerPolicy.triggerBuild(SandBoxMutiResourcesBuildTriggerPolicy.java:118)
    at jetbrains.buildServer.serverSide.impl.BuildTriggersChecker$4.run(BuildTriggersChecker.java:2)
    at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:89)
    at jetbrains.buildServer.serverSide.impl.BuildTriggersChecker.callTrigger(BuildTriggersChecker.java:50)
    at jetbrains.buildServer.serverSide.impl.BuildTriggersChecker.access$1100(BuildTriggersChecker.java:101)
    at jetbrains.buildServer.serverSide.impl.BuildTriggersChecker$BuildTriggersGroup.processTriggers(BuildTriggersChecker.java:1)
    at jetbrains.buildServer.serverSide.impl.BuildTriggersChecker.triggerBuilds(BuildTriggersChecker.java:95)
    at jetbrains.buildServer.serverSide.impl.BuildServerRunner$4.doSomething(BuildServerRunner.java:2)
    at jetbrains.buildServer.serverSide.impl.BuildServerRunner$BuildServerWorker.runAction(BuildServerRunner.java:34)
    at jetbrains.buildServer.serverSide.impl.BuildServerRunner$BuildServerWorker.run(BuildServerRunner.java:37)
    at java.lang.Thread.run(Thread.java:745)


Do you use third-party plugin installed? Please try to unistall it.
If you still see automatic thread dumps messages in server log, please attach new set of thread dumps.
0

Alina,

I had the Shared Build Number plugin (https://confluence.jetbrains.com/display/TW/TeamCity+Plugins#TeamCityPlugins-ExtendedSettingsforBuildConfigurations) installed, and removed it; then restarted the server, but it's still stuck at 50% CPU usage.

I've attached the latest teamcity-server.log and the most recent thread dumps. I do still see the following entry showing up:

[2015-11-30 15:20:25,459]   WARN - .index.BuildIndexer (metadata) - Index is corrupted, history will be re-indexed: jetbrains.buildServer.serverSide.build.index.BuildIndexException: jetbrains.buildServer.serverSide.db.UnexpectedDBException: Unexpected exception SQLException/HsqlException: SQL error when doing: Querying for list of values
SQL query: select distinct build_id from indexed_build
SQL exception: java.lang.OutOfMemoryError: Java heap space


This might be way off base - but is there any way I can force the above-referenced "history" data to be rebuilt, instead of re-indexed?

Thanks,
Jesse



Attachment(s):
teamcity-threaddumps.zip
teamcity-server.log.zip
0

Bump. Hi Alina, do you have any further suggestions to attempt to fix the issue?

Thanks,
Jesse

0

Sorry for delay. We investigated the issue and found a bug in TeamCity, see the ticket https://youtrack.jetbrains.com/issue/TW-43454.
As current workaround please stop TeamCity server, make sure there are no java processes running, delete <TeamCity data directory>\system\caches\buildsMetadata directory and start server back.
Sorry for the inconvenience.

0

Aha! No problem. I'm just glad it wasn't something I broke unintentionally.

I greatly appreciate your help and dedication to my issue, Alina.

EDIT: Clearing the specified directory (<TeamCity data directory>\system\caches\buildsMetadata) solved the issue!



Thanks much!
Jesse

0

Thank you for your assistance! Glad that everything works for you now.

0

Please sign in to leave a comment.