Deadlock
I've upgraded Teamcity yesterday from the 5.1 EAP to 5.1 and ever since then my builds are hanging. In the thread dump I see the following
2010-04-23 10:45:03 Full thread dump Java HotSpot(TM) Client VM (16.0-b13 mixed mode): "Low Memory Detector" daemon prio=6 tid=0x171d3c00 nid=0x2530 runnable [0x00000000] java.lang.Thread.State: RUNNABLE "CompilerThread0" daemon prio=10 tid=0x171e3400 nid=0x1cb0 waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE "Attach Listener" daemon prio=10 tid=0x16d6ac00 nid=0x20d4 waiting on condition [0x00000000] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" daemon prio=10 tid=0x16d69400 nid=0x2168 runnable [0x00000000] java.lang.Thread.State: RUNNABLE "Finalizer" daemon prio=8 tid=0x16d59000 nid=0x260c in Object.wait() [0x16ecf000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x030702d8> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) - locked <0x030702d8> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) "Reference Handler" daemon prio=10 tid=0x16d54400 nid=0x27b0 waiting for monitor entry [0x16e7e000] java.lang.Thread.State: BLOCKED (on object monitor) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:293) - waiting to lock <0x030703d0> (a sun.misc.Launcher$AppClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:296) - locked <0x03076cd8> (a com.intellij.rt.coverage.main.CoveragePremain$1) at com.intellij.rt.coverage.main.CoveragePremain$1.loadClass(CoveragePremain.java:47) - locked <0x03076cd8> (a com.intellij.rt.coverage.main.CoveragePremain$1) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at com.intellij.rt.coverage.instrumentation.Instrumentator$1.transform(Instrumentator.java:97) at sun.instrument.TransformerManager.transform(TransformerManager.java:169) at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:365) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:123) "main" prio=6 tid=0x003b8400 nid=0x25c8 waiting for monitor entry [0x009de000] java.lang.Thread.State: BLOCKED (on object monitor) at com.intellij.rt.coverage.instrumentation.Instrumentator$1.transform(Instrumentator.java:97) at sun.instrument.TransformerManager.transform(TransformerManager.java:169) at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:365) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632) at java.lang.ClassLoader.defineClass(ClassLoader.java:616) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) at java.net.URLClassLoader.access$000(URLClassLoader.java:58) at java.net.URLClassLoader$1.run(URLClassLoader.java:197) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) - locked <0x030703d0> (a sun.misc.Launcher$AppClassLoader) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) - locked <0x030703d0> (a sun.misc.Launcher$AppClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) "VM Thread" prio=10 tid=0x16d51800 nid=0x2664 runnable "VM Periodic Task Thread" prio=10 tid=0x171e6000 nid=0x2600 waiting on condition JNI global references: 686 Found one Java-level deadlock: ============================= "Reference Handler": waiting to lock monitor 0x16d5680c (object 0x030703d0, a sun.misc.Launcher$AppClassLoader), which is held by "main" "main": waiting to lock monitor 0x16d5820c (object 0x03076cd8, a com.intellij.rt.coverage.main.CoveragePremain$1), which is held by "Reference Handler" Java stack information for the threads listed above: =================================================== "Reference Handler": at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:293) - waiting to lock <0x030703d0> (a sun.misc.Launcher$AppClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:296) - locked <0x03076cd8> (a com.intellij.rt.coverage.main.CoveragePremain$1) at com.intellij.rt.coverage.main.CoveragePremain$1.loadClass(CoveragePremain.java:47) - locked <0x03076cd8> (a com.intellij.rt.coverage.main.CoveragePremain$1) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at com.intellij.rt.coverage.instrumentation.Instrumentator$1.transform(Instrumentator.java:97) at sun.instrument.TransformerManager.transform(TransformerManager.java:169) at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:365) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:123) "main": at com.intellij.rt.coverage.instrumentation.Instrumentator$1.transform(Instrumentator.java:97) at sun.instrument.TransformerManager.transform(TransformerManager.java:169) at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:365) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632) at java.lang.ClassLoader.defineClass(ClassLoader.java:616) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) at java.net.URLClassLoader.access$000(URLClassLoader.java:58) at java.net.URLClassLoader$1.run(URLClassLoader.java:197) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) - locked <0x030703d0> (a sun.misc.Launcher$AppClassLoader) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) - locked <0x030703d0> (a sun.misc.Launcher$AppClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) Found 1 deadlock. What can I do about this?
Please sign in to leave a comment.
He hangs on "exclude patterns", which is empty.
But I've just put teamcity.maven.disableWatcher=true back as a system property and he does get past it now. Is this related?
I've created issue in the tracker: http://youtrack.jetbrains.net/issue/TW-11820
Please provide more details, i.e. the type of the runner, include / exclude patterns used for the coverage.
Also does the workaround with disabling Maven watcher works stable? Personally, I doubt so, because it is unrelated, but everything is possible...
You're right. I can't reproduce it as well.