Deadlock

I've upgraded Teamcity yesterday from the 5.1 EAP to 5.1 and ever since then my builds are hanging. In the thread dump I see the following

2010-04-23 10:45:03
Full thread dump Java HotSpot(TM) Client VM (16.0-b13 mixed mode):

"Low Memory Detector" daemon prio=6 tid=0x171d3c00 nid=0x2530 runnable [0x00000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x171e3400 nid=0x1cb0 waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Attach Listener" daemon prio=10 tid=0x16d6ac00 nid=0x20d4 waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x16d69400 nid=0x2168 runnable [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=8 tid=0x16d59000 nid=0x260c in Object.wait() [0x16ecf000]
   java.lang.Thread.State: WAITING (on object monitor)
     at java.lang.Object.wait(Native Method)
     - waiting on <0x030702d8> (a java.lang.ref.ReferenceQueue$Lock)
     at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
     - locked <0x030702d8> (a java.lang.ref.ReferenceQueue$Lock)
     at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
     at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x16d54400 nid=0x27b0 waiting for monitor entry [0x16e7e000]
   java.lang.Thread.State: BLOCKED (on object monitor)
     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:293)
     - waiting to lock <0x030703d0> (a sun.misc.Launcher$AppClassLoader)
     at java.lang.ClassLoader.loadClass(ClassLoader.java:296)
     - locked <0x03076cd8> (a com.intellij.rt.coverage.main.CoveragePremain$1)
     at com.intellij.rt.coverage.main.CoveragePremain$1.loadClass(CoveragePremain.java:47)
     - locked <0x03076cd8> (a com.intellij.rt.coverage.main.CoveragePremain$1)
     at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
     at com.intellij.rt.coverage.instrumentation.Instrumentator$1.transform(Instrumentator.java:97)
     at sun.instrument.TransformerManager.transform(TransformerManager.java:169)
     at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:365)
     at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:123)

"main" prio=6 tid=0x003b8400 nid=0x25c8 waiting for monitor entry [0x009de000]
   java.lang.Thread.State: BLOCKED (on object monitor)
     at com.intellij.rt.coverage.instrumentation.Instrumentator$1.transform(Instrumentator.java:97)
     at sun.instrument.TransformerManager.transform(TransformerManager.java:169)
     at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:365)
     at java.lang.ClassLoader.defineClass1(Native Method)
     at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
     at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
     at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
     at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
     at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
     at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
     at java.security.AccessController.doPrivileged(Native Method)
     at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
     at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
     - locked <0x030703d0> (a sun.misc.Launcher$AppClassLoader)
     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
     - locked <0x030703d0> (a sun.misc.Launcher$AppClassLoader)
     at java.lang.ClassLoader.loadClass(ClassLoader.java:248)

"VM Thread" prio=10 tid=0x16d51800 nid=0x2664 runnable 

"VM Periodic Task Thread" prio=10 tid=0x171e6000 nid=0x2600 waiting on condition 

JNI global references: 686


Found one Java-level deadlock:
=============================
"Reference Handler":
  waiting to lock monitor 0x16d5680c (object 0x030703d0, a sun.misc.Launcher$AppClassLoader),
  which is held by "main"
"main":
  waiting to lock monitor 0x16d5820c (object 0x03076cd8, a com.intellij.rt.coverage.main.CoveragePremain$1),
  which is held by "Reference Handler"

Java stack information for the threads listed above:
===================================================
"Reference Handler":
     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:293)
     - waiting to lock <0x030703d0> (a sun.misc.Launcher$AppClassLoader)
     at java.lang.ClassLoader.loadClass(ClassLoader.java:296)
     - locked <0x03076cd8> (a com.intellij.rt.coverage.main.CoveragePremain$1)
     at com.intellij.rt.coverage.main.CoveragePremain$1.loadClass(CoveragePremain.java:47)
     - locked <0x03076cd8> (a com.intellij.rt.coverage.main.CoveragePremain$1)
     at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
     at com.intellij.rt.coverage.instrumentation.Instrumentator$1.transform(Instrumentator.java:97)
     at sun.instrument.TransformerManager.transform(TransformerManager.java:169)
     at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:365)
     at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:123)
"main":
     at com.intellij.rt.coverage.instrumentation.Instrumentator$1.transform(Instrumentator.java:97)
     at sun.instrument.TransformerManager.transform(TransformerManager.java:169)
     at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:365)
     at java.lang.ClassLoader.defineClass1(Native Method)
     at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
     at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
     at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
     at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
     at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
     at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
     at java.security.AccessController.doPrivileged(Native Method)
     at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
     at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
     - locked <0x030703d0> (a sun.misc.Launcher$AppClassLoader)
     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
     - locked <0x030703d0> (a sun.misc.Launcher$AppClassLoader)
     at java.lang.ClassLoader.loadClass(ClassLoader.java:248)

Found 1 deadlock.

What can I do about this?
3 comments

He hangs on "exclude patterns", which is empty.
But I've just put teamcity.maven.disableWatcher=true back as a system property and he does get past it now. Is this related?

0

I've created issue in the tracker: http://youtrack.jetbrains.net/issue/TW-11820
Please provide more details, i.e. the type of the runner, include / exclude patterns used for the coverage.
Also does the workaround with disabling Maven watcher works stable? Personally, I doubt so, because it is unrelated, but everything is possible...

0

You're right. I can't reproduce it as well.

0

Please sign in to leave a comment.