Skip to content

Instantly share code, notes, and snippets.

@fanf

fanf/threaddump.txt

Created April 14, 2023 06:55
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save fanf/c16e9cd26c70fed50f8cf7930fba3394 to your computer and use it in GitHub Desktop.
Save fanf/c16e9cd26c70fed50f8cf7930fba3394 to your computer and use it in GitHub Desktop.
Download ZIP
ZIO deadlock on multiple unsafe.run and foreachPar
Raw
threaddump.txt
2023-04-14 08:49:40
Full thread dump OpenJDK 64-Bit Server VM (11.0.18+10 mixed mode):
Threads class SMR info:
_java_thread_list=0x00007fd8b00016d0, length=56, elements={
0x00007fda0802c000, 0x00007fda09237800, 0x00007fda0923c000, 0x00007fda09251800,
0x00007fda09253800, 0x00007fda09255800, 0x00007fda09257800, 0x00007fda0925a000,
0x00007fda09746800, 0x00007fda09bb5800, 0x00007fd9b8001000, 0x00007fda09eb5800,
0x00007fda09fdb000, 0x00007fda09fdc000, 0x00007fda09fdd800, 0x00007fda09fdf000,
0x00007fda09fe1000, 0x00007fda09fe2800, 0x00007fda09fe4800, 0x00007fda09fe6000,
0x00007fda0a349800, 0x00007fda0a46e800, 0x00007fda0a768000, 0x00007fda0aba3800,
0x00007fda0aba5000, 0x00007fda0aba6000, 0x00007fda0aba7800, 0x00007fda0aba9800,
0x00007fda0abab000, 0x00007fda0abad000, 0x00007fda0abaf000, 0x00007fda0abb0800,
0x00007fda0abb2800, 0x00007fda0abb4000, 0x00007fda0abb6000, 0x00007fda0abbe800,
0x00007fda0abc0800, 0x00007fda0abc2000, 0x00007fda0abc4000, 0x00007fda0abc6000,
0x00007fda0acf8800, 0x00007fda0acff000, 0x00007fda0ad0e800, 0x00007fda0af86800,
0x00007fda0b1d1800, 0x00007fda0b1c9800, 0x00007fd90c002000, 0x00007fd8d4005000,
0x00007fd8d400a800, 0x00007fd8d400f800, 0x00007fd8d400c000, 0x00007fd978006800,
0x00007fd90c004000, 0x00007fd8a0001800, 0x00007fd978008800, 0x00007fd894001800
}
"main" #1 prio=5 os_prio=0 cpu=12489,09ms elapsed=35325,24s tid=0x00007fda0802c000 nid=0x26d00 in Object.wait() [0x00007fda0cffc000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@11.0.18/Native Method)
- waiting on <0x00000007ef247ed8> (a zio.internal.OneShot)
at java.lang.Object.wait(java.base@11.0.18/Object.java:328)
at zio.internal.OneShot.get(OneShot.scala:83)
- waiting to re-lock in wait() <0x00000007ef247ed8> (a zio.internal.OneShot)
at zio.Runtime$UnsafeAPIV1.run(Runtime.scala:139)
at com.normation.zio$ZioRuntime$.$anonfun$unsafeRun$1(ZioCommons.scala:458)
at com.normation.zio$ZioRuntime$$$Lambda$403/0x0000000800483040.apply(Unknown Source)
at zio.Unsafe$.unsafe(Unsafe.scala:37)
at com.normation.zio$ZioRuntime$.unsafeRun(ZioCommons.scala:458)
at bootstrap.liftweb.RudderConfig$.nodeFactRepository$lzycompute(RudderConfig.scala:1512)
- locked <0x00000007f805d090> (a bootstrap.liftweb.InitLock$)
- locked <0x00000007f6e42d28> (a bootstrap.liftweb.RudderConfig$)
at bootstrap.liftweb.RudderConfig$.nodeFactRepository(RudderConfig.scala:1461)
at bootstrap.liftweb.RudderConfig$.woFactNodeRepository$lzycompute(RudderConfig.scala:2334)
- locked <0x00000007f6e42d28> (a bootstrap.liftweb.RudderConfig$)
at bootstrap.liftweb.RudderConfig$.woFactNodeRepository(RudderConfig.scala:2334)
at bootstrap.liftweb.RudderConfig$.<clinit>(RudderConfig.scala:975)
at bootstrap.liftweb.LiftInitContextListener.contextInitialized(LiftInitContextListener.scala:134)
at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1002)
at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:553)
at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:931)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:764)
at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:360)
at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1445)
at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1409)
at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:844)
at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275)
at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
- locked <0x000000068004ab58> (a java.lang.Object)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
- locked <0x000000068005c7b0> (a java.lang.Object)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
- locked <0x000000068005aee8> (a java.lang.Object)
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
at org.eclipse.jetty.server.Server.start(Server.java:408)
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
at org.eclipse.jetty.server.Server.doStart(Server.java:372)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
- locked <0x000000068004ae90> (a java.lang.Object)
at org.eclipse.jetty.runner.Runner.run(Runner.java:520)
at org.eclipse.jetty.runner.Runner.main(Runner.java:565)
"Reference Handler" #2 daemon prio=10 os_prio=0 cpu=2,01ms elapsed=35325,17s tid=0x00007fda09237800 nid=0x26d08 waiting on condition [0x00007fd9d16fc000]
java.lang.Thread.State: RUNNABLE
at java.lang.ref.Reference.waitForReferencePendingList(java.base@11.0.18/Native Method)
at java.lang.ref.Reference.processPendingReferences(java.base@11.0.18/Reference.java:241)
at java.lang.ref.Reference$ReferenceHandler.run(java.base@11.0.18/Reference.java:213)
"Finalizer" #3 daemon prio=8 os_prio=0 cpu=0,94ms elapsed=35325,17s tid=0x00007fda0923c000 nid=0x26d09 in Object.wait() [0x00007fd9d15fc000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(java.base@11.0.18/Native Method)
- waiting on <0x0000000680009f00> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base@11.0.18/ReferenceQueue.java:155)
- waiting to re-lock in wait() <0x0000000680009f00> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(java.base@11.0.18/ReferenceQueue.java:176)
at java.lang.ref.Finalizer$FinalizerThread.run(java.base@11.0.18/Finalizer.java:170)
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=1,16ms elapsed=35325,16s tid=0x00007fda09251800 nid=0x26d0a runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Service Thread" #5 daemon prio=9 os_prio=0 cpu=0,18ms elapsed=35325,16s tid=0x00007fda09253800 nid=0x26d0b runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 cpu=7669,65ms elapsed=35325,16s tid=0x00007fda09255800 nid=0x26d0c waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
"C1 CompilerThread0" #14 daemon prio=9 os_prio=0 cpu=14347,74ms elapsed=35325,16s tid=0x00007fda09257800 nid=0x26d0d waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
No compile task
"Sweeper thread" #18 daemon prio=9 os_prio=0 cpu=378,23ms elapsed=35325,16s tid=0x00007fda0925a000 nid=0x26d0e runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Common-Cleaner" #19 daemon prio=8 os_prio=0 cpu=4,02ms elapsed=35325,06s tid=0x00007fda09746800 nid=0x26d12 in Object.wait() [0x00007fd9d08fe000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(java.base@11.0.18/Native Method)
- waiting on <no object reference available>
at java.lang.ref.ReferenceQueue.remove(java.base@11.0.18/ReferenceQueue.java:155)
- waiting to re-lock in wait() <0x0000000680028500> (a java.lang.ref.ReferenceQueue$Lock)
at jdk.internal.ref.CleanerImpl.run(java.base@11.0.18/CleanerImpl.java:148)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
at jdk.internal.misc.InnocuousThread.run(java.base@11.0.18/InnocuousThread.java:161)
"Monitor Ctrl-Break" #20 daemon prio=5 os_prio=0 cpu=20,17ms elapsed=35324,71s tid=0x00007fda09bb5800 nid=0x26d15 runnable [0x00007fd9d03fe000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(java.base@11.0.18/Native Method)
at java.net.SocketInputStream.socketRead(java.base@11.0.18/SocketInputStream.java:115)
at java.net.SocketInputStream.read(java.base@11.0.18/SocketInputStream.java:168)
at java.net.SocketInputStream.read(java.base@11.0.18/SocketInputStream.java:140)
at sun.nio.cs.StreamDecoder.readBytes(java.base@11.0.18/StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(java.base@11.0.18/StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(java.base@11.0.18/StreamDecoder.java:178)
- locked <0x000000068000a3e0> (a java.io.InputStreamReader)
at java.io.InputStreamReader.read(java.base@11.0.18/InputStreamReader.java:181)
at java.io.BufferedReader.fill(java.base@11.0.18/BufferedReader.java:161)
at java.io.BufferedReader.readLine(java.base@11.0.18/BufferedReader.java:326)
- locked <0x000000068000a3e0> (a java.io.InputStreamReader)
at java.io.BufferedReader.readLine(java.base@11.0.18/BufferedReader.java:392)
at com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:53)
"Attach Listener" #21 daemon prio=9 os_prio=0 cpu=58,45ms elapsed=35324,19s tid=0x00007fd9b8001000 nid=0x26d17 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"logback-1" #22 daemon prio=5 os_prio=0 cpu=7,21ms elapsed=35323,65s tid=0x00007fda09eb5800 nid=0x26d1b waiting on condition [0x00007fd9935fe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
- parking to wait for <0x0000000680016960> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1177)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
"qtp1239183618-24" #24 prio=5 os_prio=0 cpu=759,49ms elapsed=35322,83s tid=0x00007fda09fdb000 nid=0x26d1c waiting on condition [0x00007fd993ffd000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
- parking to wait for <0x0000000680000328> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
"qtp1239183618-25" #25 prio=5 os_prio=0 cpu=1,94ms elapsed=35322,83s tid=0x00007fda09fdc000 nid=0x26d1d waiting on condition [0x00007fd993bfd000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
- parking to wait for <0x0000000680000328> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
"qtp1239183618-26" #26 prio=5 os_prio=0 cpu=1,34ms elapsed=35322,83s tid=0x00007fda09fdd800 nid=0x26d1e waiting on condition [0x00007fd992ffd000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
- parking to wait for <0x0000000680000328> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
"qtp1239183618-27" #27 prio=5 os_prio=0 cpu=1,33ms elapsed=35322,83s tid=0x00007fda09fdf000 nid=0x26d1f waiting on condition [0x00007fd9929fd000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
- parking to wait for <0x0000000680000328> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
"qtp1239183618-28" #28 prio=5 os_prio=0 cpu=0,92ms elapsed=35322,83s tid=0x00007fda09fe1000 nid=0x26d20 waiting on condition [0x00007fd9928fd000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
- parking to wait for <0x0000000680000328> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
"qtp1239183618-29" #29 prio=5 os_prio=0 cpu=0,82ms elapsed=35322,83s tid=0x00007fda09fe2800 nid=0x26d21 waiting on condition [0x00007fd9927fd000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
- parking to wait for <0x0000000680000328> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
"qtp1239183618-30" #30 prio=5 os_prio=0 cpu=1,51ms elapsed=35322,83s tid=0x00007fda09fe4800 nid=0x26d22 waiting on condition [0x00007fd9926fd000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
- parking to wait for <0x0000000680000328> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
"qtp1239183618-31" #31 prio=5 os_prio=0 cpu=0,98ms elapsed=35322,83s tid=0x00007fda09fe6000 nid=0x26d23 waiting on condition [0x00007fd9925fd000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
- parking to wait for <0x0000000680000328> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
"Session-HouseKeeper-781f10f2-1" #32 prio=5 os_prio=0 cpu=0,65ms elapsed=35320,88s tid=0x00007fda0a349800 nid=0x26d34 waiting on condition [0x00007fd990ee1000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
- parking to wait for <0x00000007f766e420> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1182)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
"BC Entropy Daemon" #34 daemon prio=5 os_prio=0 cpu=5,35ms elapsed=35320,10s tid=0x00007fda0a46e800 nid=0x26d35 waiting on condition [0x00007fd990be2000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@11.0.18/Native Method)
at org.bouncycastle.jcajce.provider.drbg.DRBG$EntropyDaemon.run(Unknown Source)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
"logback-2" #35 daemon prio=5 os_prio=0 cpu=23,83ms elapsed=35319,46s tid=0x00007fda0a768000 nid=0x26d36 waiting on condition [0x00007fd9901fd000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
- parking to wait for <0x0000000680016960> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1182)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
"ZScheduler-Worker-0" #36 daemon prio=5 os_prio=0 cpu=4,36ms elapsed=35318,68s tid=0x00007fda0aba3800 nid=0x26d37 in Object.wait() [0x00007fd9924fd000]
java.lang.Thread.State: RUNNABLE
at bootstrap.liftweb.RudderConfig$$$Lambda$1395/0x00000008008b1040.apply(Unknown Source)
at zio.ZIO$.$anonfun$foreachPar$7(ZIO.scala:5843)
at zio.ZIO$$$Lambda$1403/0x00000008008b5040.apply(Unknown Source)
at zio.ZIO$.$anonfun$foreachParDiscard$4(ZIO.scala:5861)
at zio.ZIO$$$Lambda$1427/0x00000008008c8840.apply(Unknown Source)
at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:890)
at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:1024)
at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:967)
at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:967)
at zio.internal.FiberRuntime.evaluateEffect(FiberRuntime.scala:381)
at zio.internal.FiberRuntime.evaluateMessageWhileSuspended(FiberRuntime.scala:504)
at zio.internal.FiberRuntime.drainQueueOnCurrentThread(FiberRuntime.scala:220)
at zio.internal.FiberRuntime.run(FiberRuntime.scala:139)
at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:476)
"ZScheduler-Worker-1" #37 daemon prio=5 os_prio=0 cpu=2,59ms elapsed=35318,68s tid=0x00007fda0aba5000 nid=0x26d38 in Object.wait() [0x00007fd9923fd000]
java.lang.Thread.State: RUNNABLE
at bootstrap.liftweb.RudderConfig$$$Lambda$1395/0x00000008008b1040.apply(Unknown Source)
at zio.ZIO$.$anonfun$foreachPar$7(ZIO.scala:5843)
at zio.ZIO$$$Lambda$1403/0x00000008008b5040.apply(Unknown Source)
at zio.ZIO$.$anonfun$foreachParDiscard$4(ZIO.scala:5861)
at zio.ZIO$$$Lambda$1427/0x00000008008c8840.apply(Unknown Source)
at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:890)
at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:1024)
at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:967)
at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:967)
at zio.internal.FiberRuntime.evaluateEffect(FiberRuntime.scala:381)
at zio.internal.FiberRuntime.evaluateMessageWhileSuspended(FiberRuntime.scala:504)
at zio.internal.FiberRuntime.drainQueueOnCurrentThread(FiberRuntime.scala:220)
at zio.internal.FiberRuntime.run(FiberRuntime.scala:139)
at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:476)
"ZScheduler-Worker-2" #38 daemon prio=5 os_prio=0 cpu=2,00ms elapsed=35318,68s tid=0x00007fda0aba6000 nid=0x26d39 in Object.wait() [0x00007fd9900fd000]
java.lang.Thread.State: RUNNABLE
at bootstrap.liftweb.RudderConfig$$$Lambda$1395/0x00000008008b1040.apply(Unknown Source)
at zio.ZIO$.$anonfun$foreachPar$7(ZIO.scala:5843)
at zio.ZIO$$$Lambda$1403/0x00000008008b5040.apply(Unknown Source)
at zio.ZIO$.$anonfun$foreachParDiscard$4(ZIO.scala:5861)
at zio.ZIO$$$Lambda$1427/0x00000008008c8840.apply(Unknown Source)
at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:890)
at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:1024)
at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:967)
at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:967)
at zio.internal.FiberRuntime.evaluateEffect(FiberRuntime.scala:381)
at zio.internal.FiberRuntime.evaluateMessageWhileSuspended(FiberRuntime.scala:504)
at zio.internal.FiberRuntime.drainQueueOnCurrentThread(FiberRuntime.scala:220)
at zio.internal.FiberRuntime.run(FiberRuntime.scala:139)
at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:476)
"ZScheduler-Worker-3" #39 daemon prio=5 os_prio=0 cpu=1,98ms elapsed=35318,68s tid=0x00007fda0aba7800 nid=0x26d3a in Object.wait() [0x00007fd9135fd000]
java.lang.Thread.State: RUNNABLE
at bootstrap.liftweb.RudderConfig$$$Lambda$1395/0x00000008008b1040.apply(Unknown Source)
at zio.ZIO$.$anonfun$foreachPar$7(ZIO.scala:5843)
at zio.ZIO$$$Lambda$1403/0x00000008008b5040.apply(Unknown Source)
at zio.ZIO$.$anonfun$foreachParDiscard$4(ZIO.scala:5861)
at zio.ZIO$$$Lambda$1427/0x00000008008c8840.apply(Unknown Source)
at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:890)
at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:1024)
at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:967)
at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:967)
at zio.internal.FiberRuntime.evaluateEffect(FiberRuntime.scala:381)
at zio.internal.FiberRuntime.evaluateMessageWhileSuspended(FiberRuntime.scala:504)
at zio.internal.FiberRuntime.drainQueueOnCurrentThread(FiberRuntime.scala:220)
at zio.internal.FiberRuntime.run(FiberRuntime.scala:139)
at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:476)
"ZScheduler-Worker-4" #40 daemon prio=5 os_prio=0 cpu=0,49ms elapsed=35318,68s tid=0x00007fda0aba9800 nid=0x26d3b waiting on condition [0x00007fd9134fe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)
"ZScheduler-Worker-5" #41 daemon prio=5 os_prio=0 cpu=2,08ms elapsed=35318,68s tid=0x00007fda0abab000 nid=0x26d3c waiting on condition [0x00007fd9133fe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)
"ZScheduler-Worker-6" #42 daemon prio=5 os_prio=0 cpu=0,36ms elapsed=35318,68s tid=0x00007fda0abad000 nid=0x26d3d waiting on condition [0x00007fd9132fe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)
"ZScheduler-Worker-7" #43 daemon prio=5 os_prio=0 cpu=0,22ms elapsed=35318,68s tid=0x00007fda0abaf000 nid=0x26d3e waiting on condition [0x00007fd9131fe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)
"ZScheduler-Worker-8" #44 daemon prio=5 os_prio=0 cpu=0,38ms elapsed=35318,68s tid=0x00007fda0abb0800 nid=0x26d3f waiting on condition [0x00007fd9130fe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)
"ZScheduler-Worker-9" #45 daemon prio=5 os_prio=0 cpu=0,38ms elapsed=35318,68s tid=0x00007fda0abb2800 nid=0x26d40 waiting on condition [0x00007fd912ffe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)
"ZScheduler-Worker-10" #46 daemon prio=5 os_prio=0 cpu=0,23ms elapsed=35318,68s tid=0x00007fda0abb4000 nid=0x26d41 waiting on condition [0x00007fd912efe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)
"ZScheduler-Worker-11" #47 daemon prio=5 os_prio=0 cpu=0,28ms elapsed=35318,68s tid=0x00007fda0abb6000 nid=0x26d42 waiting on condition [0x00007fd912dfe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)
"ZScheduler-Worker-12" #48 daemon prio=5 os_prio=0 cpu=0,31ms elapsed=35318,68s tid=0x00007fda0abbe800 nid=0x26d43 waiting on condition [0x00007fd912cfe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)
"ZScheduler-Worker-13" #49 daemon prio=5 os_prio=0 cpu=0,28ms elapsed=35318,68s tid=0x00007fda0abc0800 nid=0x26d44 waiting on condition [0x00007fd912bfe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)
"ZScheduler-Worker-14" #50 daemon prio=5 os_prio=0 cpu=0,27ms elapsed=35318,68s tid=0x00007fda0abc2000 nid=0x26d45 waiting on condition [0x00007fd912afe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)
"ZScheduler-Worker-15" #51 daemon prio=5 os_prio=0 cpu=0,33ms elapsed=35318,68s tid=0x00007fda0abc4000 nid=0x26d46 waiting on condition [0x00007fd9129fe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)
"ZScheduler-Supervisor" #52 daemon prio=5 os_prio=0 cpu=291,48ms elapsed=35318,68s tid=0x00007fda0abc6000 nid=0x26d47 waiting on condition [0x00007fd9128fe000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
at java.util.concurrent.locks.LockSupport.parkUntil(java.base@11.0.18/LockSupport.java:391)
at zio.internal.ZScheduler$$anon$3.run(ZScheduler.scala:316)
"Write Timeout Handler Timer" #53 daemon prio=5 os_prio=0 cpu=511,03ms elapsed=35318,55s tid=0x00007fda0acf8800 nid=0x26d48 in Object.wait() [0x00007fd9125fe000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(java.base@11.0.18/Native Method)
- waiting on <no object reference available>
at java.util.TimerThread.mainLoop(java.base@11.0.18/Timer.java:553)
- waiting to re-lock in wait() <0x00000007f7713e80> (a java.util.TaskQueue)
at java.util.TimerThread.run(java.base@11.0.18/Timer.java:506)
"Connection reader for connection 0 in pool 'rudder-authenticated-ro' to localhost:1389" #55 daemon prio=5 os_prio=0 cpu=426,76ms elapsed=35318,55s tid=0x00007fda0acff000 nid=0x26d4a runnable [0x00007fd9124fe000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(java.base@11.0.18/Native Method)
at java.net.SocketInputStream.socketRead(java.base@11.0.18/SocketInputStream.java:115)
at java.net.SocketInputStream.read(java.base@11.0.18/SocketInputStream.java:168)
at java.net.SocketInputStream.read(java.base@11.0.18/SocketInputStream.java:140)
at java.io.BufferedInputStream.fill(java.base@11.0.18/BufferedInputStream.java:252)
at java.io.BufferedInputStream.read(java.base@11.0.18/BufferedInputStream.java:271)
- locked <0x00000007f76dd9d8> (a java.io.BufferedInputStream)
at com.unboundid.asn1.ASN1StreamReader.read(ASN1StreamReader.java:1199)
at com.unboundid.asn1.ASN1StreamReader.readType(ASN1StreamReader.java:360)
at com.unboundid.asn1.ASN1StreamReader.beginSequence(ASN1StreamReader.java:1118)
at com.unboundid.ldap.protocol.LDAPMessage.readLDAPResponseFrom(LDAPMessage.java:1198)
at com.unboundid.ldap.sdk.LDAPConnectionReader.run(LDAPConnectionReader.java:242)
"Health Check Thread for LDAPConnectionPool(serverSet=SingleServerSet(server=localhost:1389, includesAuthentication=false, includesPostConnectProcessing=false), maxConnections=2)" #56 daemon prio=5 os_prio=0 cpu=3,07ms elapsed=35318,50s tid=0x00007fda0ad0e800 nid=0x26d4b waiting on condition [0x00007fd9123fd000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
- parking to wait for <0x00000007f77efe98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
at java.util.concurrent.LinkedBlockingQueue.poll(java.base@11.0.18/LinkedBlockingQueue.java:458)
at com.unboundid.ldap.sdk.LDAPConnectionPoolHealthCheckThread.run(LDAPConnectionPoolHealthCheckThread.java:137)
"HikariPool-1 housekeeper" #57 daemon prio=5 os_prio=0 cpu=4,91ms elapsed=35317,90s tid=0x00007fda0af86800 nid=0x26d4c waiting on condition [0x00007fd911cfd000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
- parking to wait for <0x00000007f77b8d40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1182)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
"Connection reader for connection 1 in pool 'rudder-authenticated-rw' to localhost:1389" #64 daemon prio=5 os_prio=0 cpu=8,46ms elapsed=35317,19s tid=0x00007fda0b1d1800 nid=0x26d56 runnable [0x00007fd9115fe000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(java.base@11.0.18/Native Method)
at java.net.SocketInputStream.socketRead(java.base@11.0.18/SocketInputStream.java:115)
at java.net.SocketInputStream.read(java.base@11.0.18/SocketInputStream.java:168)
at java.net.SocketInputStream.read(java.base@11.0.18/SocketInputStream.java:140)
at java.io.BufferedInputStream.fill(java.base@11.0.18/BufferedInputStream.java:252)
at java.io.BufferedInputStream.read(java.base@11.0.18/BufferedInputStream.java:271)
- locked <0x00000007f7783240> (a java.io.BufferedInputStream)
at com.unboundid.asn1.ASN1StreamReader.read(ASN1StreamReader.java:1199)
at com.unboundid.asn1.ASN1StreamReader.readType(ASN1StreamReader.java:360)
at com.unboundid.asn1.ASN1StreamReader.beginSequence(ASN1StreamReader.java:1118)
at com.unboundid.ldap.protocol.LDAPMessage.readLDAPResponseFrom(LDAPMessage.java:1198)
at com.unboundid.ldap.sdk.LDAPConnectionReader.run(LDAPConnectionReader.java:242)
"Health Check Thread for LDAPConnectionPool(serverSet=SingleServerSet(server=localhost:1389, includesAuthentication=false, includesPostConnectProcessing=false), maxConnections=2)" #65 daemon prio=5 os_prio=0 cpu=2,35ms elapsed=35317,19s tid=0x00007fda0b1c9800 nid=0x26d57 waiting on condition [0x00007fd9112fd000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
- parking to wait for <0x00000007f77140d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
at java.util.concurrent.LinkedBlockingQueue.poll(java.base@11.0.18/LinkedBlockingQueue.java:458)
at com.unboundid.ldap.sdk.LDAPConnectionPoolHealthCheckThread.run(LDAPConnectionPoolHealthCheckThread.java:137)
"logback-3" #67 daemon prio=5 os_prio=0 cpu=1,73ms elapsed=35314,46s tid=0x00007fd90c002000 nid=0x26d5e waiting on condition [0x00007fd910afe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
- parking to wait for <0x0000000680016960> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1177)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
"ZScheduler-0" #68 daemon prio=5 os_prio=0 cpu=0,29ms elapsed=35309,98s tid=0x00007fd8d4005000 nid=0x26d61 waiting on condition [0x00007fd911bfe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)
"ZScheduler-1" #69 daemon prio=5 os_prio=0 cpu=0,29ms elapsed=35309,98s tid=0x00007fd8d400a800 nid=0x26d62 waiting on condition [0x00007fd9104fe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)
"ZScheduler-2" #70 daemon prio=5 os_prio=0 cpu=0,31ms elapsed=35309,98s tid=0x00007fd8d400f800 nid=0x26d63 waiting on condition [0x00007fd9103fe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)
"ZScheduler-3" #71 daemon prio=5 os_prio=0 cpu=0,28ms elapsed=35309,98s tid=0x00007fd8d400c000 nid=0x26d64 waiting on condition [0x00007fd9102fe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)
"logback-4" #72 daemon prio=5 os_prio=0 cpu=1,81ms elapsed=35309,46s tid=0x00007fd978006800 nid=0x26d65 waiting on condition [0x00007fd9101fe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
- parking to wait for <0x0000000680016960> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1177)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
"logback-5" #73 daemon prio=5 os_prio=0 cpu=1,38ms elapsed=35304,46s tid=0x00007fd90c004000 nid=0x26d67 waiting on condition [0x00007fd9d01fc000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
- parking to wait for <0x0000000680016960> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1177)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
"logback-6" #74 daemon prio=5 os_prio=0 cpu=1,29ms elapsed=35299,46s tid=0x00007fd8a0001800 nid=0x26d68 waiting on condition [0x00007fd9d0bfe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
- parking to wait for <0x0000000680016960> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1177)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
"logback-7" #75 daemon prio=5 os_prio=0 cpu=1,53ms elapsed=35294,46s tid=0x00007fd978008800 nid=0x26d6c waiting on condition [0x00007fd9109fe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
- parking to wait for <0x0000000680016960> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1177)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
"logback-8" #76 daemon prio=5 os_prio=0 cpu=1,15ms elapsed=35289,46s tid=0x00007fd894001800 nid=0x26d6d waiting on condition [0x00007fd9116fe000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
- parking to wait for <0x0000000680016960> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1177)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
"VM Thread" os_prio=0 cpu=59,93ms elapsed=35325,18s tid=0x00007fda0922f800 nid=0x26d07 runnable
"GC Thread#0" os_prio=0 cpu=76,16ms elapsed=35325,24s tid=0x00007fda08045800 nid=0x26d01 runnable
"GC Thread#1" os_prio=0 cpu=65,89ms elapsed=35320,94s tid=0x00007fd9bc071000 nid=0x26d26 runnable
"GC Thread#2" os_prio=0 cpu=59,53ms elapsed=35320,94s tid=0x00007fd9bc072800 nid=0x26d27 runnable
"GC Thread#3" os_prio=0 cpu=49,84ms elapsed=35320,94s tid=0x00007fd9bc074000 nid=0x26d28 runnable
"GC Thread#4" os_prio=0 cpu=49,98ms elapsed=35320,94s tid=0x00007fd9bc075800 nid=0x26d29 runnable
"GC Thread#5" os_prio=0 cpu=68,39ms elapsed=35320,94s tid=0x00007fd9bc077800 nid=0x26d2a runnable
"GC Thread#6" os_prio=0 cpu=46,51ms elapsed=35320,94s tid=0x00007fd9bc079000 nid=0x26d2b runnable
"GC Thread#7" os_prio=0 cpu=48,79ms elapsed=35320,94s tid=0x00007fd9bc07a800 nid=0x26d2c runnable
"GC Thread#8" os_prio=0 cpu=63,71ms elapsed=35320,94s tid=0x00007fd9bc07c000 nid=0x26d2d runnable
"GC Thread#9" os_prio=0 cpu=62,03ms elapsed=35320,94s tid=0x00007fd9bc07d800 nid=0x26d2e runnable
"GC Thread#10" os_prio=0 cpu=48,83ms elapsed=35320,94s tid=0x00007fd9bc07f000 nid=0x26d2f runnable
"GC Thread#11" os_prio=0 cpu=62,77ms elapsed=35320,94s tid=0x00007fd9bc080800 nid=0x26d30 runnable
"GC Thread#12" os_prio=0 cpu=63,02ms elapsed=35320,93s tid=0x00007fd9bc082000 nid=0x26d31 runnable
"G1 Main Marker" os_prio=0 cpu=3,63ms elapsed=35325,24s tid=0x00007fda080a4000 nid=0x26d02 runnable
"G1 Conc#0" os_prio=0 cpu=58,81ms elapsed=35325,24s tid=0x00007fda080a6000 nid=0x26d03 runnable
"G1 Conc#1" os_prio=0 cpu=56,04ms elapsed=35320,91s tid=0x00007fd9d4001000 nid=0x26d32 runnable
"G1 Conc#2" os_prio=0 cpu=58,71ms elapsed=35320,91s tid=0x00007fd9d4002800 nid=0x26d33 runnable
"G1 Refine#0" os_prio=0 cpu=8,95ms elapsed=35325,21s tid=0x00007fda0915e000 nid=0x26d04 runnable
"G1 Young RemSet Sampling" os_prio=0 cpu=147,88ms elapsed=35325,20s tid=0x00007fda09160000 nid=0x26d05 runnable
"StrDedup" os_prio=0 cpu=40,53ms elapsed=35325,20s tid=0x00007fda09168000 nid=0x26d06 runnable
"VM Periodic Task Thread" os_prio=0 cpu=372,16ms elapsed=35324,71s tid=0x00007fda09cc0000 nid=0x26d16 waiting on condition
JNI global refs: 25, weak refs: 0
@fanf
Copy link
Author

fanf commented Apr 14, 2023

Env: ZIO 2.0.12 on jvm 11 and scala 2.13.10

The logic:

  • LiftInitContextListener is early called in the war init
  • it unsafe runs an init method from our RudderConfig object where all our services (classes and objects) are instantiated, mostly referenced as lazy vals
  • that init method call triggers RudderConfig instantiation, which triggers instantiation of the required lazy val in the needed order
  • one of this service is nodeFactRepository: the class needs some effectful code to get instanciated, and that code is unsafe run

Observations:

  • in the nodeFactRepository code, if I switch the foreach into a foreachPar (highlighted line), I get a deadlock while I have progression without it
  • I put signal handler to try to get the fiber dump on SIGUSR2 : I'm not able to trigger it. I also registered java threaddump on SIGUSR1 and that one works, letting me things that the deadlock is somewhere in ZIO scheduler, even if we see in the threaddump that threre's lot of ZScheduler, ZScheduler-Worker, ZScheduler-Supervisor threads waiting

LiftInitContextListener.scala
image

RudderConfig.scala
image

ZioCommons.scala
image

image

@fanf
Copy link
Author

fanf commented Apr 14, 2023 •
edited

Interestingly (I guess?) if I change unsafeRun to add the blocking tag, I get the same deadlock on the very first call of unsafeRun, with a similar thread dump (but simpler):
image

The main difference is the presence of zio-default-blocking related to LDAPConnectionProvider.scala:82
image

2023-04-14 09:34:49
Full thread dump OpenJDK 64-Bit Server VM (11.0.18+10 mixed mode):

Threads class SMR info:
_java_thread_list=0x00007fd58aa12040, length=50, elements={
0x00007fd58802c000, 0x00007fd589237800, 0x00007fd58923c000, 0x00007fd589251800,
0x00007fd589253800, 0x00007fd589255800, 0x00007fd589257800, 0x00007fd58925a000,
0x00007fd589557000, 0x00007fd589b32000, 0x00007fd589cbd000, 0x00007fd589ea1000,
0x00007fd589ea2800, 0x00007fd589ea4800, 0x00007fd589ea6800, 0x00007fd589ea8800,
0x00007fd589eaa000, 0x00007fd589eac000, 0x00007fd589eae000, 0x00007fd538001000,
0x00007fd58a0be000, 0x00007fd58a1e7800, 0x00007fd58a316000, 0x00007fd58a8d0800,
0x00007fd58a8d2800, 0x00007fd58a8d4000, 0x00007fd58a8d6000, 0x00007fd58a8d7800,
0x00007fd58a8d9800, 0x00007fd58a8db800, 0x00007fd58a8dd000, 0x00007fd58a8df000,
0x00007fd58a8e0800, 0x00007fd58a8e2800, 0x00007fd58a8e8000, 0x00007fd58a8ea000,
0x00007fd58a8eb800, 0x00007fd58a8ed800, 0x00007fd58a8ef800, 0x00007fd58a8f1800,
0x00007fd440002000, 0x00007fd444048000, 0x00007fd444056800, 0x00007fd44406d000,
0x00007fd500002800, 0x00007fd48c001800, 0x00007fd500003800, 0x00007fd52087c000,
0x00007fd48c002800, 0x00007fd51cb0d000
}

"main" #1 prio=5 os_prio=0 cpu=2249,51ms elapsed=115,88s tid=0x00007fd58802c000 nid=0x279c9 in Object.wait()  [0x00007fd58d314000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(java.base@11.0.18/Native Method)
	- waiting on <0x00000007fe2cd4d0> (a zio.internal.OneShot)
	at java.lang.Object.wait(java.base@11.0.18/Object.java:328)
	at zio.internal.OneShot.get(OneShot.scala:83)
	- waiting to re-lock in wait() <0x00000007fe2cd4d0> (a zio.internal.OneShot)
	at zio.Runtime$UnsafeAPIV1.run(Runtime.scala:139)
	at com.normation.zio$ZioRuntime$.$anonfun$unsafeRun$1(ZioCommons.scala:458)
	at com.normation.zio$ZioRuntime$$$Lambda$403/0x0000000800483040.apply(Unknown Source)
	at zio.Unsafe$.unsafe(Unsafe.scala:37)
	at com.normation.zio$ZioRuntime$.unsafeRun(ZioCommons.scala:458)
	at bootstrap.liftweb.RudderConfig$.<clinit>(RudderConfig.scala:966)
	at bootstrap.liftweb.LiftInitContextListener.contextInitialized(LiftInitContextListener.scala:134)
	at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1002)
	at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:553)
	at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:931)
	at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:764)
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:360)
	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1445)
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1409)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:844)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275)
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
	- locked <0x00000007fac5c6b8> (a java.lang.Object)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
	- locked <0x00000007fac6b140> (a java.lang.Object)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
	- locked <0x00000007fac6aac0> (a java.lang.Object)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.server.Server.start(Server.java:408)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.server.Server.doStart(Server.java:372)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
	- locked <0x00000007fac5ca38> (a java.lang.Object)
	at org.eclipse.jetty.runner.Runner.run(Runner.java:520)
	at org.eclipse.jetty.runner.Runner.main(Runner.java:565)

"Reference Handler" #2 daemon prio=10 os_prio=0 cpu=0,96ms elapsed=115,85s tid=0x00007fd589237800 nid=0x279d1 waiting on condition  [0x00007fd55c1b8000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.ref.Reference.waitForReferencePendingList(java.base@11.0.18/Native Method)
	at java.lang.ref.Reference.processPendingReferences(java.base@11.0.18/Reference.java:241)
	at java.lang.ref.Reference$ReferenceHandler.run(java.base@11.0.18/Reference.java:213)

"Finalizer" #3 daemon prio=8 os_prio=0 cpu=0,54ms elapsed=115,85s tid=0x00007fd58923c000 nid=0x279d2 in Object.wait()  [0x00007fd5557fe000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(java.base@11.0.18/Native Method)
	- waiting on <0x00000007fac3ac68> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.18/ReferenceQueue.java:155)
	- waiting to re-lock in wait() <0x00000007fac3ac68> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.18/ReferenceQueue.java:176)
	at java.lang.ref.Finalizer$FinalizerThread.run(java.base@11.0.18/Finalizer.java:170)

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=0,79ms elapsed=115,84s tid=0x00007fd589251800 nid=0x279d3 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Service Thread" #5 daemon prio=9 os_prio=0 cpu=0,10ms elapsed=115,84s tid=0x00007fd589253800 nid=0x279d4 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 cpu=1460,44ms elapsed=115,84s tid=0x00007fd589255800 nid=0x279d5 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"C1 CompilerThread0" #14 daemon prio=9 os_prio=0 cpu=3052,56ms elapsed=115,84s tid=0x00007fd589257800 nid=0x279d6 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task

"Sweeper thread" #18 daemon prio=9 os_prio=0 cpu=41,37ms elapsed=115,84s tid=0x00007fd58925a000 nid=0x279d7 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Common-Cleaner" #19 daemon prio=8 os_prio=0 cpu=1,01ms elapsed=115,80s tid=0x00007fd589557000 nid=0x279db in Object.wait()  [0x00007fd554afe000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(java.base@11.0.18/Native Method)
	- waiting on <no object reference available>
	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.18/ReferenceQueue.java:155)
	- waiting to re-lock in wait() <0x00000007fac58678> (a java.lang.ref.ReferenceQueue$Lock)
	at jdk.internal.ref.CleanerImpl.run(java.base@11.0.18/CleanerImpl.java:148)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
	at jdk.internal.misc.InnocuousThread.run(java.base@11.0.18/InnocuousThread.java:161)

"Monitor Ctrl-Break" #20 daemon prio=5 os_prio=0 cpu=11,98ms elapsed=115,66s tid=0x00007fd589b32000 nid=0x279de runnable  [0x00007fd5547fe000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(java.base@11.0.18/Native Method)
	at java.net.SocketInputStream.socketRead(java.base@11.0.18/SocketInputStream.java:115)
	at java.net.SocketInputStream.read(java.base@11.0.18/SocketInputStream.java:168)
	at java.net.SocketInputStream.read(java.base@11.0.18/SocketInputStream.java:140)
	at sun.nio.cs.StreamDecoder.readBytes(java.base@11.0.18/StreamDecoder.java:284)
	at sun.nio.cs.StreamDecoder.implRead(java.base@11.0.18/StreamDecoder.java:326)
	at sun.nio.cs.StreamDecoder.read(java.base@11.0.18/StreamDecoder.java:178)
	- locked <0x00000007fac23d00> (a java.io.InputStreamReader)
	at java.io.InputStreamReader.read(java.base@11.0.18/InputStreamReader.java:181)
	at java.io.BufferedReader.fill(java.base@11.0.18/BufferedReader.java:161)
	at java.io.BufferedReader.readLine(java.base@11.0.18/BufferedReader.java:326)
	- locked <0x00000007fac23d00> (a java.io.InputStreamReader)
	at java.io.BufferedReader.readLine(java.base@11.0.18/BufferedReader.java:392)
	at com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:53)

"logback-1" #21 daemon prio=5 os_prio=0 cpu=7,01ms elapsed=115,44s tid=0x00007fd589cbd000 nid=0x279e0 waiting on condition  [0x00007fd50bffe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007facb0820> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1177)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)

"qtp574268151-23" #23 prio=5 os_prio=0 cpu=395,04ms elapsed=115,19s tid=0x00007fd589ea1000 nid=0x279e1 waiting on condition  [0x00007fd50bbfe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007fac75ee8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)

"qtp574268151-24" #24 prio=5 os_prio=0 cpu=0,87ms elapsed=115,19s tid=0x00007fd589ea2800 nid=0x279e2 waiting on condition  [0x00007fd50b5fe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007fac75ee8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)

"qtp574268151-25" #25 prio=5 os_prio=0 cpu=0,62ms elapsed=115,19s tid=0x00007fd589ea4800 nid=0x279e3 waiting on condition  [0x00007fd50b1fe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007fac75ee8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)

"qtp574268151-26" #26 prio=5 os_prio=0 cpu=0,64ms elapsed=115,19s tid=0x00007fd589ea6800 nid=0x279e4 waiting on condition  [0x00007fd50abfe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007fac75ee8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)

"qtp574268151-27" #27 prio=5 os_prio=0 cpu=0,47ms elapsed=115,19s tid=0x00007fd589ea8800 nid=0x279e5 waiting on condition  [0x00007fd50aafe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007fac75ee8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)

"qtp574268151-28" #28 prio=5 os_prio=0 cpu=0,51ms elapsed=115,19s tid=0x00007fd589eaa000 nid=0x279e6 waiting on condition  [0x00007fd50a9fe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007fac75ee8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)

"qtp574268151-29" #29 prio=5 os_prio=0 cpu=0,29ms elapsed=115,19s tid=0x00007fd589eac000 nid=0x279e7 waiting on condition  [0x00007fd50a8fe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007fac75ee8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)

"qtp574268151-30" #30 prio=5 os_prio=0 cpu=0,27ms elapsed=115,19s tid=0x00007fd589eae000 nid=0x279e8 waiting on condition  [0x00007fd50a7fe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007fac75ee8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)

"Attach Listener" #31 daemon prio=9 os_prio=0 cpu=3,47ms elapsed=114,85s tid=0x00007fd538001000 nid=0x279ea waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Session-HouseKeeper-e98770d-1" #32 prio=5 os_prio=0 cpu=0,18ms elapsed=114,21s tid=0x00007fd58a0be000 nid=0x279fa waiting on condition  [0x00007fd5090e2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007fac76110> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1182)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)

"BC Entropy Daemon" #34 daemon prio=5 os_prio=0 cpu=0,99ms elapsed=113,87s tid=0x00007fd58a1e7800 nid=0x279fb waiting on condition  [0x00007fd508de2000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(java.base@11.0.18/Native Method)
	at org.bouncycastle.jcajce.provider.drbg.DRBG$EntropyDaemon.run(Unknown Source)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)

"logback-2" #35 daemon prio=5 os_prio=0 cpu=1,33ms elapsed=113,57s tid=0x00007fd58a316000 nid=0x279fc waiting on condition  [0x00007fd5083fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007facb0820> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1177)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)

"ZScheduler-Worker-0" #36 daemon prio=5 os_prio=0 cpu=0,63ms elapsed=113,16s tid=0x00007fd58a8d0800 nid=0x279fd waiting on condition  [0x00007fd5080fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)

"ZScheduler-Worker-1" #37 daemon prio=5 os_prio=0 cpu=0,49ms elapsed=113,16s tid=0x00007fd58a8d2800 nid=0x279fe waiting on condition  [0x00007fd4937fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)

"ZScheduler-Worker-2" #38 daemon prio=5 os_prio=0 cpu=0,26ms elapsed=113,16s tid=0x00007fd58a8d4000 nid=0x279ff waiting on condition  [0x00007fd4936fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)

"ZScheduler-Worker-3" #39 daemon prio=5 os_prio=0 cpu=0,30ms elapsed=113,16s tid=0x00007fd58a8d6000 nid=0x27a00 waiting on condition  [0x00007fd4935fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)

"ZScheduler-Worker-4" #40 daemon prio=5 os_prio=0 cpu=0,39ms elapsed=113,16s tid=0x00007fd58a8d7800 nid=0x27a01 waiting on condition  [0x00007fd4934fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)

"ZScheduler-Worker-5" #41 daemon prio=5 os_prio=0 cpu=0,40ms elapsed=113,16s tid=0x00007fd58a8d9800 nid=0x27a02 waiting on condition  [0x00007fd4933fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)

"ZScheduler-Worker-6" #42 daemon prio=5 os_prio=0 cpu=0,29ms elapsed=113,16s tid=0x00007fd58a8db800 nid=0x27a03 waiting on condition  [0x00007fd4932fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)

"ZScheduler-Worker-7" #43 daemon prio=5 os_prio=0 cpu=0,26ms elapsed=113,15s tid=0x00007fd58a8dd000 nid=0x27a04 waiting on condition  [0x00007fd4931fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)

"ZScheduler-Worker-8" #44 daemon prio=5 os_prio=0 cpu=0,17ms elapsed=113,15s tid=0x00007fd58a8df000 nid=0x27a05 waiting on condition  [0x00007fd4930fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)

"ZScheduler-Worker-9" #45 daemon prio=5 os_prio=0 cpu=0,22ms elapsed=113,15s tid=0x00007fd58a8e0800 nid=0x27a06 waiting on condition  [0x00007fd492ffe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)

"ZScheduler-Worker-10" #46 daemon prio=5 os_prio=0 cpu=0,23ms elapsed=113,16s tid=0x00007fd58a8e2800 nid=0x27a07 waiting on condition  [0x00007fd492efe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)

"ZScheduler-Worker-11" #47 daemon prio=5 os_prio=0 cpu=0,17ms elapsed=113,16s tid=0x00007fd58a8e8000 nid=0x27a08 waiting on condition  [0x00007fd492dfe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)

"ZScheduler-Worker-12" #48 daemon prio=5 os_prio=0 cpu=0,11ms elapsed=113,16s tid=0x00007fd58a8ea000 nid=0x27a09 waiting on condition  [0x00007fd492cfe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)

"ZScheduler-Worker-13" #49 daemon prio=5 os_prio=0 cpu=0,19ms elapsed=113,16s tid=0x00007fd58a8eb800 nid=0x27a0a waiting on condition  [0x00007fd492bfe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)

"ZScheduler-Worker-14" #50 daemon prio=5 os_prio=0 cpu=0,22ms elapsed=113,16s tid=0x00007fd58a8ed800 nid=0x27a0b waiting on condition  [0x00007fd492afe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)

"ZScheduler-Worker-15" #51 daemon prio=5 os_prio=0 cpu=0,12ms elapsed=113,15s tid=0x00007fd58a8ef800 nid=0x27a0c waiting on condition  [0x00007fd4929fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)

"ZScheduler-Supervisor" #52 daemon prio=5 os_prio=0 cpu=47,86ms elapsed=113,15s tid=0x00007fd58a8f1800 nid=0x27a0d waiting on condition  [0x00007fd4928fe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.parkUntil(java.base@11.0.18/LockSupport.java:391)
	at zio.internal.ZScheduler$$anon$3.run(ZScheduler.scala:316)

"zio-default-blocking-2" #54 daemon prio=5 os_prio=0 cpu=33,07ms elapsed=113,08s tid=0x00007fd440002000 nid=0x27a0f in Object.wait()  [0x00007fd4924fc000]
   java.lang.Thread.State: RUNNABLE
	at bootstrap.liftweb.RudderConfig$$$Lambda$419/0x000000080048c840.apply(Unknown Source)
	at com.normation.ldap.sdk.LDAPConnectionProvider.$anonfun$map$2(LDAPConnectionProvider.scala:82)
	at com.normation.ldap.sdk.LDAPConnectionProvider$$Lambda$462/0x00000008004c7440.apply(Unknown Source)
	at zio.ZIOCompanionVersionSpecific.$anonfun$attempt$1(ZIOCompanionVersionSpecific.scala:100)
	at zio.ZIOCompanionVersionSpecific$$Lambda$454/0x00000008004c2440.apply(Unknown Source)
	at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:904)
	at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:890)
	at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:967)
	at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:967)
	at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:1024)
	at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:967)
	at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:890)
	at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:1024)
	at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:967)
	at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:967)
	at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:1024)
	at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:967)
	at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:890)
	at zio.internal.FiberRuntime.evaluateEffect(FiberRuntime.scala:381)
	at zio.internal.FiberRuntime.evaluateMessageWhileSuspended(FiberRuntime.scala:504)
	at zio.internal.FiberRuntime.drainQueueOnCurrentThread(FiberRuntime.scala:220)
	at zio.internal.FiberRuntime.run(FiberRuntime.scala:139)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)

"Write Timeout Handler Timer" #55 daemon prio=5 os_prio=0 cpu=47,95ms elapsed=113,06s tid=0x00007fd444048000 nid=0x27a10 in Object.wait()  [0x00007fd4923fe000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(java.base@11.0.18/Native Method)
	- waiting on <no object reference available>
	at java.util.TimerThread.mainLoop(java.base@11.0.18/Timer.java:553)
	- waiting to re-lock in wait() <0x00000007fe1713a8> (a java.util.TaskQueue)
	at java.util.TimerThread.run(java.base@11.0.18/Timer.java:506)

"Connection reader for connection 0 in pool 'rudder-authenticated-ro' to localhost:1389" #57 daemon prio=5 os_prio=0 cpu=26,49ms elapsed=113,06s tid=0x00007fd444056800 nid=0x27a12 runnable  [0x00007fd4922fe000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(java.base@11.0.18/Native Method)
	at java.net.SocketInputStream.socketRead(java.base@11.0.18/SocketInputStream.java:115)
	at java.net.SocketInputStream.read(java.base@11.0.18/SocketInputStream.java:168)
	at java.net.SocketInputStream.read(java.base@11.0.18/SocketInputStream.java:140)
	at java.io.BufferedInputStream.fill(java.base@11.0.18/BufferedInputStream.java:252)
	at java.io.BufferedInputStream.read(java.base@11.0.18/BufferedInputStream.java:271)
	- locked <0x00000007fe18ecd8> (a java.io.BufferedInputStream)
	at com.unboundid.asn1.ASN1StreamReader.read(ASN1StreamReader.java:1199)
	at com.unboundid.asn1.ASN1StreamReader.readType(ASN1StreamReader.java:360)
	at com.unboundid.asn1.ASN1StreamReader.beginSequence(ASN1StreamReader.java:1118)
	at com.unboundid.ldap.protocol.LDAPMessage.readLDAPResponseFrom(LDAPMessage.java:1198)
	at com.unboundid.ldap.sdk.LDAPConnectionReader.run(LDAPConnectionReader.java:242)

"Health Check Thread for LDAPConnectionPool(serverSet=SingleServerSet(server=localhost:1389, includesAuthentication=false, includesPostConnectProcessing=false), maxConnections=2)" #58 daemon prio=5 os_prio=0 cpu=1,24ms elapsed=113,03s tid=0x00007fd44406d000 nid=0x27a13 waiting on condition  [0x00007fd4921fe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007fdefa630> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
	at java.util.concurrent.LinkedBlockingQueue.poll(java.base@11.0.18/LinkedBlockingQueue.java:458)
	at com.unboundid.ldap.sdk.LDAPConnectionPoolHealthCheckThread.run(LDAPConnectionPoolHealthCheckThread.java:137)

"logback-3" #59 daemon prio=5 os_prio=0 cpu=0,94ms elapsed=108,57s tid=0x00007fd500002800 nid=0x27a16 waiting on condition  [0x00007fd554cfe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007facb0820> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1177)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)

"logback-4" #60 daemon prio=5 os_prio=0 cpu=0,79ms elapsed=103,57s tid=0x00007fd48c001800 nid=0x27a17 waiting on condition  [0x00007fd554dfe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007facb0820> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1177)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)

"logback-5" #61 daemon prio=5 os_prio=0 cpu=0,38ms elapsed=98,57s tid=0x00007fd500003800 nid=0x27a1a waiting on condition  [0x00007fd5549fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007facb0820> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1177)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)

"logback-6" #62 daemon prio=5 os_prio=0 cpu=0,39ms elapsed=93,57s tid=0x00007fd52087c000 nid=0x27a31 waiting on condition  [0x00007fd5548fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007facb0820> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1177)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)

"logback-7" #63 daemon prio=5 os_prio=0 cpu=0,41ms elapsed=88,57s tid=0x00007fd48c002800 nid=0x27a33 waiting on condition  [0x00007fd554bfe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007facb0820> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1177)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)

"logback-8" #64 daemon prio=5 os_prio=0 cpu=2,23ms elapsed=83,57s tid=0x00007fd51cb0d000 nid=0x27a47 waiting on condition  [0x00007fd50a4fe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007facb0820> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1182)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)

"VM Thread" os_prio=0 cpu=17,99ms elapsed=115,85s tid=0x00007fd58922f800 nid=0x279d0 runnable  

"GC Thread#0" os_prio=0 cpu=24,53ms elapsed=115,88s tid=0x00007fd588045800 nid=0x279ca runnable  

"GC Thread#1" os_prio=0 cpu=15,39ms elapsed=114,24s tid=0x00007fd53c001000 nid=0x279ec runnable  

"GC Thread#2" os_prio=0 cpu=25,44ms elapsed=114,24s tid=0x00007fd53c002800 nid=0x279ed runnable  

"GC Thread#3" os_prio=0 cpu=15,97ms elapsed=114,24s tid=0x00007fd53c004000 nid=0x279ee runnable  

"GC Thread#4" os_prio=0 cpu=15,85ms elapsed=114,24s tid=0x00007fd53c006000 nid=0x279ef runnable  

"GC Thread#5" os_prio=0 cpu=16,23ms elapsed=114,24s tid=0x00007fd53c007800 nid=0x279f0 runnable  

"GC Thread#6" os_prio=0 cpu=17,76ms elapsed=114,24s tid=0x00007fd53c009000 nid=0x279f1 runnable  

"GC Thread#7" os_prio=0 cpu=16,04ms elapsed=114,24s tid=0x00007fd53c00a800 nid=0x279f2 runnable  

"GC Thread#8" os_prio=0 cpu=17,67ms elapsed=114,24s tid=0x00007fd53c00c000 nid=0x279f3 runnable  

"GC Thread#9" os_prio=0 cpu=18,49ms elapsed=114,24s tid=0x00007fd53c00d800 nid=0x279f4 runnable  

"GC Thread#10" os_prio=0 cpu=16,00ms elapsed=114,24s tid=0x00007fd53c00f000 nid=0x279f5 runnable  

"GC Thread#11" os_prio=0 cpu=19,17ms elapsed=114,24s tid=0x00007fd53c010800 nid=0x279f6 runnable  

"GC Thread#12" os_prio=0 cpu=22,14ms elapsed=114,24s tid=0x00007fd53c012000 nid=0x279f7 runnable  

"G1 Main Marker" os_prio=0 cpu=0,92ms elapsed=115,88s tid=0x00007fd5880a4000 nid=0x279cb runnable  

"G1 Conc#0" os_prio=0 cpu=25,28ms elapsed=115,88s tid=0x00007fd5880a6000 nid=0x279cc runnable  

"G1 Conc#1" os_prio=0 cpu=24,99ms elapsed=114,22s tid=0x00007fd558001000 nid=0x279f8 runnable  

"G1 Conc#2" os_prio=0 cpu=24,89ms elapsed=114,22s tid=0x00007fd558002800 nid=0x279f9 runnable  

"G1 Refine#0" os_prio=0 cpu=0,19ms elapsed=115,86s tid=0x00007fd58915e000 nid=0x279cd runnable  

"G1 Young RemSet Sampling" os_prio=0 cpu=14,80ms elapsed=115,86s tid=0x00007fd589160000 nid=0x279ce runnable  
"StrDedup" os_prio=0 cpu=0,18ms elapsed=115,86s tid=0x00007fd589168000 nid=0x279cf runnable  

"VM Periodic Task Thread" os_prio=0 cpu=53,66ms elapsed=115,66s tid=0x00007fd589b34800 nid=0x279df waiting on condition  

JNI global refs: 24, weak refs: 0

@fanf
Copy link
Author

fanf commented Apr 14, 2023

Some more tests: it does not seems to be an init problem with nodeFactRepository class because :

  • I put out of the init the getFacts method (the one with the faulty foreachPar
  • I cherry picked only the need services and force their init early
    -I checked that I can call methods on them and unsafe run and it returns
  • I built by end inputs for getFacts
  • I call getFacts just after the needed classes init
  • it locks.
  • amazingly, if the input size is one, it runs, but as low as 2 (even with parallelism at 4), it locks;

image

@fanf
Copy link
Author

fanf commented Apr 14, 2023

Even simple foreachPar blocks:
image

Threadump for it:

Full thread dump

"main" #1 prio=5 os_prio=0 cpu=2279,61ms elapsed=117,41s tid=0x00007f29b802c000 nid=0x33726 in Object.wait()  [0x00007f29bcd15000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(java.base@11.0.18/Native Method)
	- waiting on <0x00000007fbeead78> (a zio.internal.OneShot)
	at java.lang.Object.wait(java.base@11.0.18/Object.java:328)
	at zio.internal.OneShot.get(OneShot.scala:83)
	- waiting to re-lock in wait() <0x00000007fbeead78> (a zio.internal.OneShot)
	at zio.Runtime$UnsafeAPIV1.run(Runtime.scala:139)
	at bootstrap.liftweb.RudderConfig$.$anonfun$new$4(RudderConfig.scala:390)
	at bootstrap.liftweb.RudderConfig$$$Lambda$208/0x0000000800358840.apply(Unknown Source)
	at zio.Unsafe$.unsafe(Unsafe.scala:37)
	at bootstrap.liftweb.RudderConfig$.<clinit>(RudderConfig.scala:388)
	at bootstrap.liftweb.LiftInitContextListener.contextInitialized(LiftInitContextListener.scala:134)
	at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1002)
	at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:553)
	at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:931)
	at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:764)
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:360)
	at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1445)
	at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1409)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:844)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275)
	at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
	- locked <0x00000007f4000558> (a java.lang.Object)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
	- locked <0x00000007f30091c8> (a java.lang.Object)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
	- locked <0x00000007f3009240> (a java.lang.Object)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
	at org.eclipse.jetty.server.Server.start(Server.java:408)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)
	at org.eclipse.jetty.server.Server.doStart(Server.java:372)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
	- locked <0x00000007f3009350> (a java.lang.Object)
	at org.eclipse.jetty.runner.Runner.run(Runner.java:520)
	at org.eclipse.jetty.runner.Runner.main(Runner.java:565)


"zio-default-blocking-2" #54 daemon prio=5 os_prio=0 cpu=12,44ms elapsed=114,57s tid=0x00007f287c002000 nid=0x3376d in Object.wait()  [0x00007f28c6afd000]
   java.lang.Thread.State: RUNNABLE
	at bootstrap.liftweb.RudderConfig$$$Lambda$242/0x000000080037d040.apply$mcI$sp(Unknown Source)
	at zio.ZIO$.$anonfun$withParallelism$1(ZIO.scala:4981)
	at zio.ZIO$$$Lambda$243/0x000000080037d440.apply(Unknown Source)
	at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:904)
	at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:890)
	at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:1024)
	at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:967)
	at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:890)
	at zio.internal.FiberRuntime.evaluateEffect(FiberRuntime.scala:381)
	at zio.internal.FiberRuntime.evaluateMessageWhileSuspended(FiberRuntime.scala:504)
	at zio.internal.FiberRuntime.drainQueueOnCurrentThread(FiberRuntime.scala:220)
	at zio.internal.FiberRuntime.run(FiberRuntime.scala:139)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)


"qtp574268151-23" #23 prio=5 os_prio=0 cpu=429,68ms elapsed=116,56s tid=0x00007f29b9ffa800 nid=0x3373e waiting on condition  [0x00007f2943bfe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007f4005b30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)


"qtp574268151-24" #24 prio=5 os_prio=0 cpu=0,79ms elapsed=116,56s tid=0x00007f29b9ffc000 nid=0x3373f waiting on condition  [0x00007f29437fe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007f4005b30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)


"qtp574268151-25" #25 prio=5 os_prio=0 cpu=0,52ms elapsed=116,56s tid=0x00007f29b9ffe000 nid=0x33740 waiting on condition  [0x00007f2942ffe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007f4005b30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)


"qtp574268151-26" #26 prio=5 os_prio=0 cpu=0,62ms elapsed=116,56s tid=0x00007f29b9fff800 nid=0x33741 waiting on condition  [0x00007f2942bfe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007f4005b30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)


"qtp574268151-27" #27 prio=5 os_prio=0 cpu=0,27ms elapsed=116,56s tid=0x00007f29ba001800 nid=0x33742 waiting on condition  [0x00007f29425fe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007f4005b30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)


"qtp574268151-28" #28 prio=5 os_prio=0 cpu=0,18ms elapsed=116,56s tid=0x00007f29ba003000 nid=0x33743 waiting on condition  [0x00007f29424fe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007f4005b30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)


"qtp574268151-29" #29 prio=5 os_prio=0 cpu=0,41ms elapsed=116,56s tid=0x00007f29ba005000 nid=0x33744 waiting on condition  [0x00007f29423fe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007f4005b30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)


"qtp574268151-30" #30 prio=5 os_prio=0 cpu=0,11ms elapsed=116,56s tid=0x00007f29ba007000 nid=0x33745 waiting on condition  [0x00007f29422fe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007f4005b30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:875)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:925)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)


"Common-Cleaner" #19 daemon prio=8 os_prio=0 cpu=1,04ms elapsed=117,30s tid=0x00007f29b95c6800 nid=0x33737 in Object.wait()  [0x00007f29806fe000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(java.base@11.0.18/Native Method)
	- waiting on <0x00000007f400a058> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.18/ReferenceQueue.java:155)
	- waiting to re-lock in wait() <0x00000007f400a058> (a java.lang.ref.ReferenceQueue$Lock)
	at jdk.internal.ref.CleanerImpl.run(java.base@11.0.18/CleanerImpl.java:148)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)
	at jdk.internal.misc.InnocuousThread.run(java.base@11.0.18/InnocuousThread.java:161)


"ZScheduler-Worker-0" #36 daemon prio=5 os_prio=0 cpu=0,65ms elapsed=114,67s tid=0x00007f29ba66d800 nid=0x3375b waiting on condition  [0x00007f28c7efe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)


"ZScheduler-Worker-1" #37 daemon prio=5 os_prio=0 cpu=0,43ms elapsed=114,67s tid=0x00007f29ba66f000 nid=0x3375c waiting on condition  [0x00007f28c7dfe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)


"ZScheduler-Worker-2" #38 daemon prio=5 os_prio=0 cpu=0,39ms elapsed=114,67s tid=0x00007f29ba671000 nid=0x3375d waiting on condition  [0x00007f28c7cfe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)


"ZScheduler-Worker-3" #39 daemon prio=5 os_prio=0 cpu=0,35ms elapsed=114,67s tid=0x00007f29ba672800 nid=0x3375e waiting on condition  [0x00007f28c7bfe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)


"ZScheduler-Worker-4" #40 daemon prio=5 os_prio=0 cpu=0,31ms elapsed=114,67s tid=0x00007f29ba674800 nid=0x3375f waiting on condition  [0x00007f28c7afe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)


"ZScheduler-Worker-5" #41 daemon prio=5 os_prio=0 cpu=0,23ms elapsed=114,67s tid=0x00007f29ba676800 nid=0x33760 waiting on condition  [0x00007f28c79fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)


"ZScheduler-Worker-6" #42 daemon prio=5 os_prio=0 cpu=0,31ms elapsed=114,67s tid=0x00007f29ba678000 nid=0x33761 waiting on condition  [0x00007f28c78fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)


"ZScheduler-Worker-7" #43 daemon prio=5 os_prio=0 cpu=0,31ms elapsed=114,67s tid=0x00007f29ba67a000 nid=0x33762 waiting on condition  [0x00007f28c77fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)


"ZScheduler-Worker-8" #44 daemon prio=5 os_prio=0 cpu=0,23ms elapsed=114,67s tid=0x00007f29ba67b800 nid=0x33763 waiting on condition  [0x00007f28c76fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)


"ZScheduler-Worker-9" #45 daemon prio=5 os_prio=0 cpu=0,18ms elapsed=114,67s tid=0x00007f29ba67d800 nid=0x33764 waiting on condition  [0x00007f28c75fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)


"ZScheduler-Worker-10" #46 daemon prio=5 os_prio=0 cpu=0,12ms elapsed=114,67s tid=0x00007f29ba67f800 nid=0x33765 waiting on condition  [0x00007f28c74fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)


"ZScheduler-Worker-11" #47 daemon prio=5 os_prio=0 cpu=0,18ms elapsed=114,67s tid=0x00007f29ba681000 nid=0x33766 waiting on condition  [0x00007f28c73fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)


"ZScheduler-Worker-12" #48 daemon prio=5 os_prio=0 cpu=0,11ms elapsed=114,67s tid=0x00007f29ba683000 nid=0x33767 waiting on condition  [0x00007f28c72fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)


"ZScheduler-Worker-13" #49 daemon prio=5 os_prio=0 cpu=0,24ms elapsed=114,67s tid=0x00007f29ba685000 nid=0x33768 waiting on condition  [0x00007f28c71fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)


"ZScheduler-Worker-14" #50 daemon prio=5 os_prio=0 cpu=0,17ms elapsed=114,67s tid=0x00007f29ba687000 nid=0x33769 waiting on condition  [0x00007f28c70fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)


"ZScheduler-Worker-15" #51 daemon prio=5 os_prio=0 cpu=0,13ms elapsed=114,67s tid=0x00007f29ba689000 nid=0x3376a waiting on condition  [0x00007f28c6ffe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:323)
	at zio.internal.ZScheduler$$anon$4.run(ZScheduler.scala:451)


"ZScheduler-Supervisor" #52 daemon prio=5 os_prio=0 cpu=58,13ms elapsed=114,67s tid=0x00007f29ba68b000 nid=0x3376b waiting on condition  [0x00007f28c6efe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	at java.util.concurrent.locks.LockSupport.parkUntil(java.base@11.0.18/LockSupport.java:391)
	at zio.internal.ZScheduler$$anon$3.run(ZScheduler.scala:316)


"Monitor Ctrl-Break" #20 daemon prio=5 os_prio=0 cpu=8,26ms elapsed=117,13s tid=0x00007f29b9980800 nid=0x3373a runnable  [0x00007f29803fe000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(java.base@11.0.18/Native Method)
	at java.net.SocketInputStream.socketRead(java.base@11.0.18/SocketInputStream.java:115)
	at java.net.SocketInputStream.read(java.base@11.0.18/SocketInputStream.java:168)
	at java.net.SocketInputStream.read(java.base@11.0.18/SocketInputStream.java:140)
	at sun.nio.cs.StreamDecoder.readBytes(java.base@11.0.18/StreamDecoder.java:284)
	at sun.nio.cs.StreamDecoder.implRead(java.base@11.0.18/StreamDecoder.java:326)
	at sun.nio.cs.StreamDecoder.read(java.base@11.0.18/StreamDecoder.java:178)
	- locked <0x00000007f4006488> (a java.io.InputStreamReader)
	at java.io.InputStreamReader.read(java.base@11.0.18/InputStreamReader.java:181)
	at java.io.BufferedReader.fill(java.base@11.0.18/BufferedReader.java:161)
	at java.io.BufferedReader.readLine(java.base@11.0.18/BufferedReader.java:326)
	- locked <0x00000007f4006488> (a java.io.InputStreamReader)
	at java.io.BufferedReader.readLine(java.base@11.0.18/BufferedReader.java:392)
	at com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:53)


"logback-1" #21 daemon prio=5 os_prio=0 cpu=8,25ms elapsed=116,81s tid=0x00007f29b9d88800 nid=0x3373d waiting on condition  [0x00007f2943ffe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007f40022a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1182)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)


"Session-HouseKeeper-e98770d-1" #32 prio=5 os_prio=0 cpu=0,19ms elapsed=115,51s tid=0x00007f29ba284000 nid=0x33757 waiting on condition  [0x00007f29801fc000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007ffe7a980> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.18/LockSupport.java:234)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.18/AbstractQueuedSynchronizer.java:2123)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1182)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)


"BC Entropy Daemon" #34 daemon prio=5 os_prio=0 cpu=1,08ms elapsed=115,09s tid=0x00007f29ba3a8000 nid=0x33759 waiting on condition  [0x00007f29420fe000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(java.base@11.0.18/Native Method)
	at org.bouncycastle.jcajce.provider.drbg.DRBG$EntropyDaemon.run(Unknown Source)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)


"logback-2" #35 daemon prio=5 os_prio=0 cpu=1,73ms elapsed=114,71s tid=0x00007f29ba650000 nid=0x3375a waiting on condition  [0x00007f28c7ffe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007f40022a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1177)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)


"logback-3" #55 daemon prio=5 os_prio=0 cpu=0,90ms elapsed=109,71s tid=0x00007f2928002800 nid=0x3376e waiting on condition  [0x00007f29805fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007f40022a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1177)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)


"logback-4" #56 daemon prio=5 os_prio=0 cpu=0,90ms elapsed=104,71s tid=0x00007f28c0001000 nid=0x33776 waiting on condition  [0x00007f29807fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007f40022a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1177)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)


"logback-5" #57 daemon prio=5 os_prio=0 cpu=0,41ms elapsed=99,71s tid=0x00007f2928004000 nid=0x33779 waiting on condition  [0x00007f29804fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007f40022a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1177)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)


"logback-6" #58 daemon prio=5 os_prio=0 cpu=0,49ms elapsed=94,71s tid=0x00007f2948847000 nid=0x33781 waiting on condition  [0x00007f29808fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007f40022a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1177)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)


"logback-7" #59 daemon prio=5 os_prio=0 cpu=0,42ms elapsed=89,71s tid=0x00007f28c0002800 nid=0x33787 waiting on condition  [0x00007f29421fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007f40022a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1177)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)


"logback-8" #60 daemon prio=5 os_prio=0 cpu=0,38ms elapsed=84,71s tid=0x00007f29516ca800 nid=0x33789 waiting on condition  [0x00007f28c69fe000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park(java.base@11.0.18/Native Method)
	- parking to wait for  <0x00000007f40022a0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.18/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.18/AbstractQueuedSynchronizer.java:2081)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:1177)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(java.base@11.0.18/ScheduledThreadPoolExecutor.java:899)
	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.18/ThreadPoolExecutor.java:1054)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.18/ThreadPoolExecutor.java:1114)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.18/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)


"Reference Handler" #2 daemon prio=10 os_prio=0 cpu=0,70ms elapsed=117,36s tid=0x00007f29b9237800 nid=0x3372e waiting on condition  [0x00007f29813f6000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.ref.Reference.waitForReferencePendingList(java.base@11.0.18/Native Method)
	at java.lang.ref.Reference.processPendingReferences(java.base@11.0.18/Reference.java:241)
	at java.lang.ref.Reference$ReferenceHandler.run(java.base@11.0.18/Reference.java:213)


"Finalizer" #3 daemon prio=8 os_prio=0 cpu=0,60ms elapsed=117,36s tid=0x00007f29b923c000 nid=0x3372f in Object.wait()  [0x00007f29812f6000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(java.base@11.0.18/Native Method)
	- waiting on <0x00000007f400b408> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.18/ReferenceQueue.java:155)
	- waiting to re-lock in wait() <0x00000007f400b408> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(java.base@11.0.18/ReferenceQueue.java:176)
	at java.lang.ref.Finalizer$FinalizerThread.run(java.base@11.0.18/Finalizer.java:170)


"Signal Dispatcher" #4 daemon prio=9 os_prio=0 cpu=0,41ms elapsed=117,35s tid=0x00007f29b9251800 nid=0x33730 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE


"Service Thread" #5 daemon prio=9 os_prio=0 cpu=0,10ms elapsed=117,35s tid=0x00007f29b9253800 nid=0x33731 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE


"C2 CompilerThread0" #6 daemon prio=9 os_prio=0 cpu=2036,35ms elapsed=117,35s tid=0x00007f29b9255800 nid=0x33732 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task


"C1 CompilerThread0" #14 daemon prio=9 os_prio=0 cpu=2985,92ms elapsed=117,35s tid=0x00007f29b9257800 nid=0x33733 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   No compile task


"Sweeper thread" #18 daemon prio=9 os_prio=0 cpu=45,36ms elapsed=117,35s tid=0x00007f29b925a000 nid=0x33734 runnable  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE


"Attach Listener" #31 daemon prio=9 os_prio=0 cpu=4,42ms elapsed=116,37s tid=0x00007f2968001000 nid=0x33748 waiting on condition  [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE


"VM Thread" os_prio=0 cpu=15,97ms elapsed=117,37s tid=0x00007f29b922f800 nid=0x3372d runnable  


"GC Thread#0" os_prio=0 cpu=11,02ms elapsed=117,41s tid=0x00007f29b8045800 nid=0x33727 runnable  


"GC Thread#1" os_prio=0 cpu=8,66ms elapsed=115,55s tid=0x00007f296c031000 nid=0x33749 runnable  


"GC Thread#2" os_prio=0 cpu=14,93ms elapsed=115,55s tid=0x00007f296c032800 nid=0x3374a runnable  


"GC Thread#3" os_prio=0 cpu=9,34ms elapsed=115,55s tid=0x00007f296c034000 nid=0x3374b runnable  


"GC Thread#4" os_prio=0 cpu=8,27ms elapsed=115,55s tid=0x00007f296c036000 nid=0x3374c runnable  


"GC Thread#5" os_prio=0 cpu=8,30ms elapsed=115,55s tid=0x00007f296c037800 nid=0x3374d runnable  


"GC Thread#6" os_prio=0 cpu=16,33ms elapsed=115,55s tid=0x00007f296c039000 nid=0x3374e runnable  


"GC Thread#7" os_prio=0 cpu=8,52ms elapsed=115,55s tid=0x00007f296c03a800 nid=0x3374f runnable  


"GC Thread#8" os_prio=0 cpu=8,53ms elapsed=115,55s tid=0x00007f296c03c000 nid=0x33750 runnable  


"GC Thread#9" os_prio=0 cpu=12,48ms elapsed=115,55s tid=0x00007f296c03d800 nid=0x33751 runnable  


"GC Thread#10" os_prio=0 cpu=8,30ms elapsed=115,55s tid=0x00007f296c03f000 nid=0x33752 runnable  


"GC Thread#11" os_prio=0 cpu=8,27ms elapsed=115,55s tid=0x00007f296c040800 nid=0x33753 runnable  


"GC Thread#12" os_prio=0 cpu=8,15ms elapsed=115,55s tid=0x00007f296c042000 nid=0x33754 runnable  


"G1 Main Marker" os_prio=0 cpu=1,72ms elapsed=117,41s tid=0x00007f29b80a4000 nid=0x33728 runnable  


"G1 Conc#0" os_prio=0 cpu=12,98ms elapsed=117,41s tid=0x00007f29b80a6000 nid=0x33729 runnable  


"G1 Conc#1" os_prio=0 cpu=11,83ms elapsed=115,53s tid=0x00007f2984001000 nid=0x33755 runnable  


"G1 Conc#2" os_prio=0 cpu=12,51ms elapsed=115,53s tid=0x00007f2984002800 nid=0x33756 runnable  


"G1 Refine#0" os_prio=0 cpu=0,70ms elapsed=117,38s tid=0x00007f29b915e000 nid=0x3372a runnable  


"G1 Young RemSet Sampling" os_prio=0 cpu=17,14ms elapsed=117,38s tid=0x00007f29b9160000 nid=0x3372b runnable  


"StrDedup" os_prio=0 cpu=0,51ms elapsed=117,38s tid=0x00007f29b9168000 nid=0x3372c runnable  


"VM Periodic Task Thread" os_prio=0 cpu=65,46ms elapsed=117,13s tid=0x00007f29b9c04800 nid=0x3373b waiting on condition  
JNI global refs: 24, weak refs: 0

@fanf
Copy link
Author

fanf commented Apr 14, 2023

Stringer things: the above example with Console.printLine ALWAYS blocks, even with foreach. If I switch back to ZIO.attempt(println(...)), it works as before (blocks on foreachPar with input >1, works with foreach and input of size 1)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment