Skip to content

Instantly share code, notes, and snippets.

@agunnerson-ibm

agunnerson-ibm/job.log Secret

Last active September 16, 2016 14:22
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save agunnerson-ibm/dccb6c8c2edc6a498f9c377d96f43481 to your computer and use it in GitHub Desktop.
Save agunnerson-ibm/dccb6c8c2edc6a498f9c377d96f43481 to your computer and use it in GitHub Desktop.
Download ZIP
Jenkins Kubernetes OOM Unable to create native thread bug
Raw
job.log
09:03:00 Started by upstream project "[REDACTED]" build number 236
09:03:00 originally caused by:
09:03:00 Started by timer
09:03:00 Building remotely on dynamic-qa-cucumber-138fb00db9972 (dynamic-qa-cucumber) in workspace /home/jenkins/workspace/[REDACTED]
09:03:00 java.io.IOException: remote file operation failed: /home/jenkins/workspace/[REDACTED] at hudson.remoting.Channel@2fca3b75:dynamic-qa-cucumber-138fb00db9972: java.io.IOException: Remote call on dynamic-qa-cucumber-138fb00db9972 failed
09:03:00 at hudson.FilePath.act(FilePath.java:1014)
09:03:00 at hudson.FilePath.act(FilePath.java:996)
09:03:00 at hudson.FilePath.mkdirs(FilePath.java:1179)
09:03:00 at hudson.model.AbstractProject.checkout(AbstractProject.java:1267)
09:03:00 at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:604)
09:03:00 at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
09:03:00 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
09:03:00 at hudson.model.Run.execute(Run.java:1741)
09:03:00 at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
09:03:00 at hudson.model.ResourceController.execute(ResourceController.java:98)
09:03:00 at hudson.model.Executor.run(Executor.java:410)
09:03:00 Caused by: java.io.IOException: Remote call on dynamic-qa-cucumber-138fb00db9972 failed
09:03:00 at hudson.remoting.Channel.call(Channel.java:789)
09:03:00 at hudson.FilePath.act(FilePath.java:1007)
09:03:00 ... 10 more
09:03:00 Caused by: java.lang.OutOfMemoryError: unable to create new native thread
09:03:00 at java.lang.Thread.start0(Native Method)
09:03:00 at java.lang.Thread.start(Thread.java:714)
09:03:00 at hudson.remoting.AtmostOneThreadExecutor.execute(AtmostOneThreadExecutor.java:94)
09:03:00 at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
09:03:00 at hudson.remoting.JarCacheSupport.resolve(JarCacheSupport.java:61)
09:03:00 at hudson.remoting.ResourceImageInJar._resolveJarURL(ResourceImageInJar.java:85)
09:03:00 at hudson.remoting.ResourceImageInJar.resolve(ResourceImageInJar.java:42)
09:03:00 at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:248)
09:03:00 at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
09:03:00 at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
09:03:00 at java.lang.Class.getDeclaredFields0(Native Method)
09:03:00 at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
09:03:00 at java.lang.Class.getDeclaredField(Class.java:2068)
09:03:00 at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1659)
09:03:00 at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:72)
09:03:00 at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:480)
09:03:00 at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:468)
09:03:00 at java.security.AccessController.doPrivileged(Native Method)
09:03:00 at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:468)
09:03:00 at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:365)
09:03:00 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:602)
09:03:00 at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1623)
09:03:00 at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
09:03:00 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
09:03:00 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
09:03:00 at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
09:03:00 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
09:03:00 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
09:03:00 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
09:03:00 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
09:03:00 at hudson.remoting.UserRequest.deserialize(UserRequest.java:217)
09:03:00 at hudson.remoting.UserRequest.perform(UserRequest.java:131)
09:03:00 at hudson.remoting.UserRequest.perform(UserRequest.java:50)
09:03:00 at hudson.remoting.Request$2.run(Request.java:332)
09:03:00 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
09:03:00 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
09:03:00 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
09:03:00 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
09:03:00 at hudson.remoting.Engine$1$1.run(Engine.java:85)
09:03:00 at java.lang.Thread.run(Thread.java:745)
09:03:00 at ......remote call to dynamic-qa-cucumber-138fb00db9972(Native Method)
09:03:00 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
09:03:00 at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
09:03:00 at hudson.remoting.Channel.call(Channel.java:781)
09:03:00 ... 11 more
09:03:00 [xUnit] [INFO] - Starting to record.
09:03:00 [xUnit] [INFO] - Processing JUnit
Raw
process_limits_master.txt
[andrew.gunnerson@[REDACTED] ~]$ sudo docker exec -it jenkins bash
[sudo] password for andrew.gunnerson:
jenkins@2294471054b5:/$ jps
5 jenkins.war
5801 Jps
jenkins@2294471054b5:/$ cat /proc/5/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size unlimited unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 1048576 1048576 processes
Max open files 1048576 1048576 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 63466 63466 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us
jenkins@2294471054b5:/$ ps -p 5 -elT | wc -l
61
jenkins@2294471054b5:/$
Raw
slave.jar.log
Warning: JnlpProtocol3 is disabled by default, use JNLP_PROTOCOL_OPTS to alter the behavior
Sep 16, 2016 1:02:54 PM hudson.remoting.jnlp.Main createEngine
INFO: Setting up slave: dynamic-qa-cucumber-138fb00db9972
Sep 16, 2016 1:02:54 PM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Sep 16, 2016 1:02:54 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://[REDACTED]:8080/]
Sep 16, 2016 1:02:54 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Sep 16, 2016 1:02:54 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to [REDACTED]:50000
Sep 16, 2016 1:02:54 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP3-connect is not enabled, skipping
Sep 16, 2016 1:02:54 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP2-connect
Sep 16, 2016 1:02:54 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Sep 16, 2016 1:02:55 PM hudson.remoting.SynchronousCommandTransport$ReaderThread run
SEVERE: I/O error in channel channel
java.io.IOException: Unexpected termination of the channel
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2328)
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2797)
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
Sep 16, 2016 1:02:55 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
Sep 16, 2016 1:03:05 PM jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$2$1 onReconnect
INFO: Restarting agent via jenkins.slaves.restarter.UnixSlaveRestarter@7822ab88
Sep 16, 2016 1:03:06 PM hudson.remoting.jnlp.Main createEngine
INFO: Setting up slave: dynamic-qa-cucumber-138fb00db9972
Sep 16, 2016 1:03:06 PM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Sep 16, 2016 1:03:07 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [[REDACTED]:8080/]
Sep 16, 2016 1:03:07 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Sep 16, 2016 1:03:07 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to [REDACTED]:50000
Sep 16, 2016 1:03:07 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP3-connect is not enabled, skipping
Sep 16, 2016 1:03:07 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP2-connect
Sep 16, 2016 1:03:07 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Server didn't accept the handshake: dynamic-qa-cucumber-138fb00db9972 is already connected to this master. Rejecting this connection.
Sep 16, 2016 1:03:07 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to [REDACTED]:50000
Sep 16, 2016 1:03:07 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP-connect
Sep 16, 2016 1:03:07 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Server didn't accept the handshake: dynamic-qa-cucumber-138fb00db9972 is already connected to this master. Rejecting this connection.
Sep 16, 2016 1:03:07 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to [REDACTED]:50000
Sep 16, 2016 1:03:07 PM hudson.remoting.jnlp.Main$CuiListener error
SEVERE: The server rejected the connection: None of the protocols were accepted
java.lang.Exception: The server rejected the connection: None of the protocols were accepted
at hudson.remoting.Engine.onConnectionRejected(Engine.java:335)
at hudson.remoting.Engine.run(Engine.java:307)
Raw
slave.log
JNLP agent connected from /[REDACTED]
<===[JENKINS REMOTING CAPACITY]===>Slave.jar version: 2.60
This is a Unix agent
Failed to establish the connection with the agent dynamic-qa-cucumber-138fb00db9972
java.io.IOException: Remote call on dynamic-qa-cucumber-138fb00db9972 failed
at hudson.remoting.Channel.call(Channel.java:789)
at org.jenkinsci.modules.slave_installer.impl.ComputerListenerImpl.onOnline(ComputerListenerImpl.java:32)
at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:573)
at jenkins.slaves.JnlpSlaveAgentProtocol$Handler.jnlpConnect(JnlpSlaveAgentProtocol.java:127)
at jenkins.slaves.DefaultJnlpSlaveReceiver.handle(DefaultJnlpSlaveReceiver.java:69)
at jenkins.slaves.JnlpSlaveAgentProtocol2$Handler2.run(JnlpSlaveAgentProtocol2.java:60)
at jenkins.slaves.JnlpSlaveAgentProtocol2.handle(JnlpSlaveAgentProtocol2.java:32)
at hudson.TcpSlaveAgentListener$ConnectionHandler.run(TcpSlaveAgentListener.java:182)
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at hudson.remoting.AtmostOneThreadExecutor.execute(AtmostOneThreadExecutor.java:94)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
at hudson.remoting.JarCacheSupport.resolve(JarCacheSupport.java:61)
at hudson.remoting.ResourceImageInJar._resolveJarURL(ResourceImageInJar.java:85)
at hudson.remoting.ResourceImageInJar.resolve(ResourceImageInJar.java:42)
at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:248)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.getDeclaredFields0(Native Method)
at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
at java.lang.Class.getDeclaredField(Class.java:2068)
at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1659)
at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:72)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:480)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:468)
at java.security.AccessController.doPrivileged(Native Method)
at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:468)
at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:365)
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:602)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1623)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1993)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at hudson.remoting.UserRequest.deserialize(UserRequest.java:217)
at hudson.remoting.UserRequest.perform(UserRequest.java:131)
at hudson.remoting.UserRequest.perform(UserRequest.java:50)
at hudson.remoting.Request$2.run(Request.java:332)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at hudson.remoting.Engine$1$1.run(Engine.java:85)
at java.lang.Thread.run(Thread.java:745)
at ......remote call to dynamic-qa-cucumber-138fb00db9972(Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:253)
at hudson.remoting.Channel.call(Channel.java:781)
... 7 more
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment