demo of a Java-level stack overflow causing a native crash

command_prompt

$ java -version
java version "1.6.0_30"
Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode)
$ javac -g StackOverflowCrashDemo.java 
$ javah StackOverflowCrashDemo
$ ll
total 12
-rw-r--r-- 1 sajia users 893 May  2 13:29 StackOverflowCrashDemo.class
-rw-r--r-- 1 sajia users 464 May  2 13:29 StackOverflowCrashDemo.h
-rw-r--r-- 1 sajia users 415 May  2 13:29 StackOverflowCrashDemo.java
$ gcc -o libdemo.so -I $JAVA_HOME/include -I$JAVA_HOME/include/linux -shared -fPIC demo.c 
$ java -Xss128k StackOverflowCrashDemo
Segmentation fault (core dumped)
$ ll
total 8556
-rw------- 1 sajia users 238821376 May  2 14:08 core.2088
-rw-r--r-- 1 sajia users       226 May  2 14:08 demo.c
-rwxr-xr-x 1 sajia users      5700 May  2 14:08 libdemo.so
-rw-r--r-- 1 sajia users       737 May  2 14:07 StackOverflowCrashDemo.class
-rw-r--r-- 1 sajia users       464 May  2 13:29 StackOverflowCrashDemo.h
-rw-r--r-- 1 sajia users       320 May  2 14:04 StackOverflowCrashDemo.java
$ jstack $JAVA_HOME/bin/java core.2088 > jstack2088.log
Attaching to core core.2088 from executable /home/sajia/sdk/jdk1.6.0_30/bin/java, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 20.5-b03
$ sudo cat /var/log/messages | grep -i java
May  2 14:08:47 testmachine kernel: : java[2089]: segfault at 0000000041a2efa8 rip 00002aaab81e2507 rsp 0000000041a2f018 error 6

demo.c

#include <jni.h>
#include "StackOverflowCrashDemo.h"

#define K 1024
#define PAGE_SIZE (4*K)

void Java_StackOverflowCrashDemo_useLotsOfStackSpace(JNIEnv* env, jclass unused) {
  char dummy[10*PAGE_SIZE];
  dummy[0] = 0xFF;
}

gdb_session

$ gdb $JAVA_HOME/bin/java core.2088 
GNU gdb Fedora (6.8-37.el5)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...
(no debugging symbols found)
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /home/sajia/sdk/jdk1.6.0_30/jre/lib/amd64/jli/libjli.so...(no debugging symbols found)...done.
Loaded symbols for /home/sajia/sdk/jdk1.6.0_30/bin/../jre/lib/amd64/jli/libjli.so
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...
(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /home/sajia/sdk/jdk1.6.0_30/jre/lib/amd64/server/libjvm.so...(no debugging symbols found)...done.
Loaded symbols for /home/sajia/sdk/jdk1.6.0_30/jre/lib/amd64/server/libjvm.so
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /home/sajia/sdk/jdk1.6.0_30/jre/lib/amd64/libverify.so...
(no debugging symbols found)...done.
Loaded symbols for /home/sajia/sdk/jdk1.6.0_30/jre/lib/amd64/libverify.so
Reading symbols from /home/sajia/sdk/jdk1.6.0_30/jre/lib/amd64/libjava.so...(no debugging symbols found)...done.
Loaded symbols for /home/sajia/sdk/jdk1.6.0_30/jre/lib/amd64/libjava.so
Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /home/sajia/sdk/jdk1.6.0_30/jre/lib/amd64/libzip.so...
(no debugging symbols found)...done.
Loaded symbols for /home/sajia/sdk/jdk1.6.0_30/jre/lib/amd64/libzip.so
Reading symbols from /home/sajia/experiment/crash_stackoverflow/libdemo.so...(no debugging symbols found)...done.
Loaded symbols for /home/sajia/experiment/crash_stackoverflow/libdemo.so
Core was generated by `java -Xss128k StackOverflowCrashDemo'.
Program terminated with signal 11, Segmentation fault.
[New process 2089]
[New process 2110]
[New process 2109]
[New process 2108]
[New process 2107]
[New process 2106]
[New process 2105]
[New process 2104]
[New process 2103]
[New process 2102]
[New process 2101]
[New process 2100]
[New process 2099]
[New process 2098]
[New process 2097]
[New process 2096]
[New process 2095]
[New process 2094]
[New process 2093]
[New process 2092]
[New process 2091]
[New process 2090]
[New process 2088]
#0  0x00002aaab81e2507 in Java_StackOverflowCrashDemo_useLotsOfStackSpace ()
   from /home/sajia/experiment/crash_stackoverflow/libdemo.so
(gdb) bt
#0  0x00002aaab81e2507 in Java_StackOverflowCrashDemo_useLotsOfStackSpace ()
   from /home/sajia/experiment/crash_stackoverflow/libdemo.so
#1  0x00002aaaab040b0c in ?? ()
#2  0x0000000000000000 in ?? ()
(gdb) info threads
  23 process 2088  0x000000347f4077e5 in pthread_join () from /lib64/libpthread.so.0
  22 process 2090  0x000000347f40ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  21 process 2091  0x000000347f40ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  20 process 2092  0x000000347f40ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  19 process 2093  0x000000347f40ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  18 process 2094  0x000000347f40ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  17 process 2095  0x000000347f40ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  16 process 2096  0x000000347f40ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  15 process 2097  0x000000347f40ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  14 process 2098  0x000000347f40ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  13 process 2099  0x000000347f40ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  12 process 2100  0x000000347f40ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  11 process 2101  0x000000347f40ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  10 process 2102  0x000000347f40ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  9 process 2103  0x000000347f40ae00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  8 process 2104  0x000000347f40ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  7 process 2105  0x000000347f40ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  6 process 2106  0x000000347f40c9b1 in sem_wait () from /lib64/libpthread.so.0
  5 process 2107  0x000000347f40ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  4 process 2108  0x000000347f40ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  3 process 2109  0x000000347f40ab99 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  2 process 2110  0x000000347f40ae00 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
* 1 process 2089  0x00002aaab81e2507 in Java_StackOverflowCrashDemo_useLotsOfStackSpace ()
   from /home/sajia/experiment/crash_stackoverflow/libdemo.so
(gdb) quit
$ 

jstack2088.log

Deadlock Detection:

No deadlocks found.

Thread 2106: (state = BLOCKED)


Thread 2105: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=134 (Interpreted frame)
 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 (Interpreted frame)


Thread 2104: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
 - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116 (Interpreted frame)


Thread 2089: (state = IN_NATIVE)
 - StackOverflowCrashDemo.useLotsOfStackSpace() @bci=0 (Interpreted frame)
 - StackOverflowCrashDemo.doTest(int) @bci=0, line=7 (Interpreted frame)
 - StackOverflowCrashDemo.doTest(int) @bci=10, line=9 (Interpreted frame)
 - StackOverflowCrashDemo.doTest(int) @bci=10, line=9 (Interpreted frame)
 - StackOverflowCrashDemo.doTest(int) @bci=10, line=9 (Interpreted frame)
  ... omitted 865 same frames ...
 - StackOverflowCrashDemo.doTest(int) @bci=10, line=9 (Interpreted frame)
 - StackOverflowCrashDemo.doTest(int) @bci=10, line=9 (Interpreted frame)
 - StackOverflowCrashDemo.doTest(int) @bci=10, line=9 (Interpreted frame)
 - StackOverflowCrashDemo.main(java.lang.String[]) @bci=1, line=14 (Interpreted frame)

StackOverflowCrashDemo.h

/* DO NOT EDIT THIS FILE - it is machine generated */
#include <jni.h>
/* Header for class StackOverflowCrashDemo */

#ifndef _Included_StackOverflowCrashDemo
#define _Included_StackOverflowCrashDemo
#ifdef __cplusplus
extern "C" {
#endif
/*
 * Class:     StackOverflowCrashDemo
 * Method:    useLotsOfStackSpace
 * Signature: ()V
 */
JNIEXPORT void JNICALL Java_StackOverflowCrashDemo_useLotsOfStackSpace
  (JNIEnv *, jclass);

#ifdef __cplusplus
}
#endif
#endif

StackOverflowCrashDemo.java

public class StackOverflowCrashDemo {
  static {
    System.loadLibrary("demo");
  }

  private static void doTest(int n) {
    useLotsOfStackSpace();
    if (n != 0) {
      doTest(n - 1);
    }
  }

  public static void main(String[] args) {
    doTest(-1);
  }

  private static native void useLotsOfStackSpace();
}