dl4jrun.log

Data (.tar.gz file) already exists at /tmp/dl4j_w2vSentiment/aclImdb_v1.tar.gz
Data (extracted) already exists at /tmp/dl4j_w2vSentiment/aclImdb
15:38:49.015 [main] INFO  org.nd4j.linalg.factory.Nd4jBackend - Loaded [CpuBackend] backend
15:38:53.521 [main] INFO  org.nd4j.nativeblas.NativeOpsHolder - Number of threads used for linear algebra: 6
15:38:53.524 [main] WARN  o.n.l.c.nativecpu.CpuNDArrayFactory - *********************************** CPU Feature Check Warning ***********************************
15:38:53.524 [main] WARN  o.n.l.c.nativecpu.CpuNDArrayFactory - Warning: Initializing ND4J with Generic x86 binary on a CPU with AVX/AVX2 support
15:38:53.524 [main] WARN  o.n.l.c.nativecpu.CpuNDArrayFactory - Using ND4J with AVX/AVX2 will improve performance. See deeplearning4j.org/cpu for more details
15:38:53.524 [main] WARN  o.n.l.c.nativecpu.CpuNDArrayFactory - Or set environment variable ND4J_IGNORE_AVX=true to suppress this warning
15:38:53.524 [main] WARN  o.n.l.c.nativecpu.CpuNDArrayFactory - *************************************************************************************************
15:38:53.542 [main] INFO  org.nd4j.nativeblas.Nd4jBlas - Number of threads used for OpenMP BLAS: 6
15:38:53.550 [main] INFO  o.n.l.a.o.e.DefaultOpExecutioner - Backend used: [CPU]; OS: [Linux]
15:38:53.550 [main] INFO  o.n.l.a.o.e.DefaultOpExecutioner - Cores: [12]; Memory: [5.3GB];
15:38:53.550 [main] INFO  o.n.l.a.o.e.DefaultOpExecutioner - Blas vendor: [OPENBLAS]
15:38:53.656 [main] INFO  o.d.nn.graph.ComputationGraph - Starting ComputationGraph with WorkspaceModes set to [training: ENABLED; inference: ENABLED], cacheMode set to [NONE]
Number of parameters by layer:
	cnn3	90100
	cnn4	120100
	cnn5	150100
	globalPool	0
	out	602
Loading word vectors and creating DataSetIterators
15:38:53.848 [main] DEBUG o.d.m.e.loader.WordVectorSerializer - Trying DL4j format...
15:38:53.859 [main] DEBUG o.d.m.e.loader.WordVectorSerializer - Trying CSVReader...
15:38:53.882 [main] DEBUG o.d.m.e.loader.WordVectorSerializer - Trying BinaryReader...
Starting training
=================================================================
==8472==ERROR: AddressSanitizer: heap-use-after-free on address 0x614000001e80 at pc 0x7f5d4134b27b bp 0x7f5f68814b10 sp 0x7f5f68814b00
READ of size 8 at 0x614000001e80 thread T1
    #0 0x7f5d4134b27a in std::_Hashtable<long long, std::pair<long long const, nd4j::ops::DeclarableOp*>, std::allocator<std::pair<long long const, nd4j::ops::DeclarableOp*> >, std::__detail::_Select1st, std::equal_to<long long>, std::hash<long long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_bucket_index(long long const&, unsigned long) const /usr/lib/gcc/x86_64-pc-linux-gnu/8.3.0/include/g++-v8/bits/hashtable.h:643
    #1 0x7f5d4134831d in std::_Hashtable<long long, std::pair<long long const, nd4j::ops::DeclarableOp*>, std::allocator<std::pair<long long const, nd4j::ops::DeclarableOp*> >, std::__detail::_Select1st, std::equal_to<long long>, std::hash<long long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::count(long long const&) const /usr/lib/gcc/x86_64-pc-linux-gnu/8.3.0/include/g++-v8/bits/hashtable.h:1472
    #2 0x7f5d413460ac in std::unordered_map<long long, nd4j::ops::DeclarableOp*, std::hash<long long>, std::equal_to<long long>, std::allocator<std::pair<long long const, nd4j::ops::DeclarableOp*> > >::count(long long const&) const /usr/lib/gcc/x86_64-pc-linux-gnu/8.3.0/include/g++-v8/bits/unordered_map.h:939
    #3 0x7f5d41342bda in nd4j::ops::OpRegistrator::getOperation(long long) /home/zamlz/src/java/deeplearning4j-konduit/libnd4j/include/ops/declarable/impl/OpRegistrator.cpp:203
    #4 0x7f5d3359fd8a in execCustomOp2 /home/zamlz/src/java/deeplearning4j-konduit/libnd4j/blas/cpu/NativeOps.cpp:2066
    #5 0x7f5d498e53de in Java_org_nd4j_nativeblas_Nd4jCpu_execCustomOp2 (/home/zamlz/.javacpp/cache/nd4j-native-1.0.0-SNAPSHOT-linux-x86_64.jar/org/nd4j/nativeblas/linux-x86_64/libjnind4jcpu.so+0x1fd3de)
    #6 0x7f5f590d47b5  (<unknown module>)

0x614000001e80 is located 64 bytes inside of 408-byte region [0x614000001e40,0x614000001fd8)
freed by thread T55 here:
    #0 0x7f5f6ceb2210 in operator delete(void*) /var/tmp/portage/sys-devel/gcc-8.3.0-r1/work/gcc-8.3.0/libsanitizer/asan/asan_new_delete.cc:135
    #1 0x7f5d4133fd76 in nd4j::ops::OpRegistrator::sigSegVHandler(int) /home/zamlz/src/java/deeplearning4j-konduit/libnd4j/include/ops/declarable/impl/OpRegistrator.cpp:105
    #2 0x7f5f6c9c7fff  (/lib64/libc.so.6+0x38fff)
    #3 0x7f5f68edd3e1 in jni_AllocObject (/opt/oracle-jdk-bin-1.8.0.202/jre/lib/amd64/server/libjvm.so+0x6c63e1)
    #4 0x7f5d498a3c62 in Java_org_nd4j_nativeblas_Nd4jCpu_shapeBuffer__ILorg_bytedeco_javacpp_LongPointer_2Lorg_bytedeco_javacpp_LongPointer_2ICJZ (/home/zamlz/.javacpp/cache/nd4j-native-1.0.0-SNAPSHOT-linux-x86_64.jar/org/nd4j/nativeblas/linux-x86_64/libjnind4jcpu.so+0x1bbc62)
    #5 0x7f5f590d47b5  (<unknown module>)
    #6 0x7f5f590c51e6  (<unknown module>)
    #7 0x7f5f590c51e6  (<unknown module>)
    #8 0x7f5f590c50ff  (<unknown module>)
    #9 0x7f5f590c51e6  (<unknown module>)
    #10 0x7f5f590c53b5  (<unknown module>)
    #11 0x7f5f590c53b5  (<unknown module>)
    #12 0x7f5f590c53b5  (<unknown module>)
    #13 0x7f5f590c51e6  (<unknown module>)
    #14 0x7f5f590c50ff  (<unknown module>)
    #15 0x7f5f590c50ff  (<unknown module>)
    #16 0x7f5f590c51e6  (<unknown module>)
    #17 0x7f5f590c51e6  (<unknown module>)
    #18 0x7f5f590c51e6  (<unknown module>)
    #19 0x7f5f590c50ff  (<unknown module>)
    #20 0x7f5f590c50ff  (<unknown module>)
    #21 0x7f5f590c50ff  (<unknown module>)
    #22 0x7f5f590c51e6  (<unknown module>)
    #23 0x7f5f590c50ff  (<unknown module>)
    #24 0x7f5f590c51e6  (<unknown module>)
    #25 0x7f5f590bd7a6  (<unknown module>)
    #26 0x7f5f68e9f39a in JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*) (/opt/oracle-jdk-bin-1.8.0.202/jre/lib/amd64/server/libjvm.so+0x68839a)
    #27 0x7f5f68e9cc62 in JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*) (/opt/oracle-jdk-bin-1.8.0.202/jre/lib/amd64/server/libjvm.so+0x685c62)
    #28 0x7f5f68e9d226 in JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*) (/opt/oracle-jdk-bin-1.8.0.202/jre/lib/amd64/server/libjvm.so+0x686226)
    #29 0x7f5f68f0939b in thread_entry(JavaThread*, Thread*) (/opt/oracle-jdk-bin-1.8.0.202/jre/lib/amd64/server/libjvm.so+0x6f239b)

previously allocated by thread T1 here:
    #0 0x7f5f6ceb13a0 in operator new(unsigned long) /var/tmp/portage/sys-devel/gcc-8.3.0-r1/work/gcc-8.3.0/libsanitizer/asan/asan_new_delete.cc:90
    #1 0x7f5d4133f8f3 in nd4j::ops::OpRegistrator::getInstance() /home/zamlz/src/java/deeplearning4j-konduit/libnd4j/include/ops/declarable/impl/OpRegistrator.cpp:56
    #2 0x7f5d3ee68e44 in nd4j::ops::__registrator_crelu<nd4j::ops::crelu>::__registrator_crelu() /home/zamlz/src/java/deeplearning4j-konduit/libnd4j/include/ops/declarable/generic/activations/crelu.cpp:29
    #3 0x7f5d3ee68db6 in __static_initialization_and_destruction_0 /home/zamlz/src/java/deeplearning4j-konduit/libnd4j/include/ops/declarable/generic/activations/crelu.cpp:29
    #4 0x7f5d3ee68ddd in _GLOBAL__sub_I_crelu.cpp /home/zamlz/src/java/deeplearning4j-konduit/libnd4j/include/ops/declarable/generic/activations/crelu.cpp:114
    #5 0x7f5f6dba02e9 in call_init /usr/src/debug/sys-libs/glibc-2.29-r7/glibc-2.29/elf/dl-init.c:72

Thread T1 created by T0 here:
    #0 0x7f5f6ce13a73 in __interceptor_pthread_create /var/tmp/portage/sys-devel/gcc-8.3.0-r1/work/gcc-8.3.0/libsanitizer/asan/asan_interceptors.cc:202
    #1 0x7f5f6cb7180e in ContinueInNewThread0 (/opt/oracle-jdk-bin-1.8.0.202/bin/../lib/amd64/jli/libjli.so+0xd80e)
    #2 0x7f5f6cb6dc41 in ContinueInNewThread (/opt/oracle-jdk-bin-1.8.0.202/bin/../lib/amd64/jli/libjli.so+0x9c41)
    #3 0x7f5f6cb6e367 in JLI_Launch (/opt/oracle-jdk-bin-1.8.0.202/bin/../lib/amd64/jli/libjli.so+0xa367)
    #4 0x400539 in main (/opt/oracle-jdk-bin-1.8.0.202/bin/java+0x400539)
    #5 0x7f5f6c9b2eda in __libc_start_main ../csu/libc-start.c:308

Thread T55 created by T1 here:
    #0 0x7f5f6ce13a73 in __interceptor_pthread_create /var/tmp/portage/sys-devel/gcc-8.3.0-r1/work/gcc-8.3.0/libsanitizer/asan/asan_interceptors.cc:202
    #1 0x7f5f6912ed2b in os::create_thread(Thread*, os::ThreadType, unsigned long) (/opt/oracle-jdk-bin-1.8.0.202/jre/lib/amd64/server/libjvm.so+0x917d2b)
    #2 0x7f5f68f0fab4 in JVM_StartThread (/opt/oracle-jdk-bin-1.8.0.202/jre/lib/amd64/server/libjvm.so+0x6f8ab4)
    #3 0x7f5f590d47b5  (<unknown module>)
    #4 0x7f5f590c53b5  (<unknown module>)
    #5 0x7f5f590c53b5  (<unknown module>)
    #6 0x7f5f590c53b5  (<unknown module>)
    #7 0x7f5f590c53b5  (<unknown module>)
    #8 0x7f5f590c53b5  (<unknown module>)
    #9 0x7f5f590c53b5  (<unknown module>)
    #10 0x7f5f590c53b5  (<unknown module>)
    #11 0x7f5f590c53b5  (<unknown module>)
    #12 0x7f5f590c53b5  (<unknown module>)
    #13 0x7f5f590c53b5  (<unknown module>)
    #14 0x7f5f590bd7a6  (<unknown module>)
    #15 0x7f5f68e9f39a in JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*) (/opt/oracle-jdk-bin-1.8.0.202/jre/lib/amd64/server/libjvm.so+0x68839a)
    #16 0x7f5f68eea350 in jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*) [clone .isra.96] [clone .constprop.117] (/opt/oracle-jdk-bin-1.8.0.202/jre/lib/amd64/server/libjvm.so+0x6d3350)
    #17 0x7f5f68eec61a in jni_CallStaticVoidMethod (/opt/oracle-jdk-bin-1.8.0.202/jre/lib/amd64/server/libjvm.so+0x6d561a)
    #18 0x7f5f6cb6c89b in JavaMain (/opt/oracle-jdk-bin-1.8.0.202/bin/../lib/amd64/jli/libjli.so+0x889b)
    #19 0x7f5f6cd84457 in start_thread /var/tmp/portage/sys-libs/glibc-2.29-r7/work/glibc-2.29/nptl/pthread_create.c:486

SUMMARY: AddressSanitizer: heap-use-after-free /usr/lib/gcc/x86_64-pc-linux-gnu/8.3.0/include/g++-v8/bits/hashtable.h:643 in std::_Hashtable<long long, std::pair<long long const, nd4j::ops::DeclarableOp*>, std::allocator<std::pair<long long const, nd4j::ops::DeclarableOp*> >, std::__detail::_Select1st, std::equal_to<long long>, std::hash<long long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_bucket_index(long long const&, unsigned long) const
Shadow bytes around the buggy address:
  0x0c287fff8380: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c287fff8390: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c287fff83a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c287fff83b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c287fff83c0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
=>0x0c287fff83d0:[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c287fff83e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c287fff83f0: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa
  0x0c287fff8400: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c287fff8410: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c287fff8420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==8472==ABORTING