Skip to content

Instantly share code, notes, and snippets.

@neomatrix369
Created October 21, 2019 20:47
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save neomatrix369/8125f18f2a1d6da236ab5bd26831e69d to your computer and use it in GitHub Desktop.
Save neomatrix369/8125f18f2a1d6da236ab5bd26831e69d to your computer and use it in GitHub Desktop.
Logs of the GPU version of dl4j-nlp failing due to OOME (dl4j-nlp-cuda)
openjdk version "1.8.0_192"
OpenJDK Runtime Environment (build 1.8.0_192-20181024121959.buildslave.jdk8u-src-tar--b12)
GraalVM 1.0.0-rc9 (build 25.192-b12-jvmci-0.49, mixed mode)
Data (.tar.gz file) already exists at /tmp/dl4j_w2vSentiment/aclImdb_v1.tar.gz
Data (extracted) already exists at /tmp/dl4j_w2vSentiment/aclImdb
2019-10-21 21:42:44,309ate - 21:42:44,309 INFO ~ Loaded [JCublasBackend] backend
2019-10-21 21:42:46,256ate - 21:42:46,256 INFO ~ Number of threads used for NativeOps: 32
2019-10-21 21:42:47,140ate - 21:42:47,140 INFO ~ Number of threads used for BLAS: 0
2019-10-21 21:42:47,145ate - 21:42:47,145 INFO ~ Backend used: [CUDA]; OS: [Linux]
2019-10-21 21:42:47,145ate - 21:42:47,145 INFO ~ Cores: [8]; Memory: [3.4GB];
2019-10-21 21:42:47,145ate - 21:42:47,145 INFO ~ Blas vendor: [CUBLAS]
2019-10-21 21:42:47,145ate - 21:42:47,145 INFO ~ Device Name: [GeForce GTX 1050]; CC: [6.1]; Total/free memory: [4238737408]
2019-10-21 21:42:47,207ate - 21:42:47,207 INFO ~ Starting ComputationGraph with WorkspaceModes set to [training: ENABLED; inference: ENABLED], cacheMode set to [NONE]
2019-10-21 21:42:47,265ate - 21:42:47,265 INFO ~ cuDNN not found: use cuDNN for better GPU performance by including the deeplearning4j-cuda module. For more information, please refer to: https://deeplearning4j.org/docs/latest/deeplearning4j-config-cudnn
java.lang.ClassNotFoundException: org.deeplearning4j.nn.layers.convolution.CudnnConvolutionHelper
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.deeplearning4j.nn.layers.convolution.ConvolutionLayer.initializeHelper(ConvolutionLayer.java:75)
at org.deeplearning4j.nn.layers.convolution.ConvolutionLayer.<init>(ConvolutionLayer.java:67)
at org.deeplearning4j.nn.conf.layers.ConvolutionLayer.instantiate(ConvolutionLayer.java:168)
at org.deeplearning4j.nn.conf.graph.LayerVertex.instantiate(LayerVertex.java:107)
at org.deeplearning4j.nn.graph.ComputationGraph.init(ComputationGraph.java:570)
at org.deeplearning4j.nn.graph.ComputationGraph.init(ComputationGraph.java:439)
at org.deeplearning4j.examples.convolution.sentenceclassification.CnnSentenceClassificationExample.main(CnnSentenceClassificationExample.java:132)
Number of parameters by layer:
cnn3 90100
cnn4 120100
cnn5 150100
globalPool 0
out 602
Loading word vectors and creating DataSetIterators
Exception in thread "main" java.lang.OutOfMemoryError: Cannot allocate new FloatPointer(300): totalBytes = 2194M, physicalBytes = 7013M
at org.bytedeco.javacpp.FloatPointer.<init>(FloatPointer.java:76)
at org.bytedeco.javacpp.FloatPointer.<init>(FloatPointer.java:41)
at org.nd4j.compression.impl.AbstractCompressor.compress(AbstractCompressor.java:159)
at org.nd4j.compression.impl.AbstractCompressor.compress(AbstractCompressor.java:134)
at org.nd4j.storage.CompressedRamStorage.store(CompressedRamStorage.java:84)
at org.deeplearning4j.models.embeddings.loader.WordVectorSerializer.loadStaticModel(WordVectorSerializer.java:2785)
at org.deeplearning4j.examples.convolution.sentenceclassification.CnnSentenceClassificationExample.main(CnnSentenceClassificationExample.java:141)
Caused by: java.lang.OutOfMemoryError: Physical memory usage is too high: physicalBytes (7013M) > maxPhysicalBytes (6891M)
at org.bytedeco.javacpp.Pointer.deallocator(Pointer.java:585)
at org.bytedeco.javacpp.Pointer.init(Pointer.java:125)
at org.bytedeco.javacpp.FloatPointer.allocateArray(Native Method)
at org.bytedeco.javacpp.FloatPointer.<init>(FloatPointer.java:68)
... 6 more
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment