Last active
March 30, 2017 08:23
-
-
Save rkfg/717b42620c4e146c29b88232fd7eaef6 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 11:21:40,746 INFO ~ Loaded [JCublasBackend] backend | |
| 11:21:41,201 INFO ~ Number of threads used for NativeOps: 32 | |
| 11:21:41,334 INFO ~ Backend used: [CUDA]; OS: [Linux] | |
| 11:21:41,334 INFO ~ Cores: [8]; Memory: [23.1GB]; | |
| 11:21:41,334 INFO ~ Blas vendor: [CUBLAS] | |
| 11:21:41,335 INFO ~ Device name: [GeForce GTX 1070]; CC: [6.1]; Total/free memory: [8507752448] | |
| Building the dictionary... | |
| Dictionary is ready, size is 40035 | |
| Total dictionary size is 40035. Processing the dataset... | |
| Done. Corpus size is 281312 | |
| Creating a new network... | |
| Number of parameters: 223265379 | |
| Epoch 1 | |
| 11:21:45,957 INFO ~ Allocating [LOOP_ITER] workspace of 0 bytes... | |
| 11:21:46,117 INFO ~ [LOOP_ITER] spilled DEVICE array of 1152 bytes, capacity of 288 elements | |
| 11:21:46,125 INFO ~ [LOOP_ITER] spilled DEVICE array of 1152 bytes, capacity of 288 elements | |
| 11:21:46,126 INFO ~ [LOOP_ITER] spilled DEVICE array of 1152 bytes, capacity of 288 elements | |
| 11:21:46,138 WARN ~ Workspace initialization OVER_TIME was selected, but number of cycles isn't positive value! | |
| 11:21:46,138 INFO ~ Allocating [LOOP_FF] workspace of 0 bytes... | |
| 11:21:46,138 INFO ~ Allocating [LOOP_FF] workspace of 0 bytes... | |
| 11:21:46,138 INFO ~ [LOOP_FF] spilled DEVICE array of 1152 bytes, capacity of 288 elements | |
| CUDA error at /home/rkfg/soft/svn-soft/libnd4j/blas/cuda/NativeOps.cu:4801 code=11(cudaErrorInvalidValue) "result" | |
| Failed on [1110846756352] -> [1110844720640], size: [1152], direction: [2], result: [11] | |
| 11:21:46,140 INFO ~ Allocating [LOOP_ITER] workspace of 3456 bytes... | |
| [WARNING] | |
| java.lang.reflect.InvocationTargetException | |
| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) | |
| at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) | |
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
| at java.lang.reflect.Method.invoke(Method.java:498) | |
| at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294) | |
| at java.lang.Thread.run(Thread.java:745) | |
| Caused by: java.lang.IllegalStateException: MemcpyAsync failed: AllocationShape(offset=0, length=288, stride=1, elementSize=4, dataType=FLOAT) | |
| at org.nd4j.jita.flow.impl.SynchronousFlowController.synchronizeToHost(SynchronousFlowController.java:63) | |
| at org.nd4j.jita.flow.impl.GridFlowController.synchronizeToHost(GridFlowController.java:36) | |
| at org.nd4j.jita.handler.impl.CudaZeroHandler.synchronizeThreadDevice(CudaZeroHandler.java:1227) | |
| at org.nd4j.jita.allocator.impl.AtomicAllocator.synchronizeHostData(AtomicAllocator.java:321) | |
| at org.nd4j.linalg.jcublas.buffer.BaseCudaDataBuffer.getFloat(BaseCudaDataBuffer.java:965) | |
| at org.nd4j.linalg.jcublas.buffer.CudaFloatDataBuffer.getDouble(CudaFloatDataBuffer.java:237) | |
| at org.nd4j.linalg.api.shape.Shape.getDouble(Shape.java:163) | |
| at org.nd4j.linalg.api.ndarray.BaseNDArray.getDouble(BaseNDArray.java:1697) | |
| at org.nd4j.linalg.api.ndarray.BaseNDArray.getInt(BaseNDArray.java:1666) | |
| at org.deeplearning4j.nn.layers.feedforward.embedding.EmbeddingLayer.preOutput(EmbeddingLayer.java:93) | |
| at org.deeplearning4j.nn.layers.feedforward.embedding.EmbeddingLayer.activate(EmbeddingLayer.java:113) | |
| at org.deeplearning4j.nn.graph.vertex.impl.LayerVertex.doForward(LayerVertex.java:103) | |
| at org.deeplearning4j.nn.graph.ComputationGraph.feedForward(ComputationGraph.java:1180) | |
| at org.deeplearning4j.nn.graph.ComputationGraph.computeGradientAndScore(ComputationGraph.java:1053) | |
| at org.deeplearning4j.optimize.solvers.BaseOptimizer.gradientAndScore(BaseOptimizer.java:160) | |
| at org.deeplearning4j.optimize.solvers.StochasticGradientDescent.optimize(StochasticGradientDescent.java:56) | |
| at org.deeplearning4j.optimize.Solver.optimize(Solver.java:59) | |
| at org.deeplearning4j.nn.graph.ComputationGraph.fit(ComputationGraph.java:853) | |
| at dlchat.EncoderDecoderLSTM.train(EncoderDecoderLSTM.java:248) | |
| at dlchat.EncoderDecoderLSTM.run(EncoderDecoderLSTM.java:201) | |
| at dlchat.EncoderDecoderLSTM.main(EncoderDecoderLSTM.java:149) | |
| ... 6 more | |
| [WARNING] | |
| java.lang.RuntimeException: java.lang.InterruptedException | |
| at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.ensureCacheHolder(CudaCachingZeroProvider.java:130) | |
| at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.free(CudaCachingZeroProvider.java:160) | |
| at org.nd4j.jita.memory.impl.CudaFullCachingProvider.free(CudaFullCachingProvider.java:146) | |
| at org.nd4j.jita.handler.impl.CudaZeroHandler.free(CudaZeroHandler.java:480) | |
| at org.nd4j.jita.handler.impl.CudaZeroHandler.purgeZeroObject(CudaZeroHandler.java:1130) | |
| at org.nd4j.jita.allocator.impl.AtomicAllocator.purgeZeroObject(AtomicAllocator.java:443) | |
| at org.nd4j.jita.allocator.impl.AtomicAllocator$UnifiedGarbageCollectorThread.run(AtomicAllocator.java:658) | |
| Caused by: java.lang.InterruptedException | |
| at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302) | |
| at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) | |
| at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.ensureCacheHolder(CudaCachingZeroProvider.java:125) | |
| ... 6 more | |
| [WARNING] | |
| java.lang.RuntimeException: java.lang.InterruptedException | |
| at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.ensureCacheHolder(CudaCachingZeroProvider.java:130) | |
| at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.free(CudaCachingZeroProvider.java:160) | |
| at org.nd4j.jita.memory.impl.CudaFullCachingProvider.free(CudaFullCachingProvider.java:146) | |
| at org.nd4j.jita.handler.impl.CudaZeroHandler.free(CudaZeroHandler.java:480) | |
| at org.nd4j.jita.handler.impl.CudaZeroHandler.purgeZeroObject(CudaZeroHandler.java:1130) | |
| at org.nd4j.jita.allocator.impl.AtomicAllocator.purgeZeroObject(AtomicAllocator.java:443) | |
| at org.nd4j.jita.allocator.impl.AtomicAllocator$UnifiedGarbageCollectorThread.run(AtomicAllocator.java:658) | |
| Caused by: java.lang.InterruptedException | |
| at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302) | |
| at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) | |
| at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.ensureCacheHolder(CudaCachingZeroProvider.java:125) | |
| ... 6 more | |
| [WARNING] | |
| java.lang.RuntimeException: java.lang.InterruptedException | |
| at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.ensureCacheHolder(CudaCachingZeroProvider.java:130) | |
| at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.free(CudaCachingZeroProvider.java:160) | |
| at org.nd4j.jita.memory.impl.CudaFullCachingProvider.free(CudaFullCachingProvider.java:146) | |
| at org.nd4j.jita.handler.impl.CudaZeroHandler.free(CudaZeroHandler.java:480) | |
| at org.nd4j.jita.handler.impl.CudaZeroHandler.purgeZeroObject(CudaZeroHandler.java:1130) | |
| at org.nd4j.jita.allocator.impl.AtomicAllocator.purgeZeroObject(AtomicAllocator.java:443) | |
| at org.nd4j.jita.allocator.impl.AtomicAllocator$UnifiedGarbageCollectorThread.run(AtomicAllocator.java:658) | |
| Caused by: java.lang.InterruptedException | |
| at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302) | |
| at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) | |
| at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.ensureCacheHolder(CudaCachingZeroProvider.java:125) | |
| ... 6 more | |
| [WARNING] | |
| java.lang.RuntimeException: java.lang.InterruptedException | |
| at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.ensureCacheHolder(CudaCachingZeroProvider.java:130) | |
| at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.free(CudaCachingZeroProvider.java:160) | |
| at org.nd4j.jita.memory.impl.CudaFullCachingProvider.free(CudaFullCachingProvider.java:146) | |
| at org.nd4j.jita.handler.impl.CudaZeroHandler.free(CudaZeroHandler.java:480) | |
| at org.nd4j.jita.handler.impl.CudaZeroHandler.purgeZeroObject(CudaZeroHandler.java:1130) | |
| at org.nd4j.jita.allocator.impl.AtomicAllocator.purgeZeroObject(AtomicAllocator.java:443) | |
| at org.nd4j.jita.allocator.impl.AtomicAllocator$UnifiedGarbageCollectorThread.run(AtomicAllocator.java:658) | |
| Caused by: java.lang.InterruptedException | |
| at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302) | |
| at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) | |
| at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.ensureCacheHolder(CudaCachingZeroProvider.java:125) | |
| ... 6 more | |
| [WARNING] | |
| java.lang.RuntimeException: java.lang.InterruptedException | |
| at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.ensureCacheHolder(CudaCachingZeroProvider.java:130) | |
| at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.free(CudaCachingZeroProvider.java:160) | |
| at org.nd4j.jita.memory.impl.CudaFullCachingProvider.free(CudaFullCachingProvider.java:146) | |
| at org.nd4j.jita.handler.impl.CudaZeroHandler.free(CudaZeroHandler.java:480) | |
| at org.nd4j.jita.handler.impl.CudaZeroHandler.purgeZeroObject(CudaZeroHandler.java:1130) | |
| at org.nd4j.jita.allocator.impl.AtomicAllocator.purgeZeroObject(AtomicAllocator.java:443) | |
| at org.nd4j.jita.allocator.impl.AtomicAllocator$UnifiedGarbageCollectorThread.run(AtomicAllocator.java:658) | |
| Caused by: java.lang.InterruptedException | |
| at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302) | |
| at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) | |
| at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.ensureCacheHolder(CudaCachingZeroProvider.java:125) | |
| ... 6 more | |
| [WARNING] thread Thread[ResourceGC thread 0,5,dlchat.EncoderDecoderLSTM] was interrupted but is still alive after waiting at least 15000msecs | |
| [WARNING] thread Thread[ResourceGC thread 0,5,dlchat.EncoderDecoderLSTM] will linger despite being asked to die via interruption | |
| [WARNING] thread Thread[ResourceGC thread 1,5,dlchat.EncoderDecoderLSTM] will linger despite being asked to die via interruption | |
| [WARNING] thread Thread[ResourceGC thread 2,5,dlchat.EncoderDecoderLSTM] will linger despite being asked to die via interruption | |
| [WARNING] thread Thread[ResourceGC thread 3,5,dlchat.EncoderDecoderLSTM] will linger despite being asked to die via interruption | |
| [WARNING] thread Thread[UniGC thread 1,5,dlchat.EncoderDecoderLSTM] will linger despite being asked to die via interruption | |
| [WARNING] thread Thread[NativeRandomDeallocator thread 0,5,dlchat.EncoderDecoderLSTM] will linger despite being asked to die via interruption | |
| [WARNING] thread Thread[Thread-22,5,dlchat.EncoderDecoderLSTM] will linger despite being asked to die via interruption | |
| [WARNING] NOTE: 7 thread(s) did not finish despite being asked to via interruption. This is not a problem with exec:java, it is a problem with the running code. Although not serious, it should be remedied. | |
| [WARNING] Couldn't destroy threadgroup org.codehaus.mojo.exec.ExecJavaMojo$IsolatedThreadGroup[name=dlchat.EncoderDecoderLSTM,maxpri=10] | |
| java.lang.IllegalThreadStateException | |
| at java.lang.ThreadGroup.destroy(ThreadGroup.java:778) | |
| at org.codehaus.mojo.exec.ExecJavaMojo.execute(ExecJavaMojo.java:328) | |
| at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) | |
| at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207) | |
| at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) | |
| at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) | |
| at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116) | |
| at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80) | |
| at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) | |
| at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128) | |
| at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307) | |
| at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193) | |
| at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106) | |
| at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863) | |
| at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288) | |
| at org.apache.maven.cli.MavenCli.main(MavenCli.java:199) | |
| at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) | |
| at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) | |
| at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | |
| at java.lang.reflect.Method.invoke(Method.java:498) | |
| at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) | |
| at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) | |
| at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) | |
| at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) | |
| [INFO] ------------------------------------------------------------------------ | |
| [INFO] BUILD FAILURE | |
| [INFO] ------------------------------------------------------------------------ | |
| [INFO] Total time: 21.761 s | |
| [INFO] Finished at: 2017-03-30T11:22:01+03:00 | |
| [INFO] Final Memory: 77M/2078M | |
| [INFO] ------------------------------------------------------------------------ | |
| [WARNING] | |
| java.lang.RuntimeException: java.lang.InterruptedException | |
| at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.ensureCacheHolder(CudaCachingZeroProvider.java:130) | |
| at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.free(CudaCachingZeroProvider.java:160) | |
| at org.nd4j.jita.memory.impl.CudaFullCachingProvider.free(CudaFullCachingProvider.java:146) | |
| at org.nd4j.jita.handler.impl.CudaZeroHandler.free(CudaZeroHandler.java:480) | |
| at org.nd4j.jita.handler.impl.CudaZeroHandler.purgeZeroObject(CudaZeroHandler.java:1130) | |
| at org.nd4j.jita.allocator.impl.AtomicAllocator.purgeZeroObject(AtomicAllocator.java:443) | |
| at org.nd4j.jita.allocator.impl.AtomicAllocator$UnifiedGarbageCollectorThread.run(AtomicAllocator.java:658) | |
| Caused by: java.lang.InterruptedException | |
| at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1302) | |
| at java.util.concurrent.Semaphore.acquire(Semaphore.java:312) | |
| at org.nd4j.jita.memory.impl.CudaCachingZeroProvider.ensureCacheHolder(CudaCachingZeroProvider.java:125) | |
| ... 6 more | |
| [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.5.0:java (default-cli) on project dlchat: An exception occured while executing the Java class. null: InvocationTargetException: MemcpyAsync failed: AllocationShape(offset=0, length=288, stride=1, elementSize=4, dataType=FLOAT) -> [Help 1] | |
| [ERROR] | |
| [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. | |
| [ERROR] Re-run Maven using the -X switch to enable full debug logging. | |
| [ERROR] | |
| [ERROR] For more information about the errors and possible solutions, please read the following articles: | |
| [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment