Kubernetes error nvidia/cuda:8.0-cudnn7-devel

error_nvidia_cuda_8_0-cudnn7_devel.txt

Connecting to sql postgresql+psycopg2://postgres@pgbouncer:6432/tensorflight
{"message":"Starting processing plan 71d37138-38b2-40dc-9710-8115d7f8228f","plan_id":"71d37138-38b2-40dc-9710-8115d7f8228f","severity":"INFO","thread":140054849971968}
{"message":"Connecting to the sql database","severity":"INFO","thread":140054849971968}
{"message":"Disconnecting from the sql database","severity":"INFO","thread":140054849971968}
[03:18:58] /workspace/mxnet/dmlc-core/include/dmlc/./logging.h:308: [03:18:58] /workspace/mxnet/dmlc-core/include/dmlc/./logging.h:308: [03:18:58] /workspace/mxnet/mshadow/mshadow/./tensor_gpu-inl.h:35: Check failed: e == cudaSuccess CUDA: unknown error

Stack trace returned 7 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet-0.12.0-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f60eb428bec]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet-0.12.0-py2.7.egg/mxnet/libmxnet.so(_ZN7mshadow9SetDeviceINS_3gpuEEEvi+0xd0) [0x7f60ed877690]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet-0.12.0-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEZZNS2_23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlS5_E_E9_M_invokeERKSt9_Any_dataOS5_+0x97) [0x7f60ed87fb37]
[bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet-0.12.0-py2.7.egg/mxnet/libmxnet.so(_ZNSt6thread5_ImplISt12_Bind_simpleIFSt8functionIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEES8_EEE6_M_runEv+0x4a) [0x7f60ed879bea]
[bt] (4) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80) [0x7f610b257c80]
[bt] (5) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f610f5256ba]
[bt] (6) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f610f25b3dd]

terminate called after throwing an instance of 'dmlc::Error'
  what():  [03:18:58] /workspace/mxnet/mshadow/mshadow/./tensor_gpu-inl.h:35: Check failed: e == cudaSuccess CUDA: unknown error

Stack trace returned 7 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet-0.12.0-py2.7.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f60eb428bec]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet-0.12.0-py2.7.egg/mxnet/libmxnet.so(_ZN7mshadow9SetDeviceINS_3gpuEEEvi+0xd0) [0x7f60ed877690]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet-0.12.0-py2.7.egg/mxnet/libmxnet.so(_ZNSt17_Function_handlerIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEZZNS2_23ThreadedEnginePerDevice13PushToExecuteEPNS2_8OprBlockEbENKUlvE1_clEvEUlS5_E_E9_M_invokeERKSt9_Any_dataOS5_+0x97) [0x7f60ed87fb37]
[bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet-0.12.0-py2.7.egg/mxnet/libmxnet.so(_ZNSt6thread5_ImplISt12_Bind_simpleIFSt8functionIFvSt10shared_ptrIN5mxnet6engine10ThreadPool11SimpleEventEEEES8_EEE6_M_runEv+0x4a) [0x7f60ed879bea]
[bt] (4) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80) [0x7f610b257c80]
[bt] (5) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f610f5256ba]
[bt] (6) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f610f25b3dd]