jax_cuda_allocation2.txt

batch_size 1
batch_size 2
batch_size 4
batch_size 8
batch_size 16
batch_size 32
batch_size 64
batch_size 128
batch_size 256
batch_size 512
2019-06-10 22:17:12.532436: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 7.34G (7886798848 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
batch_size 600
2019-06-10 22:17:13.803746: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 499.75M (524025856 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:13.805103: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 499.75M (524025856 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:13.806284: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 1.53G (1638948864 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:13.807533: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 1.53G (1638948864 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:13.848989: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 1.19G (1275068416 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:13.850206: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 1.19GiB (1275068416B) on device ordinal 0
2019-06-10 22:17:13.943128: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 316.00M (331350016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:13.943978: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 316.00MiB (331350016B) on device ordinal 0
2019-06-10 22:17:14.048656: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 316.00M (331350016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:14.049515: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 316.00MiB (331350016B) on device ordinal 0
2019-06-10 22:17:14.152965: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 316.00M (331350016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:14.153825: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 316.00MiB (331350016B) on device ordinal 0
2019-06-10 22:17:14.261369: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 316.00M (331350016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:14.262308: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 316.00MiB (331350016B) on device ordinal 0
2019-06-10 22:17:14.369720: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 616.00M (645922816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:14.370655: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 616.00MiB (645922816B) on device ordinal 0
2019-06-10 22:17:14.494047: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 166.00M (174063616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:14.495201: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 166.00MiB (174063616B) on device ordinal 0
2019-06-10 22:17:14.618670: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 166.00M (174063616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:14.621021: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 166.00MiB (174063616B) on device ordinal 0
2019-06-10 22:17:14.744821: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 166.00M (174063616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:14.746863: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 166.00MiB (174063616B) on device ordinal 0
2019-06-10 22:17:14.869085: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 166.00M (174063616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:14.872209: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 166.00MiB (174063616B) on device ordinal 0
2019-06-10 22:17:14.995290: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 166.00M (174063616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:14.996423: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 166.00MiB (174063616B) on device ordinal 0
2019-06-10 22:17:15.118034: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 166.00M (174063616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:15.119173: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 166.00MiB (174063616B) on device ordinal 0
2019-06-10 22:17:15.243076: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 166.00M (174063616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:15.244208: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 166.00MiB (174063616B) on device ordinal 0
2019-06-10 22:17:15.367508: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 166.00M (174063616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:15.368650: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 166.00MiB (174063616B) on device ordinal 0
2019-06-10 22:17:15.495982: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 166.00M (174063616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:15.498256: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 166.00MiB (174063616B) on device ordinal 0
2019-06-10 22:17:15.589421: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 166.00M (174063616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:15.590655: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 166.00MiB (174063616B) on device ordinal 0
2019-06-10 22:17:15.607971: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 616.00M (645922816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:15.608029: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 616.00MiB (645922816B) on device ordinal 0
2019-06-10 22:17:15.673801: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 316.00M (331350016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:15.675585: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 316.00MiB (331350016B) on device ordinal 0
2019-06-10 22:17:15.756095: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 316.00M (331350016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:15.757488: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 316.00MiB (331350016B) on device ordinal 0
2019-06-10 22:17:15.831842: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 316.00M (331350016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:15.833157: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 316.00MiB (331350016B) on device ordinal 0
2019-06-10 22:17:15.911936: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 316.00M (331350016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:15.912903: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 316.00MiB (331350016B) on device ordinal 0
2019-06-10 22:17:15.990003: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 316.00M (331350016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:15.990807: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 316.00MiB (331350016B) on device ordinal 0
2019-06-10 22:17:16.070090: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 316.00M (331350016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:16.070696: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 316.00MiB (331350016B) on device ordinal 0
2019-06-10 22:17:16.149617: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 316.00M (331350016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:16.150236: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 316.00MiB (331350016B) on device ordinal 0
2019-06-10 22:17:16.230101: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 316.00M (331350016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:16.230981: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 316.00MiB (331350016B) on device ordinal 0
2019-06-10 22:17:16.240233: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 1.19G (1275068416 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:16.240672: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 1.19GiB (1275068416B) on device ordinal 0
2019-06-10 22:17:16.378905: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 2.65G (2847932416 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:16.379883: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 2.65G (2847932416 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:16.418920: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 700.00M (734003200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:16.419861: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 700.00M (734003200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:16.501216: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 616.00M (645922816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:16.501825: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 616.00MiB (645922816B) on device ordinal 0
2019-06-10 22:17:16.555829: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 728.50M (763887616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:16.557110: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 728.50M (763887616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:16.646843: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 353.50M (370671616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:16.647773: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 353.50M (370671616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:16.648720: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 499.75M (524025856 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:16.649635: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 499.75M (524025856 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:16.667067: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 1.53G (1638948864 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:16.668200: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 1.53G (1638948864 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
batch_size 700
2019-06-10 22:17:18.886930: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 556.00M (583008256 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:18.888034: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 556.00M (583008256 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:18.889149: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 1.73G (1861771264 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:18.890321: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 1.73G (1861771264 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:18.952128: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 1.38G (1484783616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:18.953245: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 1.38GiB (1484783616B) on device ordinal 0
2019-06-10 22:17:19.031929: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 366.00M (383778816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:19.032851: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 366.00MiB (383778816B) on device ordinal 0
2019-06-10 22:17:19.121167: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 366.00M (383778816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:19.122083: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 366.00MiB (383778816B) on device ordinal 0
2019-06-10 22:17:19.212384: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 366.00M (383778816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:19.213283: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 366.00MiB (383778816B) on device ordinal 0
2019-06-10 22:17:19.300781: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 366.00M (383778816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:19.301681: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 366.00MiB (383778816B) on device ordinal 0
2019-06-10 22:17:19.392604: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 716.00M (750780416 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:19.393502: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 716.00MiB (750780416B) on device ordinal 0
2019-06-10 22:17:19.534670: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 191.00M (200278016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:19.537247: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 191.00MiB (200278016B) on device ordinal 0
2019-06-10 22:17:19.678570: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 191.00M (200278016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:19.680989: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 191.00MiB (200278016B) on device ordinal 0
2019-06-10 22:17:19.822307: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 191.00M (200278016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:19.825466: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 191.00MiB (200278016B) on device ordinal 0
2019-06-10 22:17:19.962057: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 191.00M (200278016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:19.963257: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 191.00MiB (200278016B) on device ordinal 0
2019-06-10 22:17:20.073613: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 191.00M (200278016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:20.075902: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 191.00MiB (200278016B) on device ordinal 0
2019-06-10 22:17:20.185423: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 191.00M (200278016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:20.186707: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 191.00MiB (200278016B) on device ordinal 0
2019-06-10 22:17:20.297310: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 191.00M (200278016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:20.298609: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 191.00MiB (200278016B) on device ordinal 0
2019-06-10 22:17:20.408133: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 191.00M (200278016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:20.409792: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 191.00MiB (200278016B) on device ordinal 0
2019-06-10 22:17:20.521034: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 191.00M (200278016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:20.523132: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 191.00MiB (200278016B) on device ordinal 0
2019-06-10 22:17:20.660954: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 191.00M (200278016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:20.662343: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 191.00MiB (200278016B) on device ordinal 0
2019-06-10 22:17:20.679203: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 716.00M (750780416 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:20.679259: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 716.00MiB (750780416B) on device ordinal 0
2019-06-10 22:17:20.790069: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 366.00M (383778816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:20.791549: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 366.00MiB (383778816B) on device ordinal 0
2019-06-10 22:17:20.920092: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 366.00M (383778816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:20.921598: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 366.00MiB (383778816B) on device ordinal 0
2019-06-10 22:17:21.051492: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 366.00M (383778816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:21.052950: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 366.00MiB (383778816B) on device ordinal 0
2019-06-10 22:17:21.177293: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 366.00M (383778816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:21.177999: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 366.00MiB (383778816B) on device ordinal 0
2019-06-10 22:17:21.302353: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 366.00M (383778816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:21.303014: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 366.00MiB (383778816B) on device ordinal 0
2019-06-10 22:17:21.401560: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 366.00M (383778816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:21.402224: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 366.00MiB (383778816B) on device ordinal 0
2019-06-10 22:17:21.528947: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 366.00M (383778816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:21.529603: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 366.00MiB (383778816B) on device ordinal 0
2019-06-10 22:17:21.659703: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 366.00M (383778816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:21.661360: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 366.00MiB (383778816B) on device ordinal 0
2019-06-10 22:17:21.673797: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 1.38G (1484783616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:21.674218: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 1.38GiB (1484783616B) on device ordinal 0
2019-06-10 22:17:21.875324: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 3.09G (3319791616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:21.876781: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 3.09G (3319791616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:21.923255: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 812.50M (851968000 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:21.924711: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 812.50M (851968000 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:22.018791: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 716.00M (750780416 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:22.019795: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 716.00MiB (750780416B) on device ordinal 0
2019-06-10 22:17:22.082995: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 847.25M (888406016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:22.084274: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 847.25M (888406016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:22.187030: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 409.75M (429654016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:22.188174: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 409.75M (429654016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:22.189282: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 556.00M (583008256 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:22.190436: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 556.00M (583008256 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:22.214886: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 1.73G (1861771264 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:22.215820: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 1.73G (1861771264 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
batch_size 800
2019-06-10 22:17:24.874898: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 612.25M (641990656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:24.875887: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 612.25M (641990656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:24.876794: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 1.94G (2084593664 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:24.877684: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 1.94G (2084593664 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:24.898972: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 250.00M (262144000 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:24.899969: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 250.00M (262144000 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:24.919541: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 1.58G (1694498816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:24.920260: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 1.58GiB (1694498816B) on device ordinal 0
2019-06-10 22:17:25.014049: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 416.00M (436207616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:25.015598: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 416.00MiB (436207616B) on device ordinal 0
2019-06-10 22:17:25.118588: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 416.00M (436207616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:25.120347: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 416.00MiB (436207616B) on device ordinal 0
2019-06-10 22:17:25.223596: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 416.00M (436207616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:25.224552: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 416.00MiB (436207616B) on device ordinal 0
2019-06-10 22:17:25.325619: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 416.00M (436207616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:25.326629: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 416.00MiB (436207616B) on device ordinal 0
2019-06-10 22:17:25.432368: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 816.00M (855638016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:25.433322: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 816.00MiB (855638016B) on device ordinal 0
2019-06-10 22:17:25.552150: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 216.00M (226492416 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:25.553389: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 216.00MiB (226492416B) on device ordinal 0
2019-06-10 22:17:25.677032: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 216.00M (226492416 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:25.678419: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 216.00MiB (226492416B) on device ordinal 0
2019-06-10 22:17:25.801426: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 216.00M (226492416 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:25.803014: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 216.00MiB (226492416B) on device ordinal 0
2019-06-10 22:17:25.930168: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 216.00M (226492416 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:25.931448: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 216.00MiB (226492416B) on device ordinal 0
2019-06-10 22:17:26.053618: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 216.00M (226492416 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:26.056847: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 216.00MiB (226492416B) on device ordinal 0
2019-06-10 22:17:26.185060: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 216.00M (226492416 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:26.188765: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 216.00MiB (226492416B) on device ordinal 0
2019-06-10 22:17:26.314430: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 216.00M (226492416 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:26.316559: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 216.00MiB (226492416B) on device ordinal 0
2019-06-10 22:17:26.445008: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 216.00M (226492416 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:26.446270: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 216.00MiB (226492416B) on device ordinal 0
2019-06-10 22:17:26.570414: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 216.00M (226492416 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:26.572827: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 216.00MiB (226492416B) on device ordinal 0
2019-06-10 22:17:26.698375: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 216.00M (226492416 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:26.702021: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 216.00MiB (226492416B) on device ordinal 0
2019-06-10 22:17:26.719399: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 816.00M (855638016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:26.719452: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 816.00MiB (855638016B) on device ordinal 0
2019-06-10 22:17:26.807375: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 416.00M (436207616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:26.808327: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 416.00MiB (436207616B) on device ordinal 0
2019-06-10 22:17:26.907845: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 416.00M (436207616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:26.909373: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 416.00MiB (436207616B) on device ordinal 0
2019-06-10 22:17:27.009518: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 416.00M (436207616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:27.010541: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 416.00MiB (436207616B) on device ordinal 0
2019-06-10 22:17:27.113479: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 416.00M (436207616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:27.114233: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 416.00MiB (436207616B) on device ordinal 0
2019-06-10 22:17:27.219821: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 416.00M (436207616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:27.220970: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 416.00MiB (436207616B) on device ordinal 0
2019-06-10 22:17:27.324839: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 416.00M (436207616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:27.325923: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 416.00MiB (436207616B) on device ordinal 0
2019-06-10 22:17:27.423484: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 416.00M (436207616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:27.424430: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 416.00MiB (436207616B) on device ordinal 0
2019-06-10 22:17:27.525955: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 416.00M (436207616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:27.527443: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 416.00MiB (436207616B) on device ordinal 0
2019-06-10 22:17:27.556587: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 1.58G (1694498816 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:27.557335: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 1.58GiB (1694498816B) on device ordinal 0
2019-06-10 22:17:27.669863: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 18.25M (19136512 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:27.674785: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 18.25MiB (19136512B) on device ordinal 0
2019-06-10 22:17:27.805918: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 18.25M (19136512 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:27.808401: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 18.25MiB (19136512B) on device ordinal 0
2019-06-10 22:17:27.931332: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 18.25M (19136512 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:27.932592: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 18.25MiB (19136512B) on device ordinal 0
2019-06-10 22:17:28.055548: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 18.25M (19136512 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:28.057869: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 18.25MiB (19136512B) on device ordinal 0
2019-06-10 22:17:28.160126: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 816.00M (855638016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:28.161929: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 816.00MiB (855638016B) on device ordinal 0
2019-06-10 22:17:28.242816: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 966.00M (1012924416 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:28.243744: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 966.00M (1012924416 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:28.359089: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 466.00M (488636416 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:28.360305: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 466.00M (488636416 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:28.361456: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 612.25M (641990656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:28.362579: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 612.25M (641990656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:28.382955: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 1.94G (2084593664 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:28.384116: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 1.94G (2084593664 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:28.385241: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 250.00M (262144000 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:28.386299: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 250.00M (262144000 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:28.527741: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:828] failed to allocate 18.25M (19136512 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-06-10 22:17:28.529088: E external/org_tensorflow/tensorflow/compiler/xla/service/gpu/cudnn_conv_algorithm_picker.cc:555] Resource exhausted: Failed to allocate request for 18.25MiB (19136512B) on device ordinal 0