Skip to content

Instantly share code, notes, and snippets.

@munyari

munyari/tmp

Created March 16, 2017 03:51
Show Gist options
  • Save munyari/950049faba75dc3cd6ade87328d478bc to your computer and use it in GitHub Desktop.
Save munyari/950049faba75dc3cd6ade87328d478bc to your computer and use it in GitHub Desktop.
Download ZIP
Raw
tmp
2017-03-15 20:04:49,273 INFO - Preparing to run TaskInstance <TaskInstance: munyari/cats-dogs:49 (id: J3PHVrEZzjYAfp6THbaVfn) (checksum: 481cad7c140aaebf6f3e857b29f25813) (last update: 2017-03-15 20:04:49.233048) [queued]>
2017-03-15 20:04:49,282 INFO - Starting attempt 1 at 2017-03-15 20:04:49.276406
2017-03-15 20:04:49,549 INFO - Executing command in container: stdbuf -o0 sh command.sh
2017-03-15 20:04:49,549 INFO - Pulling Docker image: floydhub/tensorflow:latest-gpu-py3
2017-03-15 20:04:50,735 INFO - Starting container...
2017-03-15 20:04:51,006 INFO -
################################################################################
2017-03-15 20:04:51,006 INFO - Run Output:
2017-03-15 20:09:35,371 INFO - I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so locally
2017-03-15 20:09:35,373 INFO - I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.so locally
2017-03-15 20:09:35,374 INFO - I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so locally
2017-03-15 20:09:35,375 INFO - I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally
2017-03-15 20:09:35,375 INFO - I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so locally
2017-03-15 20:09:39,725 INFO - I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2017-03-15 20:09:39,725 INFO - I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties:
2017-03-15 20:09:39,726 INFO - name: Tesla K80
2017-03-15 20:09:39,726 INFO - major: 3 minor: 7 memoryClockRate (GHz) 0.8235
2017-03-15 20:09:39,727 INFO - pciBusID 0000:00:1e.0
2017-03-15 20:09:39,727 INFO - Total memory: 11.17GiB
2017-03-15 20:09:39,727 INFO - Free memory: 11.11GiB
2017-03-15 20:09:39,728 INFO - I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0
2017-03-15 20:09:39,728 INFO - I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0: Y
2017-03-15 20:09:39,729 INFO - I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0)
2017-03-15 20:09:44,239 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 3.45GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-03-15 20:09:44,242 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 3.05GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-03-15 20:09:45,476 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 2.58GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-03-15 20:09:46,733 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 2.40GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-03-15 20:09:47,996 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 2.40GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-03-15 20:09:48,011 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 1.34GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-03-15 20:09:48,662 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 2.40GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-03-15 20:09:49,309 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 2.22GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-03-15 20:09:50,723 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 2.58GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-03-15 20:09:57,614 INFO - Found 23000 images belonging to 2 classes.
2017-03-15 20:09:57,616 INFO - Found 2000 images belonging to 2 classes.
2017-03-15 20:09:57,617 INFO - Epoch 1/20
2017-03-15 20:09:57,722 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 1.34GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-03-15 20:10:08,639 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (256): Total Chunks: 1, Chunks in use: 0 256B allocated for chunks. 8B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:10:08,645 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (512): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:10:08,645 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1024): Total Chunks: 1, Chunks in use: 0 1.0KiB allocated for chunks. 512B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:10:08,646 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2048): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:10:08,646 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4096): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:10:08,646 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8192): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:10:08,647 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16384): Total Chunks: 1, Chunks in use: 0 16.0KiB allocated for chunks. 16.0KiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:10:08,647 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (32768): Total Chunks: 3, Chunks in use: 0 128.0KiB allocated for chunks. 96.0KiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:10:08,647 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (65536): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:10:08,647 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (131072): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:10:08,648 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (262144): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:10:08,648 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (524288): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:10:08,648 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1048576): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:10:08,648 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2097152): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:10:08,649 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4194304): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:10:08,649 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8388608): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:10:08,650 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16777216): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:10:08,650 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (33554432): Total Chunks: 1, Chunks in use: 0 57.88MiB allocated for chunks. 16.0KiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:10:08,651 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (67108864): Total Chunks: 1, Chunks in use: 0 68.00MiB allocated for chunks. 1.00MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:10:08,651 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (134217728): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:10:08,652 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (268435456): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:10:08,652 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:660] Bin for 392.00MiB was 256.00MiB, Chunk State:
2017-03-15 20:10:08,653 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0000 of size 1280
2017-03-15 20:10:08,654 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0500 of size 256
2017-03-15 20:10:08,655 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0600 of size 256
2017-03-15 20:10:08,655 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0700 of size 256
2017-03-15 20:10:08,655 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0800 of size 256
2017-03-15 20:10:08,656 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0900 of size 512
2017-03-15 20:10:08,657 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0b00 of size 256
2017-03-15 20:10:08,661 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0c00 of size 256
2017-03-15 20:10:08,661 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0d00 of size 1024
2017-03-15 20:10:08,661 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e1100 of size 256
2017-03-15 20:10:08,662 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e1200 of size 256
2017-03-15 20:10:08,662 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e1300 of size 2048
2017-03-15 20:10:08,662 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e1b00 of size 256
2017-03-15 20:10:08,662 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e1c00 of size 256
2017-03-15 20:10:08,663 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e1d00 of size 16384
2017-03-15 20:10:08,663 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e5d00 of size 256
2017-03-15 20:10:08,663 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e5e00 of size 256
2017-03-15 20:10:08,663 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e5f00 of size 256
2017-03-15 20:10:08,663 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e6000 of size 256
2017-03-15 20:10:08,664 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e6100 of size 256
2017-03-15 20:10:08,664 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e6200 of size 256
2017-03-15 20:10:08,664 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e6300 of size 256
2017-03-15 20:10:08,664 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e6400 of size 9437184
2017-03-15 20:10:08,669 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12059e6400 of size 6912
2017-03-15 20:10:08,669 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12059e7f00 of size 147456
2017-03-15 20:10:08,669 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205a0bf00 of size 294912
2017-03-15 20:10:08,670 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205a53f00 of size 589824
2017-03-15 20:10:08,670 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205ae3f00 of size 1179648
2017-03-15 20:10:08,670 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205c03f00 of size 2359296
2017-03-15 20:10:08,670 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205e43f00 of size 4718592
2017-03-15 20:10:08,671 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12062c3f00 of size 411041792
2017-03-15 20:10:08,671 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x121eac3f00 of size 67108864
2017-03-15 20:10:08,671 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ac3f00 of size 32768
2017-03-15 20:10:08,672 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acbf00 of size 256
2017-03-15 20:10:08,672 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc000 of size 256
2017-03-15 20:10:08,672 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc100 of size 256
2017-03-15 20:10:08,673 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc200 of size 256
2017-03-15 20:10:08,673 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc300 of size 256
2017-03-15 20:10:08,673 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc400 of size 256
2017-03-15 20:10:08,673 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc500 of size 256
2017-03-15 20:10:08,674 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc600 of size 256
2017-03-15 20:10:08,674 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc700 of size 256
2017-03-15 20:10:08,675 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc800 of size 256
2017-03-15 20:10:08,675 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc900 of size 256
2017-03-15 20:10:08,676 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acca00 of size 256
2017-03-15 20:10:08,677 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222accb00 of size 256
2017-03-15 20:10:08,678 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222accc00 of size 256
2017-03-15 20:10:08,681 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222accd00 of size 256
2017-03-15 20:10:08,681 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acce00 of size 256
2017-03-15 20:10:08,681 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222accf00 of size 256
2017-03-15 20:10:08,682 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acd000 of size 256
2017-03-15 20:10:08,682 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acd100 of size 256
2017-03-15 20:10:08,682 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acd200 of size 256
2017-03-15 20:10:08,682 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acd300 of size 256
2017-03-15 20:10:08,683 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acd800 of size 256
2017-03-15 20:10:08,683 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acd900 of size 256
2017-03-15 20:10:08,683 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acdb00 of size 256
2017-03-15 20:10:08,684 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acdc00 of size 256
2017-03-15 20:10:08,685 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acdd00 of size 256
2017-03-15 20:10:08,686 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acde00 of size 256
2017-03-15 20:10:08,686 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acdf00 of size 512
2017-03-15 20:10:08,687 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ace100 of size 512
2017-03-15 20:10:08,688 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ace300 of size 256
2017-03-15 20:10:08,689 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ace400 of size 256
2017-03-15 20:10:08,689 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ace500 of size 256
2017-03-15 20:10:08,690 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ace600 of size 6912
2017-03-15 20:10:08,690 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ad0100 of size 32768
2017-03-15 20:10:08,690 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ad8100 of size 16384
2017-03-15 20:10:08,690 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ae8100 of size 6912
2017-03-15 20:10:08,691 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ae9c00 of size 512
2017-03-15 20:10:08,691 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ae9e00 of size 512
2017-03-15 20:10:08,691 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222aea000 of size 1024
2017-03-15 20:10:08,691 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222aea400 of size 1024
2017-03-15 20:10:08,691 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222aea800 of size 1024
2017-03-15 20:10:08,692 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222aeac00 of size 2048
2017-03-15 20:10:08,692 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222aeb400 of size 2048
2017-03-15 20:10:08,692 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222aebc00 of size 27136
2017-03-15 20:10:08,692 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222af2600 of size 256
2017-03-15 20:10:08,694 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222af2700 of size 147456
2017-03-15 20:10:08,694 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222b16700 of size 147456
2017-03-15 20:10:08,694 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222b3a700 of size 512
2017-03-15 20:10:08,694 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222b3a900 of size 294912
2017-03-15 20:10:08,695 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222b82900 of size 294912
2017-03-15 20:10:08,695 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222bca900 of size 512
2017-03-15 20:10:08,695 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222bcab00 of size 589824
2017-03-15 20:10:08,695 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222c5ab00 of size 589824
2017-03-15 20:10:08,696 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ceab00 of size 1024
2017-03-15 20:10:08,696 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ceaf00 of size 3538944
2017-03-15 20:10:08,696 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x122304af00 of size 1179648
2017-03-15 20:10:08,696 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x122316af00 of size 1024
2017-03-15 20:10:08,697 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x122316b300 of size 1024
2017-03-15 20:10:08,698 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x122316b700 of size 2359296
2017-03-15 20:10:08,698 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12233ab700 of size 1179648
2017-03-15 20:10:08,698 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12234cb700 of size 16640
2017-03-15 20:10:08,698 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12234d3800 of size 16384
2017-03-15 20:10:08,699 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12234d7800 of size 16384
2017-03-15 20:10:08,699 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12234db800 of size 32768
2017-03-15 20:10:08,699 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12234ef700 of size 147456
2017-03-15 20:10:08,699 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1223513700 of size 294912
2017-03-15 20:10:08,700 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x122355b700 of size 589824
2017-03-15 20:10:08,700 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12235eb700 of size 2048
2017-03-15 20:10:08,700 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12235ebf00 of size 9437184
2017-03-15 20:10:08,700 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1223eebf00 of size 9437184
2017-03-15 20:10:08,702 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12247ebf00 of size 9437184
2017-03-15 20:10:08,702 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12250ebf00 of size 14155776
2017-03-15 20:10:08,702 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1225e6bf00 of size 4718592
2017-03-15 20:10:08,702 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12262ebf00 of size 2048
2017-03-15 20:10:08,703 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12262ec700 of size 2048
2017-03-15 20:10:08,703 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12262ecf00 of size 2048
2017-03-15 20:10:08,704 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12262ed700 of size 2048
2017-03-15 20:10:08,704 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12262edf00 of size 2048
2017-03-15 20:10:08,705 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12262ee700 of size 9437184
2017-03-15 20:10:08,705 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1226bee700 of size 9437184
2017-03-15 20:10:08,706 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12274ee700 of size 9437184
2017-03-15 20:10:08,706 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1227dee700 of size 9437184
2017-03-15 20:10:08,706 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12286ee700 of size 9437184
2017-03-15 20:10:08,707 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1228fee700 of size 2359296
2017-03-15 20:10:08,707 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x122922e700 of size 2359296
2017-03-15 20:10:08,708 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x122946e700 of size 5505024
2017-03-15 20:10:08,708 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12299ae700 of size 39226368
2017-03-15 20:10:08,708 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x122bf17300 of size 89325568
2017-03-15 20:10:08,709 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1231447300 of size 9437184
2017-03-15 20:10:08,709 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1231d47300 of size 6422528
2017-03-15 20:10:08,709 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1235d47300 of size 9437184
2017-03-15 20:10:08,710 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1236647300 of size 9437184
2017-03-15 20:10:08,710 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1236f47300 of size 9437184
2017-03-15 20:10:08,711 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1237847300 of size 9437184
2017-03-15 20:10:08,711 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1238147300 of size 1179648
2017-03-15 20:10:08,711 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1238267300 of size 2359296
2017-03-15 20:10:08,712 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12384a7300 of size 2359296
2017-03-15 20:10:08,712 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12386e7300 of size 4718592
2017-03-15 20:10:08,713 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1238b67300 of size 9437184
2017-03-15 20:10:08,713 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1239467300 of size 90731520
2017-03-15 20:10:08,713 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x123eaee700 of size 16384
2017-03-15 20:10:08,714 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x123eaf2700 of size 67108864
2017-03-15 20:10:08,714 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242af2700 of size 16384
2017-03-15 20:10:08,714 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242af6700 of size 1024
2017-03-15 20:10:08,715 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242af6b00 of size 1024
2017-03-15 20:10:08,715 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242af6f00 of size 1024
2017-03-15 20:10:08,716 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242af7300 of size 3840
2017-03-15 20:10:08,716 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242af8200 of size 2048
2017-03-15 20:10:08,716 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242af8a00 of size 2048
2017-03-15 20:10:08,717 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242af9200 of size 2048
2017-03-15 20:10:08,717 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242af9a00 of size 2048
2017-03-15 20:10:08,718 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afa200 of size 6912
2017-03-15 20:10:08,718 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afbd00 of size 2048
2017-03-15 20:10:08,718 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afc500 of size 2048
2017-03-15 20:10:08,719 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afcd00 of size 2048
2017-03-15 20:10:08,719 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afd500 of size 2048
2017-03-15 20:10:08,720 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afdd00 of size 2560
2017-03-15 20:10:08,720 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afe700 of size 256
2017-03-15 20:10:08,720 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afe800 of size 256
2017-03-15 20:10:08,721 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afe900 of size 256
2017-03-15 20:10:08,721 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afea00 of size 256
2017-03-15 20:10:08,722 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afeb00 of size 256
2017-03-15 20:10:08,722 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afec00 of size 256
2017-03-15 20:10:08,722 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afed00 of size 9437184
2017-03-15 20:10:08,723 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12433fed00 of size 2048
2017-03-15 20:10:08,723 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12433ff500 of size 9437184
2017-03-15 20:10:08,724 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1243cff500 of size 2048
2017-03-15 20:10:08,724 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1243cffd00 of size 9437184
2017-03-15 20:10:08,725 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12445ffd00 of size 2048
2017-03-15 20:10:08,725 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244600500 of size 9437184
2017-03-15 20:10:08,725 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244f00500 of size 2048
2017-03-15 20:10:08,725 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244f00d00 of size 6912
2017-03-15 20:10:08,726 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244f02800 of size 256
2017-03-15 20:10:08,726 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244f02900 of size 147456
2017-03-15 20:10:08,726 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244f26900 of size 256
2017-03-15 20:10:08,726 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244f26a00 of size 294912
2017-03-15 20:10:08,727 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244f6ea00 of size 512
2017-03-15 20:10:08,727 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244f6ec00 of size 589824
2017-03-15 20:10:08,727 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244ffec00 of size 512
2017-03-15 20:10:08,728 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244ffee00 of size 1179648
2017-03-15 20:10:08,728 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x124511ee00 of size 1024
2017-03-15 20:10:08,728 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x124511f200 of size 2359296
2017-03-15 20:10:08,729 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x124535f200 of size 1024
2017-03-15 20:10:08,730 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x124535f600 of size 2359296
2017-03-15 20:10:08,730 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x124559f600 of size 1024
2017-03-15 20:10:08,730 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x124559fa00 of size 4718592
2017-03-15 20:10:08,731 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1245a1fa00 of size 2048
2017-03-15 20:10:08,731 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1245a20200 of size 9437184
2017-03-15 20:10:08,731 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1246320200 of size 2048
2017-03-15 20:10:08,731 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1246320a00 of size 411041792
2017-03-15 20:10:08,732 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x125eb20a00 of size 16384
2017-03-15 20:10:08,732 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x125eb24a00 of size 67108864
2017-03-15 20:10:08,732 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1262b24a00 of size 16384
2017-03-15 20:10:08,732 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1262b28a00 of size 32768
2017-03-15 20:10:08,733 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1262b30a00 of size 256
2017-03-15 20:10:08,733 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1262b30b00 of size 9437184
2017-03-15 20:10:08,733 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1263430b00 of size 2048
2017-03-15 20:10:08,733 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1263431300 of size 9437184
2017-03-15 20:10:08,735 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1263d31300 of size 2048
2017-03-15 20:10:08,735 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1263d31b00 of size 9437184
2017-03-15 20:10:08,735 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264631b00 of size 2048
2017-03-15 20:10:08,736 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264632300 of size 9437184
2017-03-15 20:10:08,736 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264f32300 of size 2048
2017-03-15 20:10:08,736 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264f32b00 of size 6912
2017-03-15 20:10:08,736 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264f34600 of size 256
2017-03-15 20:10:08,737 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264f34700 of size 147456
2017-03-15 20:10:08,737 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264f58700 of size 256
2017-03-15 20:10:08,737 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264f58800 of size 294912
2017-03-15 20:10:08,737 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264fa0800 of size 512
2017-03-15 20:10:08,738 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264fa0a00 of size 589824
2017-03-15 20:10:08,738 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1265030a00 of size 512
2017-03-15 20:10:08,738 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1265030c00 of size 1179648
2017-03-15 20:10:08,738 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1265150c00 of size 1024
2017-03-15 20:10:08,739 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1265151000 of size 2359296
2017-03-15 20:10:08,739 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1265391000 of size 1024
2017-03-15 20:10:08,739 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1265391400 of size 2359296
2017-03-15 20:10:08,739 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12655d1400 of size 1024
2017-03-15 20:10:08,740 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12655d1800 of size 4718592
2017-03-15 20:10:08,740 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1265a51800 of size 2048
2017-03-15 20:10:08,740 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1265a52000 of size 9437184
2017-03-15 20:10:08,740 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1266352000 of size 2048
2017-03-15 20:10:08,740 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1266352800 of size 411041792
2017-03-15 20:10:08,742 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x127eb52800 of size 16384
2017-03-15 20:10:08,742 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x127eb56800 of size 67108864
2017-03-15 20:10:08,743 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1282b56800 of size 16384
2017-03-15 20:10:08,743 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1282b5a800 of size 32768
2017-03-15 20:10:08,744 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1282b62800 of size 256
2017-03-15 20:10:08,744 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1282b62900 of size 429775616
2017-03-15 20:10:08,744 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x129c540400 of size 822083584
2017-03-15 20:10:08,745 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12cd540400 of size 836829184
2017-03-15 20:10:08,745 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12ff350400 of size 822083584
2017-03-15 20:10:08,746 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1330350400 of size 205520896
2017-03-15 20:10:08,746 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x133c750400 of size 212926464
2017-03-15 20:10:08,746 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1349260400 of size 411041792
2017-03-15 20:10:08,747 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1361a60400 of size 829423616
2017-03-15 20:10:08,747 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1393160400 of size 411041792
2017-03-15 20:10:08,748 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13ab960400 of size 411041792
2017-03-15 20:10:08,748 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13c4160400 of size 16384
2017-03-15 20:10:08,748 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13c416c500 of size 411041792
2017-03-15 20:10:08,749 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13dc96c500 of size 102760448
2017-03-15 20:10:08,749 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13e2b6c500 of size 110231552
2017-03-15 20:10:08,750 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13e948c500 of size 205520896
2017-03-15 20:10:08,750 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13f588c500 of size 220463104
2017-03-15 20:10:08,750 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1402acc500 of size 205520896
2017-03-15 20:10:08,751 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x140eecc500 of size 220463104
2017-03-15 20:10:08,751 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x141c10c500 of size 205520896
2017-03-15 20:10:08,752 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x142850c500 of size 58982400
2017-03-15 20:10:08,752 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x142bd4c500 of size 102760448
2017-03-15 20:10:08,753 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1431f4c500 of size 117964800
2017-03-15 20:10:08,753 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1438fcc500 of size 102760448
2017-03-15 20:10:08,753 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x143f1cc500 of size 117964800
2017-03-15 20:10:08,754 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x144624c500 of size 102760448
2017-03-15 20:10:08,754 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x144c44c500 of size 25690112
2017-03-15 20:10:08,755 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x144dccc500 of size 33554432
2017-03-15 20:10:08,755 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x144fccc500 of size 25690112
2017-03-15 20:10:08,755 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x145154c500 of size 33554432
2017-03-15 20:10:08,756 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x145354c500 of size 25690112
2017-03-15 20:10:08,756 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1454dcc500 of size 33554432
2017-03-15 20:10:08,757 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1456dcc500 of size 25690112
2017-03-15 20:10:08,757 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x145864c500 of size 6422528
2017-03-15 20:10:08,757 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x145d06c500 of size 67108864
2017-03-15 20:10:08,758 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x146106c500 of size 411041792
2017-03-15 20:10:08,758 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x147986c500 of size 788662016
2017-03-15 20:10:08,758 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x1222acd400 of size 1024
2017-03-15 20:10:08,758 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x1222acda00 of size 256
2017-03-15 20:10:08,759 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x1222adc100 of size 49152
2017-03-15 20:10:08,759 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x12234cf800 of size 16384
2017-03-15 20:10:08,759 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x12234e3800 of size 48896
2017-03-15 20:10:08,760 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x1232367300 of size 60686336
2017-03-15 20:10:08,761 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x13c4164400 of size 33024
2017-03-15 20:10:08,761 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x1458c6c500 of size 71303168
2017-03-15 20:10:08,762 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:693] Summary of in-use Chunks by size:
2017-03-15 20:10:08,762 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 60 Chunks of size 256 totalling 15.0KiB
2017-03-15 20:10:08,762 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 11 Chunks of size 512 totalling 5.5KiB
2017-03-15 20:10:08,763 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 16 Chunks of size 1024 totalling 16.0KiB
2017-03-15 20:10:08,763 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 1280 totalling 1.2KiB
2017-03-15 20:10:08,763 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 29 Chunks of size 2048 totalling 58.0KiB
2017-03-15 20:10:08,764 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 2560 totalling 2.5KiB
2017-03-15 20:10:08,764 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 3840 totalling 3.8KiB
2017-03-15 20:10:08,764 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 6 Chunks of size 6912 totalling 40.5KiB
2017-03-15 20:10:08,765 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 11 Chunks of size 16384 totalling 176.0KiB
2017-03-15 20:10:08,765 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 16640 totalling 16.2KiB
2017-03-15 20:10:08,765 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 27136 totalling 26.5KiB
2017-03-15 20:10:08,765 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 5 Chunks of size 32768 totalling 160.0KiB
2017-03-15 20:10:08,766 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 6 Chunks of size 147456 totalling 864.0KiB
2017-03-15 20:10:08,767 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 6 Chunks of size 294912 totalling 1.69MiB
2017-03-15 20:10:08,767 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 6 Chunks of size 589824 totalling 3.38MiB
2017-03-15 20:10:08,767 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 6 Chunks of size 1179648 totalling 6.75MiB
2017-03-15 20:10:08,768 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 10 Chunks of size 2359296 totalling 22.50MiB
2017-03-15 20:10:08,768 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 3538944 totalling 3.38MiB
2017-03-15 20:10:08,769 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 5 Chunks of size 4718592 totalling 22.50MiB
2017-03-15 20:10:08,769 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 5505024 totalling 5.25MiB
2017-03-15 20:10:08,770 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 6422528 totalling 12.25MiB
2017-03-15 20:10:08,770 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 25 Chunks of size 9437184 totalling 225.00MiB
2017-03-15 20:10:08,770 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 14155776 totalling 13.50MiB
2017-03-15 20:10:08,771 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 4 Chunks of size 25690112 totalling 98.00MiB
2017-03-15 20:10:08,771 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 3 Chunks of size 33554432 totalling 96.00MiB
2017-03-15 20:10:08,772 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 39226368 totalling 37.41MiB
2017-03-15 20:10:08,772 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 58982400 totalling 56.25MiB
2017-03-15 20:10:08,772 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 5 Chunks of size 67108864 totalling 320.00MiB
2017-03-15 20:10:08,773 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 89325568 totalling 85.19MiB
2017-03-15 20:10:08,773 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 90731520 totalling 86.53MiB
2017-03-15 20:10:08,773 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 4 Chunks of size 102760448 totalling 392.00MiB
2017-03-15 20:10:08,774 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 110231552 totalling 105.12MiB
2017-03-15 20:10:08,774 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 117964800 totalling 225.00MiB
2017-03-15 20:10:08,774 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 4 Chunks of size 205520896 totalling 784.00MiB
2017-03-15 20:10:08,775 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 212926464 totalling 203.06MiB
2017-03-15 20:10:08,775 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 220463104 totalling 420.50MiB
2017-03-15 20:10:08,775 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 8 Chunks of size 411041792 totalling 3.06GiB
2017-03-15 20:10:08,775 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 429775616 totalling 409.87MiB
2017-03-15 20:10:08,776 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 788662016 totalling 752.13MiB
2017-03-15 20:10:08,777 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 822083584 totalling 1.53GiB
2017-03-15 20:10:08,777 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 829423616 totalling 791.00MiB
2017-03-15 20:10:08,777 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 836829184 totalling 798.06MiB
2017-03-15 20:10:08,778 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:700] Sum Total of in-use chunks: 10.43GiB
2017-03-15 20:10:08,778 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:702] Stats:
2017-03-15 20:10:08,778 INFO - Limit: 11332668621
2017-03-15 20:10:08,778 INFO - InUse: 11200530176
2017-03-15 20:10:08,779 INFO - MaxInUse: 11332660736
2017-03-15 20:10:08,780 INFO - NumAllocs: 1539
2017-03-15 20:10:08,780 INFO - MaxAllocSize: 3384592384
2017-03-15 20:10:08,780 INFO -
2017-03-15 20:10:08,781 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:274] ********************************************************xx***************************************xxx
2017-03-15 20:10:08,781 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:275] Ran out of memory trying to allocate 392.00MiB. See logs for memory state.
2017-03-15 20:10:08,782 INFO - W tensorflow/core/framework/op_kernel.cc:975] Resource exhausted: OOM when allocating tensor with shape[25088,4096]
2017-03-15 20:10:09,558 INFO - Using TensorFlow backend.
2017-03-15 20:10:09,629 INFO - Traceback (most recent call last):
2017-03-15 20:10:09,629 INFO - File "/usr/local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1021, in _do_call
2017-03-15 20:10:09,630 INFO - return fn(*args)
2017-03-15 20:10:09,630 INFO - File "/usr/local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1003, in _run_fn
2017-03-15 20:10:09,631 INFO - status, run_metadata)
2017-03-15 20:10:09,631 INFO - File "/usr/local/lib/python3.5/contextlib.py", line 66, in __exit__
2017-03-15 20:10:09,632 INFO - next(self.gen)
2017-03-15 20:10:09,632 INFO - File "/usr/local/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 469, in raise_exception_on_not_ok_status
2017-03-15 20:10:09,633 INFO - pywrap_tensorflow.TF_GetCode(status))
2017-03-15 20:10:09,633 INFO - tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[25088,4096]
2017-03-15 20:10:09,634 INFO - [[Node: Square_26 = Square[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](gradients/MatMul_grad/MatMul_1)]]
2017-03-15 20:10:09,634 INFO -
2017-03-15 20:10:09,635 INFO - During handling of the above exception, another exception occurred:
2017-03-15 20:10:09,635 INFO -
2017-03-15 20:10:09,636 INFO - Traceback (most recent call last):
2017-03-15 20:10:09,637 INFO - File "demo1.py", line 129, in <module>
2017-03-15 20:10:09,638 INFO - nb_epoch=20, validation_data=valid_batches, nb_val_samples=valid_batches.nb_sample)
2017-03-15 20:10:09,638 INFO - File "/usr/local/lib/python3.5/site-packages/keras/models.py", line 935, in fit_generator
2017-03-15 20:10:09,639 INFO - initial_epoch=initial_epoch)
2017-03-15 20:10:09,640 INFO - File "/usr/local/lib/python3.5/site-packages/keras/engine/training.py", line 1557, in fit_generator
2017-03-15 20:10:09,640 INFO - class_weight=class_weight)
2017-03-15 20:10:09,641 INFO - File "/usr/local/lib/python3.5/site-packages/keras/engine/training.py", line 1320, in train_on_batch
2017-03-15 20:10:09,641 INFO - outputs = self.train_function(ins)
2017-03-15 20:10:09,642 INFO - File "/usr/local/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 1943, in __call__
2017-03-15 20:10:09,642 INFO - feed_dict=feed_dict)
2017-03-15 20:10:09,643 INFO - File "/usr/local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 766, in run
2017-03-15 20:10:09,644 INFO - run_metadata_ptr)
2017-03-15 20:10:09,644 INFO - File "/usr/local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 964, in _run
2017-03-15 20:10:09,645 INFO - feed_dict_string, options, run_metadata)
2017-03-15 20:10:09,645 INFO - File "/usr/local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1014, in _do_run
2017-03-15 20:10:09,646 INFO - target_list, options, run_metadata)
2017-03-15 20:10:09,646 INFO - File "/usr/local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1034, in _do_call
2017-03-15 20:10:09,647 INFO - raise type(e)(node_def, op, message)
2017-03-15 20:10:09,648 INFO - tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[25088,4096]
2017-03-15 20:10:09,648 INFO - [[Node: Square_26 = Square[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](gradients/MatMul_grad/MatMul_1)]]
2017-03-15 20:10:09,649 INFO -
2017-03-15 20:10:09,649 INFO - Caused by op 'Square_26', defined at:
2017-03-15 20:10:09,650 INFO - File "demo1.py", line 129, in <module>
2017-03-15 20:10:09,651 INFO - nb_epoch=20, validation_data=valid_batches, nb_val_samples=valid_batches.nb_sample)
2017-03-15 20:10:09,651 INFO - File "/usr/local/lib/python3.5/site-packages/keras/models.py", line 935, in fit_generator
2017-03-15 20:10:09,652 INFO - initial_epoch=initial_epoch)
2017-03-15 20:10:09,652 INFO - File "/usr/local/lib/python3.5/site-packages/keras/engine/training.py", line 1454, in fit_generator
2017-03-15 20:10:09,653 INFO - self._make_train_function()
2017-03-15 20:10:09,654 INFO - File "/usr/local/lib/python3.5/site-packages/keras/engine/training.py", line 760, in _make_train_function
2017-03-15 20:10:09,654 INFO - self.total_loss)
2017-03-15 20:10:09,654 INFO - File "/usr/local/lib/python3.5/site-packages/keras/optimizers.py", line 434, in get_updates
2017-03-15 20:10:09,655 INFO - v_t = (self.beta_2 * v) + (1. - self.beta_2) * K.square(g)
2017-03-15 20:10:09,655 INFO - File "/usr/local/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 1186, in square
2017-03-15 20:10:09,655 INFO - return tf.square(x)
2017-03-15 20:10:09,656 INFO - File "/usr/local/lib/python3.5/site-packages/tensorflow/python/ops/math_ops.py", line 385, in square
2017-03-15 20:10:09,656 INFO - return gen_math_ops.square(x, name=name)
2017-03-15 20:10:09,656 INFO - File "/usr/local/lib/python3.5/site-packages/tensorflow/python/ops/gen_math_ops.py", line 2716, in square
2017-03-15 20:10:09,656 INFO - result = _op_def_lib.apply_op("Square", x=x, name=name)
2017-03-15 20:10:09,657 INFO - File "/usr/local/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op
2017-03-15 20:10:09,657 INFO - op_def=op_def)
2017-03-15 20:10:09,657 INFO - File "/usr/local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2240, in create_op
2017-03-15 20:10:09,658 INFO - original_op=self._default_original_op, op_def=op_def)
2017-03-15 20:10:09,658 INFO - File "/usr/local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1128, in __init__
2017-03-15 20:10:09,659 INFO - self._traceback = _extract_stack()
2017-03-15 20:10:09,659 INFO -
2017-03-15 20:10:09,660 INFO - ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[25088,4096]
2017-03-15 20:10:09,660 INFO - [[Node: Square_26 = Square[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](gradients/MatMul_grad/MatMul_1)]]
2017-03-15 20:10:09,660 INFO -
2017-03-15 20:10:09,691 INFO - E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:652] Deallocating stream with pending work
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment