Skip to content

Instantly share code, notes, and snippets.

@munyari
Created March 16, 2017 14:06
Show Gist options
  • Save munyari/6ba14f76cacc3d9e60f7a91610c25933 to your computer and use it in GitHub Desktop.
Save munyari/6ba14f76cacc3d9e60f7a91610c25933 to your computer and use it in GitHub Desktop.
2017-03-15 20:05:28,431 INFO - Preparing to run TaskInstance <TaskInstance: munyari/cats-dogs:50 (id: y4pY4Z4HiaTXMquPe7ddb7) (checksum: 0f0e7ccc7139aa4e5c183db40be43d46) (last update: 2017-03-15 20:05:28.396849) [queued]>
2017-03-15 20:05:28,451 INFO - Starting attempt 1 at 2017-03-15 20:05:28.438673
2017-03-15 20:05:28,823 INFO - Executing command in container: stdbuf -o0 sh command.sh
2017-03-15 20:05:28,824 INFO - Pulling Docker image: floydhub/tensorflow:latest-gpu-py2
2017-03-15 20:05:31,676 INFO - Starting container...
2017-03-15 20:05:32,861 INFO -
################################################################################
2017-03-15 20:05:32,861 INFO - Run Output:
2017-03-15 20:18:19,192 INFO - Using TensorFlow backend.
2017-03-15 20:18:29,981 INFO - I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so locally
2017-03-15 20:18:29,985 INFO - I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.so locally
2017-03-15 20:18:29,986 INFO - I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so locally
2017-03-15 20:18:29,990 INFO - I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally
2017-03-15 20:18:29,990 INFO - I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so locally
2017-03-15 20:19:07,857 INFO - Found 23000 images belonging to 2 classes.
2017-03-15 20:19:08,338 INFO - Found 2000 images belonging to 2 classes.
2017-03-15 20:19:09,226 INFO - Epoch 1/20
2017-03-15 20:19:12,227 INFO - I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2017-03-15 20:19:12,227 INFO - I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties:
2017-03-15 20:19:12,228 INFO - name: Tesla K80
2017-03-15 20:19:12,228 INFO - major: 3 minor: 7 memoryClockRate (GHz) 0.8235
2017-03-15 20:19:12,229 INFO - pciBusID 0000:00:1e.0
2017-03-15 20:19:12,229 INFO - Total memory: 11.17GiB
2017-03-15 20:19:12,230 INFO - Free memory: 11.11GiB
2017-03-15 20:19:12,230 INFO - I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0
2017-03-15 20:19:12,231 INFO - I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0: Y
2017-03-15 20:19:12,231 INFO - I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0)
2017-03-15 20:19:54,308 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 3.45GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-03-15 20:19:54,311 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 3.05GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-03-15 20:19:55,548 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 2.58GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-03-15 20:19:56,836 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 2.40GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-03-15 20:19:58,217 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 2.40GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-03-15 20:19:58,218 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 1.34GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-03-15 20:19:58,893 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 2.40GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-03-15 20:19:59,548 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 2.22GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-03-15 20:20:00,975 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 2.58GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-03-15 20:20:03,281 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 3.05GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available.
2017-03-15 20:20:21,990 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (256): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:20:21,994 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (512): Total Chunks: 2, Chunks in use: 0 1.5KiB allocated for chunks. 520B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:20:21,994 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1024): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:20:21,995 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2048): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:20:21,995 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4096): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:20:21,995 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8192): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:20:21,996 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16384): Total Chunks: 1, Chunks in use: 0 16.0KiB allocated for chunks. 16.0KiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:20:21,996 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (32768): Total Chunks: 3, Chunks in use: 0 128.0KiB allocated for chunks. 64.06MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:20:21,997 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (65536): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:20:21,997 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (131072): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:20:21,998 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (262144): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:20:21,998 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (524288): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:20:21,999 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1048576): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:20:21,999 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2097152): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:20:21,999 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4194304): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:20:22,000 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8388608): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:20:22,000 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16777216): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:20:22,000 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (33554432): Total Chunks: 1, Chunks in use: 0 57.88MiB allocated for chunks. 8B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:20:22,001 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (67108864): Total Chunks: 1, Chunks in use: 0 68.00MiB allocated for chunks. 1.00MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:20:22,001 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (134217728): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:20:22,002 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (268435456): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
2017-03-15 20:20:22,003 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:660] Bin for 392.00MiB was 256.00MiB, Chunk State:
2017-03-15 20:20:22,003 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0000 of size 1280
2017-03-15 20:20:22,004 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0500 of size 256
2017-03-15 20:20:22,004 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0600 of size 256
2017-03-15 20:20:22,004 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0700 of size 256
2017-03-15 20:20:22,005 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0800 of size 256
2017-03-15 20:20:22,005 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0900 of size 512
2017-03-15 20:20:22,005 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0b00 of size 256
2017-03-15 20:20:22,006 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0c00 of size 256
2017-03-15 20:20:22,006 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0d00 of size 1024
2017-03-15 20:20:22,007 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e1100 of size 256
2017-03-15 20:20:22,007 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e1200 of size 256
2017-03-15 20:20:22,007 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e1300 of size 2048
2017-03-15 20:20:22,008 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e1b00 of size 256
2017-03-15 20:20:22,008 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e1c00 of size 256
2017-03-15 20:20:22,009 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e1d00 of size 16384
2017-03-15 20:20:22,009 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e5d00 of size 256
2017-03-15 20:20:22,010 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e5e00 of size 256
2017-03-15 20:20:22,010 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e5f00 of size 256
2017-03-15 20:20:22,010 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e6000 of size 256
2017-03-15 20:20:22,011 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e6100 of size 256
2017-03-15 20:20:22,011 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e6200 of size 256
2017-03-15 20:20:22,012 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e6300 of size 256
2017-03-15 20:20:22,012 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e6400 of size 9437184
2017-03-15 20:20:22,013 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12059e6400 of size 6912
2017-03-15 20:20:22,013 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12059e7f00 of size 147456
2017-03-15 20:20:22,013 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205a0bf00 of size 294912
2017-03-15 20:20:22,014 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205a53f00 of size 589824
2017-03-15 20:20:22,014 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205ae3f00 of size 1179648
2017-03-15 20:20:22,015 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205c03f00 of size 2359296
2017-03-15 20:20:22,015 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205e43f00 of size 4718592
2017-03-15 20:20:22,016 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12062c3f00 of size 411041792
2017-03-15 20:20:22,016 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x121eac3f00 of size 67108864
2017-03-15 20:20:22,016 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ac3f00 of size 32768
2017-03-15 20:20:22,017 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acbf00 of size 256
2017-03-15 20:20:22,017 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc000 of size 256
2017-03-15 20:20:22,018 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc100 of size 256
2017-03-15 20:20:22,018 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc200 of size 256
2017-03-15 20:20:22,019 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc300 of size 256
2017-03-15 20:20:22,019 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc400 of size 256
2017-03-15 20:20:22,020 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc500 of size 256
2017-03-15 20:20:22,020 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc600 of size 256
2017-03-15 20:20:22,021 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc700 of size 256
2017-03-15 20:20:22,021 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc800 of size 256
2017-03-15 20:20:22,024 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc900 of size 256
2017-03-15 20:20:22,025 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acca00 of size 256
2017-03-15 20:20:22,025 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222accb00 of size 256
2017-03-15 20:20:22,026 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222accc00 of size 256
2017-03-15 20:20:22,026 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222accd00 of size 256
2017-03-15 20:20:22,027 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acce00 of size 256
2017-03-15 20:20:22,028 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222accf00 of size 256
2017-03-15 20:20:22,029 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acd000 of size 256
2017-03-15 20:20:22,030 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acd100 of size 256
2017-03-15 20:20:22,030 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acd200 of size 256
2017-03-15 20:20:22,032 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acd300 of size 256
2017-03-15 20:20:22,033 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acd700 of size 256
2017-03-15 20:20:22,033 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acd800 of size 256
2017-03-15 20:20:22,034 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acdc00 of size 256
2017-03-15 20:20:22,034 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acdd00 of size 2048
2017-03-15 20:20:22,035 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ace500 of size 256
2017-03-15 20:20:22,035 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ace600 of size 6912
2017-03-15 20:20:22,036 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ad0100 of size 32768
2017-03-15 20:20:22,036 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ad8100 of size 16384
2017-03-15 20:20:22,037 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ae8100 of size 6912
2017-03-15 20:20:22,037 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ae9c00 of size 1024
2017-03-15 20:20:22,038 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222aea000 of size 1024
2017-03-15 20:20:22,038 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222aea400 of size 2048
2017-03-15 20:20:22,039 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222aeac00 of size 2048
2017-03-15 20:20:22,039 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222aeb400 of size 29184
2017-03-15 20:20:22,040 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222af2600 of size 256
2017-03-15 20:20:22,040 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222af2700 of size 16384
2017-03-15 20:20:22,041 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222afa700 of size 16384
2017-03-15 20:20:22,041 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222afe700 of size 16384
2017-03-15 20:20:22,042 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222b02700 of size 32768
2017-03-15 20:20:22,042 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222b16700 of size 147456
2017-03-15 20:20:22,043 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222b3a700 of size 512
2017-03-15 20:20:22,043 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222b3a900 of size 147456
2017-03-15 20:20:22,043 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222b5e900 of size 147456
2017-03-15 20:20:22,044 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222b82900 of size 294912
2017-03-15 20:20:22,044 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222bca900 of size 512
2017-03-15 20:20:22,045 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222bcab00 of size 294912
2017-03-15 20:20:22,045 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222c12b00 of size 294912
2017-03-15 20:20:22,046 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222c5ab00 of size 589824
2017-03-15 20:20:22,047 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ceab00 of size 1024
2017-03-15 20:20:22,047 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ceaf00 of size 3538944
2017-03-15 20:20:22,048 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x122304af00 of size 1179648
2017-03-15 20:20:22,048 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x122316af00 of size 1024
2017-03-15 20:20:22,049 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x122316b300 of size 1024
2017-03-15 20:20:22,049 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x122316b700 of size 2359296
2017-03-15 20:20:22,050 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12233ab700 of size 589824
2017-03-15 20:20:22,050 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x122343b700 of size 1769472
2017-03-15 20:20:22,051 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12235eb700 of size 2048
2017-03-15 20:20:22,051 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12235ebf00 of size 9437184
2017-03-15 20:20:22,052 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1223eebf00 of size 9437184
2017-03-15 20:20:22,052 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12247ebf00 of size 9437184
2017-03-15 20:20:22,053 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12250ebf00 of size 14155776
2017-03-15 20:20:22,053 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1225e6bf00 of size 4718592
2017-03-15 20:20:22,054 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12262ebf00 of size 2048
2017-03-15 20:20:22,054 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12262ec700 of size 2048
2017-03-15 20:20:22,055 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12262ecf00 of size 2048
2017-03-15 20:20:22,055 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12262ed700 of size 2048
2017-03-15 20:20:22,056 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12262edf00 of size 2048
2017-03-15 20:20:22,056 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12262ee700 of size 9437184
2017-03-15 20:20:22,057 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1226bee700 of size 9437184
2017-03-15 20:20:22,057 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12274ee700 of size 9437184
2017-03-15 20:20:22,057 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1227dee700 of size 9437184
2017-03-15 20:20:22,058 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12286ee700 of size 9437184
2017-03-15 20:20:22,058 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1228fee700 of size 2359296
2017-03-15 20:20:22,059 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x122922e700 of size 2359296
2017-03-15 20:20:22,059 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x122946e700 of size 5505024
2017-03-15 20:20:22,059 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12299ae700 of size 39226368
2017-03-15 20:20:22,060 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x122bf17300 of size 89325568
2017-03-15 20:20:22,060 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1231447300 of size 9437184
2017-03-15 20:20:22,060 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1231d47300 of size 6422528
2017-03-15 20:20:22,062 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1235d47300 of size 9437184
2017-03-15 20:20:22,062 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1236647300 of size 9437184
2017-03-15 20:20:22,063 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1236f47300 of size 9437184
2017-03-15 20:20:22,063 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1237847300 of size 9437184
2017-03-15 20:20:22,063 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1238147300 of size 589824
2017-03-15 20:20:22,064 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12381d7300 of size 1179648
2017-03-15 20:20:22,064 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12382f7300 of size 2359296
2017-03-15 20:20:22,065 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1238537300 of size 2359296
2017-03-15 20:20:22,065 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1238777300 of size 4718592
2017-03-15 20:20:22,065 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1238bf7300 of size 9437184
2017-03-15 20:20:22,066 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12394f7300 of size 90141696
2017-03-15 20:20:22,066 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x123eaee700 of size 16384
2017-03-15 20:20:22,068 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x123eaf2700 of size 67108864
2017-03-15 20:20:22,068 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242af2700 of size 16384
2017-03-15 20:20:22,069 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242af6700 of size 2048
2017-03-15 20:20:22,069 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242af6f00 of size 2048
2017-03-15 20:20:22,070 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242af7700 of size 2048
2017-03-15 20:20:22,070 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242af7f00 of size 6912
2017-03-15 20:20:22,071 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242af9a00 of size 256
2017-03-15 20:20:22,071 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242af9b00 of size 512
2017-03-15 20:20:22,072 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242af9d00 of size 512
2017-03-15 20:20:22,072 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242af9f00 of size 1024
2017-03-15 20:20:22,073 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afa300 of size 1024
2017-03-15 20:20:22,073 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afa700 of size 1024
2017-03-15 20:20:22,074 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afab00 of size 2048
2017-03-15 20:20:22,075 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afb300 of size 2048
2017-03-15 20:20:22,075 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afbb00 of size 256
2017-03-15 20:20:22,076 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afbc00 of size 2048
2017-03-15 20:20:22,076 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afc400 of size 2048
2017-03-15 20:20:22,077 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afcc00 of size 2048
2017-03-15 20:20:22,078 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afd400 of size 2048
2017-03-15 20:20:22,078 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afdc00 of size 256
2017-03-15 20:20:22,079 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afdd00 of size 256
2017-03-15 20:20:22,079 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afde00 of size 512
2017-03-15 20:20:22,080 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afe000 of size 512
2017-03-15 20:20:22,081 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afe200 of size 1280
2017-03-15 20:20:22,081 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afe700 of size 256
2017-03-15 20:20:22,082 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afe800 of size 256
2017-03-15 20:20:22,082 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afe900 of size 256
2017-03-15 20:20:22,083 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afea00 of size 256
2017-03-15 20:20:22,083 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afeb00 of size 256
2017-03-15 20:20:22,083 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afec00 of size 256
2017-03-15 20:20:22,084 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afed00 of size 9437184
2017-03-15 20:20:22,084 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12433fed00 of size 2048
2017-03-15 20:20:22,084 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12433ff500 of size 9437184
2017-03-15 20:20:22,085 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1243cff500 of size 2048
2017-03-15 20:20:22,085 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1243cffd00 of size 9437184
2017-03-15 20:20:22,086 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12445ffd00 of size 2048
2017-03-15 20:20:22,086 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244600500 of size 9437184
2017-03-15 20:20:22,087 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244f00500 of size 2048
2017-03-15 20:20:22,088 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244f00d00 of size 6912
2017-03-15 20:20:22,088 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244f02800 of size 256
2017-03-15 20:20:22,088 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244f02900 of size 147456
2017-03-15 20:20:22,089 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244f26900 of size 256
2017-03-15 20:20:22,089 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244f26a00 of size 294912
2017-03-15 20:20:22,089 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244f6ea00 of size 512
2017-03-15 20:20:22,090 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244f6ec00 of size 589824
2017-03-15 20:20:22,090 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244ffec00 of size 512
2017-03-15 20:20:22,090 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244ffee00 of size 1179648
2017-03-15 20:20:22,091 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x124511ee00 of size 1024
2017-03-15 20:20:22,091 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x124511f200 of size 2359296
2017-03-15 20:20:22,091 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x124535f200 of size 1024
2017-03-15 20:20:22,093 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x124535f600 of size 2359296
2017-03-15 20:20:22,093 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x124559f600 of size 1024
2017-03-15 20:20:22,093 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x124559fa00 of size 4718592
2017-03-15 20:20:22,094 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1245a1fa00 of size 2048
2017-03-15 20:20:22,094 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1245a20200 of size 9437184
2017-03-15 20:20:22,094 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1246320200 of size 2048
2017-03-15 20:20:22,095 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1246320a00 of size 411041792
2017-03-15 20:20:22,095 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x125eb20a00 of size 16384
2017-03-15 20:20:22,096 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x125eb24a00 of size 67108864
2017-03-15 20:20:22,096 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1262b24a00 of size 16384
2017-03-15 20:20:22,096 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1262b28a00 of size 32768
2017-03-15 20:20:22,097 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1262b30a00 of size 256
2017-03-15 20:20:22,098 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1262b30b00 of size 9437184
2017-03-15 20:20:22,099 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1263430b00 of size 2048
2017-03-15 20:20:22,105 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1263431300 of size 9437184
2017-03-15 20:20:22,106 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1263d31300 of size 2048
2017-03-15 20:20:22,107 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1263d31b00 of size 9437184
2017-03-15 20:20:22,107 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264631b00 of size 2048
2017-03-15 20:20:22,108 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264632300 of size 9437184
2017-03-15 20:20:22,108 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264f32300 of size 2048
2017-03-15 20:20:22,109 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264f32b00 of size 6912
2017-03-15 20:20:22,109 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264f34600 of size 256
2017-03-15 20:20:22,109 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264f34700 of size 147456
2017-03-15 20:20:22,110 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264f58700 of size 256
2017-03-15 20:20:22,111 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264f58800 of size 294912
2017-03-15 20:20:22,111 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264fa0800 of size 512
2017-03-15 20:20:22,111 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264fa0a00 of size 589824
2017-03-15 20:20:22,112 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1265030a00 of size 512
2017-03-15 20:20:22,112 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1265030c00 of size 1179648
2017-03-15 20:20:22,112 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1265150c00 of size 1024
2017-03-15 20:20:22,113 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1265151000 of size 2359296
2017-03-15 20:20:22,113 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1265391000 of size 1024
2017-03-15 20:20:22,117 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1265391400 of size 2359296
2017-03-15 20:20:22,121 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12655d1400 of size 1024
2017-03-15 20:20:22,121 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12655d1800 of size 4718592
2017-03-15 20:20:22,122 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1265a51800 of size 2048
2017-03-15 20:20:22,122 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1265a52000 of size 9437184
2017-03-15 20:20:22,123 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1266352000 of size 2048
2017-03-15 20:20:22,123 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1266352800 of size 411041792
2017-03-15 20:20:22,125 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x127eb52800 of size 16384
2017-03-15 20:20:22,127 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x127eb56800 of size 67108864
2017-03-15 20:20:22,127 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1282b56800 of size 16384
2017-03-15 20:20:22,128 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1282b5a800 of size 32768
2017-03-15 20:20:22,128 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1282b62800 of size 256
2017-03-15 20:20:22,128 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1282b62900 of size 429775616
2017-03-15 20:20:22,131 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x129c540400 of size 822083584
2017-03-15 20:20:22,131 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12cd540400 of size 836829184
2017-03-15 20:20:22,132 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12ff350400 of size 822083584
2017-03-15 20:20:22,132 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1330350400 of size 205520896
2017-03-15 20:20:22,133 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x133c750400 of size 212926464
2017-03-15 20:20:22,133 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1349260400 of size 411041792
2017-03-15 20:20:22,134 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1361a60400 of size 829423616
2017-03-15 20:20:22,135 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1393160400 of size 411041792
2017-03-15 20:20:22,135 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13ab960400 of size 411041792
2017-03-15 20:20:22,137 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13c4160400 of size 16384
2017-03-15 20:20:22,137 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13c416c400 of size 256
2017-03-15 20:20:22,138 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13c416c500 of size 411041792
2017-03-15 20:20:22,138 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13dc96c500 of size 102760448
2017-03-15 20:20:22,139 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13e2b6c500 of size 110231552
2017-03-15 20:20:22,139 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13e948c500 of size 205520896
2017-03-15 20:20:22,140 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13f588c500 of size 220463104
2017-03-15 20:20:22,140 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1402acc500 of size 205520896
2017-03-15 20:20:22,140 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x140eecc500 of size 220463104
2017-03-15 20:20:22,141 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x141c10c500 of size 205520896
2017-03-15 20:20:22,141 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x142850c500 of size 58982400
2017-03-15 20:20:22,142 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x142bd4c500 of size 102760448
2017-03-15 20:20:22,143 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1431f4c500 of size 117964800
2017-03-15 20:20:22,144 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1438fcc500 of size 102760448
2017-03-15 20:20:22,144 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x143f1cc500 of size 117964800
2017-03-15 20:20:22,145 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x144624c500 of size 102760448
2017-03-15 20:20:22,145 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x144c44c500 of size 25690112
2017-03-15 20:20:22,146 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x144dccc500 of size 33554432
2017-03-15 20:20:22,146 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x144fccc500 of size 25690112
2017-03-15 20:20:22,147 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x145154c500 of size 33554432
2017-03-15 20:20:22,147 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x145354c500 of size 25690112
2017-03-15 20:20:22,148 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1454dcc500 of size 33554432
2017-03-15 20:20:22,148 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1456dcc500 of size 25690112
2017-03-15 20:20:22,150 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x145864c500 of size 6422528
2017-03-15 20:20:22,150 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x145d06c500 of size 67108864
2017-03-15 20:20:22,150 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x146106c500 of size 411041792
2017-03-15 20:20:22,151 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x147986c500 of size 788662016
2017-03-15 20:20:22,151 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x1222acd400 of size 768
2017-03-15 20:20:22,152 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x1222acd900 of size 768
2017-03-15 20:20:22,152 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x1222adc100 of size 49152
2017-03-15 20:20:22,153 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x1222af6700 of size 16384
2017-03-15 20:20:22,153 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x1222b0a700 of size 49152
2017-03-15 20:20:22,154 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x1232367300 of size 60686336
2017-03-15 20:20:22,155 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x13c4164400 of size 32768
2017-03-15 20:20:22,156 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x1458c6c500 of size 71303168
2017-03-15 20:20:22,156 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:693] Summary of in-use Chunks by size:
2017-03-15 20:20:22,157 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 60 Chunks of size 256 totalling 15.0KiB
2017-03-15 20:20:22,157 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 11 Chunks of size 512 totalling 5.5KiB
2017-03-15 20:20:22,158 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 15 Chunks of size 1024 totalling 15.0KiB
2017-03-15 20:20:22,158 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 1280 totalling 2.5KiB
2017-03-15 20:20:22,159 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 31 Chunks of size 2048 totalling 62.0KiB
2017-03-15 20:20:22,159 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 6 Chunks of size 6912 totalling 40.5KiB
2017-03-15 20:20:22,160 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 12 Chunks of size 16384 totalling 192.0KiB
2017-03-15 20:20:22,160 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 29184 totalling 28.5KiB
2017-03-15 20:20:22,162 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 5 Chunks of size 32768 totalling 160.0KiB
2017-03-15 20:20:22,162 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 6 Chunks of size 147456 totalling 864.0KiB
2017-03-15 20:20:22,163 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 6 Chunks of size 294912 totalling 1.69MiB
2017-03-15 20:20:22,163 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 6 Chunks of size 589824 totalling 3.38MiB
2017-03-15 20:20:22,164 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 5 Chunks of size 1179648 totalling 5.62MiB
2017-03-15 20:20:22,164 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 1769472 totalling 1.69MiB
2017-03-15 20:20:22,165 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 10 Chunks of size 2359296 totalling 22.50MiB
2017-03-15 20:20:22,165 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 3538944 totalling 3.38MiB
2017-03-15 20:20:22,166 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 5 Chunks of size 4718592 totalling 22.50MiB
2017-03-15 20:20:22,166 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 5505024 totalling 5.25MiB
2017-03-15 20:20:22,167 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 6422528 totalling 12.25MiB
2017-03-15 20:20:22,168 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 25 Chunks of size 9437184 totalling 225.00MiB
2017-03-15 20:20:22,169 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 14155776 totalling 13.50MiB
2017-03-15 20:20:22,170 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 4 Chunks of size 25690112 totalling 98.00MiB
2017-03-15 20:20:22,170 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 3 Chunks of size 33554432 totalling 96.00MiB
2017-03-15 20:20:22,171 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 39226368 totalling 37.41MiB
2017-03-15 20:20:22,172 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 58982400 totalling 56.25MiB
2017-03-15 20:20:22,172 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 5 Chunks of size 67108864 totalling 320.00MiB
2017-03-15 20:20:22,173 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 89325568 totalling 85.19MiB
2017-03-15 20:20:22,174 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 90141696 totalling 85.97MiB
2017-03-15 20:20:22,174 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 4 Chunks of size 102760448 totalling 392.00MiB
2017-03-15 20:20:22,175 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 110231552 totalling 105.12MiB
2017-03-15 20:20:22,176 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 117964800 totalling 225.00MiB
2017-03-15 20:20:22,177 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 4 Chunks of size 205520896 totalling 784.00MiB
2017-03-15 20:20:22,177 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 212926464 totalling 203.06MiB
2017-03-15 20:20:22,178 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 220463104 totalling 420.50MiB
2017-03-15 20:20:22,179 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 8 Chunks of size 411041792 totalling 3.06GiB
2017-03-15 20:20:22,179 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 429775616 totalling 409.87MiB
2017-03-15 20:20:22,179 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 788662016 totalling 752.13MiB
2017-03-15 20:20:22,180 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 822083584 totalling 1.53GiB
2017-03-15 20:20:22,180 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 829423616 totalling 791.00MiB
2017-03-15 20:20:22,181 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 836829184 totalling 798.06MiB
2017-03-15 20:20:22,182 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:700] Sum Total of in-use chunks: 10.43GiB
2017-03-15 20:20:22,182 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:702] Stats:
2017-03-15 20:20:22,183 INFO - Limit: 11332668621
2017-03-15 20:20:22,183 INFO - InUse: 11200529920
2017-03-15 20:20:22,184 INFO - MaxInUse: 11332668160
2017-03-15 20:20:22,184 INFO - NumAllocs: 2445
2017-03-15 20:20:22,185 INFO - MaxAllocSize: 3087794176
2017-03-15 20:20:22,187 INFO -
2017-03-15 20:20:22,187 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:274] ********************************************************xx***************************************xxx
2017-03-15 20:20:22,188 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:275] Ran out of memory trying to allocate 392.00MiB. See logs for memory state.
2017-03-15 20:20:22,189 INFO - W tensorflow/core/framework/op_kernel.cc:975] Resource exhausted: OOM when allocating tensor with shape[25088,4096]
2017-03-15 20:20:23,489 INFO - Traceback (most recent call last):
2017-03-15 20:20:23,489 INFO - File "demo1.py", line 129, in <module>
2017-03-15 20:20:23,501 INFO - nb_epoch=20, validation_data=valid_batches, nb_val_samples=valid_batches.nb_sample)
2017-03-15 20:20:23,502 INFO - File "/usr/local/lib/python2.7/site-packages/keras/models.py", line 935, in fit_generator
2017-03-15 20:20:23,507 INFO - initial_epoch=initial_epoch)
2017-03-15 20:20:23,507 INFO - File "/usr/local/lib/python2.7/site-packages/keras/engine/training.py", line 1557, in fit_generator
2017-03-15 20:20:23,528 INFO - class_weight=class_weight)
2017-03-15 20:20:23,529 INFO - File "/usr/local/lib/python2.7/site-packages/keras/engine/training.py", line 1320, in train_on_batch
2017-03-15 20:20:23,529 INFO - outputs = self.train_function(ins)
2017-03-15 20:20:23,530 INFO - File "/usr/local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 1943, in __call__
2017-03-15 20:20:23,530 INFO - feed_dict=feed_dict)
2017-03-15 20:20:23,530 INFO - File "/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 766, in run
2017-03-15 20:20:23,534 INFO - run_metadata_ptr)
2017-03-15 20:20:23,534 INFO - File "/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 964, in _run
2017-03-15 20:20:23,534 INFO - feed_dict_string, options, run_metadata)
2017-03-15 20:20:23,536 INFO - File "/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1014, in _do_run
2017-03-15 20:20:23,537 INFO - target_list, options, run_metadata)
2017-03-15 20:20:23,537 INFO - File "/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1034, in _do_call
2017-03-15 20:20:23,537 INFO - raise type(e)(node_def, op, message)
2017-03-15 20:20:23,560 INFO - tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[25088,4096]
2017-03-15 20:20:23,560 INFO - [[Node: Square_26 = Square[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](gradients/MatMul_grad/MatMul_1)]]
2017-03-15 20:20:23,561 INFO -
2017-03-15 20:20:23,561 INFO - Caused by op u'Square_26', defined at:
2017-03-15 20:20:23,562 INFO - File "demo1.py", line 129, in <module>
2017-03-15 20:20:23,562 INFO - nb_epoch=20, validation_data=valid_batches, nb_val_samples=valid_batches.nb_sample)
2017-03-15 20:20:23,563 INFO - File "/usr/local/lib/python2.7/site-packages/keras/models.py", line 935, in fit_generator
2017-03-15 20:20:23,563 INFO - initial_epoch=initial_epoch)
2017-03-15 20:20:23,564 INFO - File "/usr/local/lib/python2.7/site-packages/keras/engine/training.py", line 1454, in fit_generator
2017-03-15 20:20:23,564 INFO - self._make_train_function()
2017-03-15 20:20:23,565 INFO - File "/usr/local/lib/python2.7/site-packages/keras/engine/training.py", line 760, in _make_train_function
2017-03-15 20:20:23,565 INFO - self.total_loss)
2017-03-15 20:20:23,566 INFO - File "/usr/local/lib/python2.7/site-packages/keras/optimizers.py", line 434, in get_updates
2017-03-15 20:20:23,566 INFO - v_t = (self.beta_2 * v) + (1. - self.beta_2) * K.square(g)
2017-03-15 20:20:23,567 INFO - File "/usr/local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 1186, in square
2017-03-15 20:20:23,567 INFO - return tf.square(x)
2017-03-15 20:20:23,568 INFO - File "/usr/local/lib/python2.7/site-packages/tensorflow/python/ops/math_ops.py", line 385, in square
2017-03-15 20:20:23,568 INFO - return gen_math_ops.square(x, name=name)
2017-03-15 20:20:23,569 INFO - File "/usr/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_math_ops.py", line 2716, in square
2017-03-15 20:20:23,569 INFO - result = _op_def_lib.apply_op("Square", x=x, name=name)
2017-03-15 20:20:23,570 INFO - File "/usr/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op
2017-03-15 20:20:23,570 INFO - op_def=op_def)
2017-03-15 20:20:23,571 INFO - File "/usr/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2240, in create_op
2017-03-15 20:20:23,572 INFO - original_op=self._default_original_op, op_def=op_def)
2017-03-15 20:20:23,572 INFO - File "/usr/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1128, in __init__
2017-03-15 20:20:23,573 INFO - self._traceback = _extract_stack()
2017-03-15 20:20:23,573 INFO -
2017-03-15 20:20:23,574 INFO - ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[25088,4096]
2017-03-15 20:20:23,574 INFO - [[Node: Square_26 = Square[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](gradients/MatMul_grad/MatMul_1)]]
2017-03-15 20:20:23,575 INFO -
2017-03-15 20:20:23,633 INFO - E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:652] Deallocating stream with pending work
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment