-
-
Save munyari/6ba14f76cacc3d9e60f7a91610c25933 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2017-03-15 20:05:28,431 INFO - Preparing to run TaskInstance <TaskInstance: munyari/cats-dogs:50 (id: y4pY4Z4HiaTXMquPe7ddb7) (checksum: 0f0e7ccc7139aa4e5c183db40be43d46) (last update: 2017-03-15 20:05:28.396849) [queued]> | |
2017-03-15 20:05:28,451 INFO - Starting attempt 1 at 2017-03-15 20:05:28.438673 | |
2017-03-15 20:05:28,823 INFO - Executing command in container: stdbuf -o0 sh command.sh | |
2017-03-15 20:05:28,824 INFO - Pulling Docker image: floydhub/tensorflow:latest-gpu-py2 | |
2017-03-15 20:05:31,676 INFO - Starting container... | |
2017-03-15 20:05:32,861 INFO - | |
################################################################################ | |
2017-03-15 20:05:32,861 INFO - Run Output: | |
2017-03-15 20:18:19,192 INFO - Using TensorFlow backend. | |
2017-03-15 20:18:29,981 INFO - I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so locally | |
2017-03-15 20:18:29,985 INFO - I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.so locally | |
2017-03-15 20:18:29,986 INFO - I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so locally | |
2017-03-15 20:18:29,990 INFO - I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally | |
2017-03-15 20:18:29,990 INFO - I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so locally | |
2017-03-15 20:19:07,857 INFO - Found 23000 images belonging to 2 classes. | |
2017-03-15 20:19:08,338 INFO - Found 2000 images belonging to 2 classes. | |
2017-03-15 20:19:09,226 INFO - Epoch 1/20 | |
2017-03-15 20:19:12,227 INFO - I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero | |
2017-03-15 20:19:12,227 INFO - I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties: | |
2017-03-15 20:19:12,228 INFO - name: Tesla K80 | |
2017-03-15 20:19:12,228 INFO - major: 3 minor: 7 memoryClockRate (GHz) 0.8235 | |
2017-03-15 20:19:12,229 INFO - pciBusID 0000:00:1e.0 | |
2017-03-15 20:19:12,229 INFO - Total memory: 11.17GiB | |
2017-03-15 20:19:12,230 INFO - Free memory: 11.11GiB | |
2017-03-15 20:19:12,230 INFO - I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 | |
2017-03-15 20:19:12,231 INFO - I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0: Y | |
2017-03-15 20:19:12,231 INFO - I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0) | |
2017-03-15 20:19:54,308 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 3.45GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available. | |
2017-03-15 20:19:54,311 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 3.05GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available. | |
2017-03-15 20:19:55,548 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 2.58GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available. | |
2017-03-15 20:19:56,836 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 2.40GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available. | |
2017-03-15 20:19:58,217 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 2.40GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available. | |
2017-03-15 20:19:58,218 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 1.34GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available. | |
2017-03-15 20:19:58,893 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 2.40GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available. | |
2017-03-15 20:19:59,548 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 2.22GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available. | |
2017-03-15 20:20:00,975 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 2.58GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available. | |
2017-03-15 20:20:03,281 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:217] Ran out of memory trying to allocate 3.05GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory is available. | |
2017-03-15 20:20:21,990 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (256): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
2017-03-15 20:20:21,994 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (512): Total Chunks: 2, Chunks in use: 0 1.5KiB allocated for chunks. 520B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
2017-03-15 20:20:21,994 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1024): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
2017-03-15 20:20:21,995 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2048): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
2017-03-15 20:20:21,995 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4096): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
2017-03-15 20:20:21,995 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8192): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
2017-03-15 20:20:21,996 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16384): Total Chunks: 1, Chunks in use: 0 16.0KiB allocated for chunks. 16.0KiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
2017-03-15 20:20:21,996 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (32768): Total Chunks: 3, Chunks in use: 0 128.0KiB allocated for chunks. 64.06MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
2017-03-15 20:20:21,997 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (65536): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
2017-03-15 20:20:21,997 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (131072): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
2017-03-15 20:20:21,998 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (262144): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
2017-03-15 20:20:21,998 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (524288): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
2017-03-15 20:20:21,999 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (1048576): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
2017-03-15 20:20:21,999 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (2097152): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
2017-03-15 20:20:21,999 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (4194304): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
2017-03-15 20:20:22,000 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (8388608): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
2017-03-15 20:20:22,000 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (16777216): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
2017-03-15 20:20:22,000 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (33554432): Total Chunks: 1, Chunks in use: 0 57.88MiB allocated for chunks. 8B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
2017-03-15 20:20:22,001 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (67108864): Total Chunks: 1, Chunks in use: 0 68.00MiB allocated for chunks. 1.00MiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
2017-03-15 20:20:22,001 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (134217728): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
2017-03-15 20:20:22,002 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin (268435456): Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
2017-03-15 20:20:22,003 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:660] Bin for 392.00MiB was 256.00MiB, Chunk State: | |
2017-03-15 20:20:22,003 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0000 of size 1280 | |
2017-03-15 20:20:22,004 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0500 of size 256 | |
2017-03-15 20:20:22,004 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0600 of size 256 | |
2017-03-15 20:20:22,004 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0700 of size 256 | |
2017-03-15 20:20:22,005 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0800 of size 256 | |
2017-03-15 20:20:22,005 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0900 of size 512 | |
2017-03-15 20:20:22,005 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0b00 of size 256 | |
2017-03-15 20:20:22,006 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0c00 of size 256 | |
2017-03-15 20:20:22,006 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e0d00 of size 1024 | |
2017-03-15 20:20:22,007 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e1100 of size 256 | |
2017-03-15 20:20:22,007 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e1200 of size 256 | |
2017-03-15 20:20:22,007 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e1300 of size 2048 | |
2017-03-15 20:20:22,008 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e1b00 of size 256 | |
2017-03-15 20:20:22,008 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e1c00 of size 256 | |
2017-03-15 20:20:22,009 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e1d00 of size 16384 | |
2017-03-15 20:20:22,009 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e5d00 of size 256 | |
2017-03-15 20:20:22,010 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e5e00 of size 256 | |
2017-03-15 20:20:22,010 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e5f00 of size 256 | |
2017-03-15 20:20:22,010 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e6000 of size 256 | |
2017-03-15 20:20:22,011 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e6100 of size 256 | |
2017-03-15 20:20:22,011 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e6200 of size 256 | |
2017-03-15 20:20:22,012 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e6300 of size 256 | |
2017-03-15 20:20:22,012 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12050e6400 of size 9437184 | |
2017-03-15 20:20:22,013 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12059e6400 of size 6912 | |
2017-03-15 20:20:22,013 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12059e7f00 of size 147456 | |
2017-03-15 20:20:22,013 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205a0bf00 of size 294912 | |
2017-03-15 20:20:22,014 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205a53f00 of size 589824 | |
2017-03-15 20:20:22,014 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205ae3f00 of size 1179648 | |
2017-03-15 20:20:22,015 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205c03f00 of size 2359296 | |
2017-03-15 20:20:22,015 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1205e43f00 of size 4718592 | |
2017-03-15 20:20:22,016 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12062c3f00 of size 411041792 | |
2017-03-15 20:20:22,016 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x121eac3f00 of size 67108864 | |
2017-03-15 20:20:22,016 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ac3f00 of size 32768 | |
2017-03-15 20:20:22,017 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acbf00 of size 256 | |
2017-03-15 20:20:22,017 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc000 of size 256 | |
2017-03-15 20:20:22,018 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc100 of size 256 | |
2017-03-15 20:20:22,018 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc200 of size 256 | |
2017-03-15 20:20:22,019 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc300 of size 256 | |
2017-03-15 20:20:22,019 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc400 of size 256 | |
2017-03-15 20:20:22,020 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc500 of size 256 | |
2017-03-15 20:20:22,020 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc600 of size 256 | |
2017-03-15 20:20:22,021 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc700 of size 256 | |
2017-03-15 20:20:22,021 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc800 of size 256 | |
2017-03-15 20:20:22,024 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acc900 of size 256 | |
2017-03-15 20:20:22,025 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acca00 of size 256 | |
2017-03-15 20:20:22,025 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222accb00 of size 256 | |
2017-03-15 20:20:22,026 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222accc00 of size 256 | |
2017-03-15 20:20:22,026 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222accd00 of size 256 | |
2017-03-15 20:20:22,027 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acce00 of size 256 | |
2017-03-15 20:20:22,028 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222accf00 of size 256 | |
2017-03-15 20:20:22,029 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acd000 of size 256 | |
2017-03-15 20:20:22,030 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acd100 of size 256 | |
2017-03-15 20:20:22,030 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acd200 of size 256 | |
2017-03-15 20:20:22,032 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acd300 of size 256 | |
2017-03-15 20:20:22,033 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acd700 of size 256 | |
2017-03-15 20:20:22,033 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acd800 of size 256 | |
2017-03-15 20:20:22,034 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acdc00 of size 256 | |
2017-03-15 20:20:22,034 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222acdd00 of size 2048 | |
2017-03-15 20:20:22,035 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ace500 of size 256 | |
2017-03-15 20:20:22,035 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ace600 of size 6912 | |
2017-03-15 20:20:22,036 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ad0100 of size 32768 | |
2017-03-15 20:20:22,036 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ad8100 of size 16384 | |
2017-03-15 20:20:22,037 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ae8100 of size 6912 | |
2017-03-15 20:20:22,037 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ae9c00 of size 1024 | |
2017-03-15 20:20:22,038 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222aea000 of size 1024 | |
2017-03-15 20:20:22,038 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222aea400 of size 2048 | |
2017-03-15 20:20:22,039 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222aeac00 of size 2048 | |
2017-03-15 20:20:22,039 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222aeb400 of size 29184 | |
2017-03-15 20:20:22,040 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222af2600 of size 256 | |
2017-03-15 20:20:22,040 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222af2700 of size 16384 | |
2017-03-15 20:20:22,041 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222afa700 of size 16384 | |
2017-03-15 20:20:22,041 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222afe700 of size 16384 | |
2017-03-15 20:20:22,042 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222b02700 of size 32768 | |
2017-03-15 20:20:22,042 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222b16700 of size 147456 | |
2017-03-15 20:20:22,043 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222b3a700 of size 512 | |
2017-03-15 20:20:22,043 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222b3a900 of size 147456 | |
2017-03-15 20:20:22,043 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222b5e900 of size 147456 | |
2017-03-15 20:20:22,044 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222b82900 of size 294912 | |
2017-03-15 20:20:22,044 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222bca900 of size 512 | |
2017-03-15 20:20:22,045 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222bcab00 of size 294912 | |
2017-03-15 20:20:22,045 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222c12b00 of size 294912 | |
2017-03-15 20:20:22,046 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222c5ab00 of size 589824 | |
2017-03-15 20:20:22,047 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ceab00 of size 1024 | |
2017-03-15 20:20:22,047 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1222ceaf00 of size 3538944 | |
2017-03-15 20:20:22,048 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x122304af00 of size 1179648 | |
2017-03-15 20:20:22,048 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x122316af00 of size 1024 | |
2017-03-15 20:20:22,049 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x122316b300 of size 1024 | |
2017-03-15 20:20:22,049 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x122316b700 of size 2359296 | |
2017-03-15 20:20:22,050 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12233ab700 of size 589824 | |
2017-03-15 20:20:22,050 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x122343b700 of size 1769472 | |
2017-03-15 20:20:22,051 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12235eb700 of size 2048 | |
2017-03-15 20:20:22,051 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12235ebf00 of size 9437184 | |
2017-03-15 20:20:22,052 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1223eebf00 of size 9437184 | |
2017-03-15 20:20:22,052 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12247ebf00 of size 9437184 | |
2017-03-15 20:20:22,053 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12250ebf00 of size 14155776 | |
2017-03-15 20:20:22,053 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1225e6bf00 of size 4718592 | |
2017-03-15 20:20:22,054 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12262ebf00 of size 2048 | |
2017-03-15 20:20:22,054 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12262ec700 of size 2048 | |
2017-03-15 20:20:22,055 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12262ecf00 of size 2048 | |
2017-03-15 20:20:22,055 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12262ed700 of size 2048 | |
2017-03-15 20:20:22,056 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12262edf00 of size 2048 | |
2017-03-15 20:20:22,056 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12262ee700 of size 9437184 | |
2017-03-15 20:20:22,057 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1226bee700 of size 9437184 | |
2017-03-15 20:20:22,057 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12274ee700 of size 9437184 | |
2017-03-15 20:20:22,057 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1227dee700 of size 9437184 | |
2017-03-15 20:20:22,058 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12286ee700 of size 9437184 | |
2017-03-15 20:20:22,058 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1228fee700 of size 2359296 | |
2017-03-15 20:20:22,059 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x122922e700 of size 2359296 | |
2017-03-15 20:20:22,059 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x122946e700 of size 5505024 | |
2017-03-15 20:20:22,059 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12299ae700 of size 39226368 | |
2017-03-15 20:20:22,060 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x122bf17300 of size 89325568 | |
2017-03-15 20:20:22,060 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1231447300 of size 9437184 | |
2017-03-15 20:20:22,060 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1231d47300 of size 6422528 | |
2017-03-15 20:20:22,062 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1235d47300 of size 9437184 | |
2017-03-15 20:20:22,062 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1236647300 of size 9437184 | |
2017-03-15 20:20:22,063 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1236f47300 of size 9437184 | |
2017-03-15 20:20:22,063 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1237847300 of size 9437184 | |
2017-03-15 20:20:22,063 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1238147300 of size 589824 | |
2017-03-15 20:20:22,064 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12381d7300 of size 1179648 | |
2017-03-15 20:20:22,064 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12382f7300 of size 2359296 | |
2017-03-15 20:20:22,065 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1238537300 of size 2359296 | |
2017-03-15 20:20:22,065 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1238777300 of size 4718592 | |
2017-03-15 20:20:22,065 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1238bf7300 of size 9437184 | |
2017-03-15 20:20:22,066 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12394f7300 of size 90141696 | |
2017-03-15 20:20:22,066 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x123eaee700 of size 16384 | |
2017-03-15 20:20:22,068 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x123eaf2700 of size 67108864 | |
2017-03-15 20:20:22,068 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242af2700 of size 16384 | |
2017-03-15 20:20:22,069 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242af6700 of size 2048 | |
2017-03-15 20:20:22,069 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242af6f00 of size 2048 | |
2017-03-15 20:20:22,070 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242af7700 of size 2048 | |
2017-03-15 20:20:22,070 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242af7f00 of size 6912 | |
2017-03-15 20:20:22,071 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242af9a00 of size 256 | |
2017-03-15 20:20:22,071 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242af9b00 of size 512 | |
2017-03-15 20:20:22,072 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242af9d00 of size 512 | |
2017-03-15 20:20:22,072 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242af9f00 of size 1024 | |
2017-03-15 20:20:22,073 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afa300 of size 1024 | |
2017-03-15 20:20:22,073 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afa700 of size 1024 | |
2017-03-15 20:20:22,074 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afab00 of size 2048 | |
2017-03-15 20:20:22,075 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afb300 of size 2048 | |
2017-03-15 20:20:22,075 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afbb00 of size 256 | |
2017-03-15 20:20:22,076 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afbc00 of size 2048 | |
2017-03-15 20:20:22,076 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afc400 of size 2048 | |
2017-03-15 20:20:22,077 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afcc00 of size 2048 | |
2017-03-15 20:20:22,078 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afd400 of size 2048 | |
2017-03-15 20:20:22,078 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afdc00 of size 256 | |
2017-03-15 20:20:22,079 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afdd00 of size 256 | |
2017-03-15 20:20:22,079 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afde00 of size 512 | |
2017-03-15 20:20:22,080 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afe000 of size 512 | |
2017-03-15 20:20:22,081 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afe200 of size 1280 | |
2017-03-15 20:20:22,081 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afe700 of size 256 | |
2017-03-15 20:20:22,082 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afe800 of size 256 | |
2017-03-15 20:20:22,082 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afe900 of size 256 | |
2017-03-15 20:20:22,083 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afea00 of size 256 | |
2017-03-15 20:20:22,083 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afeb00 of size 256 | |
2017-03-15 20:20:22,083 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afec00 of size 256 | |
2017-03-15 20:20:22,084 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1242afed00 of size 9437184 | |
2017-03-15 20:20:22,084 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12433fed00 of size 2048 | |
2017-03-15 20:20:22,084 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12433ff500 of size 9437184 | |
2017-03-15 20:20:22,085 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1243cff500 of size 2048 | |
2017-03-15 20:20:22,085 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1243cffd00 of size 9437184 | |
2017-03-15 20:20:22,086 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12445ffd00 of size 2048 | |
2017-03-15 20:20:22,086 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244600500 of size 9437184 | |
2017-03-15 20:20:22,087 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244f00500 of size 2048 | |
2017-03-15 20:20:22,088 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244f00d00 of size 6912 | |
2017-03-15 20:20:22,088 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244f02800 of size 256 | |
2017-03-15 20:20:22,088 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244f02900 of size 147456 | |
2017-03-15 20:20:22,089 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244f26900 of size 256 | |
2017-03-15 20:20:22,089 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244f26a00 of size 294912 | |
2017-03-15 20:20:22,089 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244f6ea00 of size 512 | |
2017-03-15 20:20:22,090 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244f6ec00 of size 589824 | |
2017-03-15 20:20:22,090 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244ffec00 of size 512 | |
2017-03-15 20:20:22,090 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1244ffee00 of size 1179648 | |
2017-03-15 20:20:22,091 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x124511ee00 of size 1024 | |
2017-03-15 20:20:22,091 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x124511f200 of size 2359296 | |
2017-03-15 20:20:22,091 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x124535f200 of size 1024 | |
2017-03-15 20:20:22,093 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x124535f600 of size 2359296 | |
2017-03-15 20:20:22,093 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x124559f600 of size 1024 | |
2017-03-15 20:20:22,093 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x124559fa00 of size 4718592 | |
2017-03-15 20:20:22,094 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1245a1fa00 of size 2048 | |
2017-03-15 20:20:22,094 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1245a20200 of size 9437184 | |
2017-03-15 20:20:22,094 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1246320200 of size 2048 | |
2017-03-15 20:20:22,095 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1246320a00 of size 411041792 | |
2017-03-15 20:20:22,095 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x125eb20a00 of size 16384 | |
2017-03-15 20:20:22,096 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x125eb24a00 of size 67108864 | |
2017-03-15 20:20:22,096 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1262b24a00 of size 16384 | |
2017-03-15 20:20:22,096 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1262b28a00 of size 32768 | |
2017-03-15 20:20:22,097 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1262b30a00 of size 256 | |
2017-03-15 20:20:22,098 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1262b30b00 of size 9437184 | |
2017-03-15 20:20:22,099 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1263430b00 of size 2048 | |
2017-03-15 20:20:22,105 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1263431300 of size 9437184 | |
2017-03-15 20:20:22,106 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1263d31300 of size 2048 | |
2017-03-15 20:20:22,107 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1263d31b00 of size 9437184 | |
2017-03-15 20:20:22,107 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264631b00 of size 2048 | |
2017-03-15 20:20:22,108 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264632300 of size 9437184 | |
2017-03-15 20:20:22,108 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264f32300 of size 2048 | |
2017-03-15 20:20:22,109 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264f32b00 of size 6912 | |
2017-03-15 20:20:22,109 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264f34600 of size 256 | |
2017-03-15 20:20:22,109 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264f34700 of size 147456 | |
2017-03-15 20:20:22,110 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264f58700 of size 256 | |
2017-03-15 20:20:22,111 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264f58800 of size 294912 | |
2017-03-15 20:20:22,111 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264fa0800 of size 512 | |
2017-03-15 20:20:22,111 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1264fa0a00 of size 589824 | |
2017-03-15 20:20:22,112 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1265030a00 of size 512 | |
2017-03-15 20:20:22,112 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1265030c00 of size 1179648 | |
2017-03-15 20:20:22,112 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1265150c00 of size 1024 | |
2017-03-15 20:20:22,113 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1265151000 of size 2359296 | |
2017-03-15 20:20:22,113 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1265391000 of size 1024 | |
2017-03-15 20:20:22,117 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1265391400 of size 2359296 | |
2017-03-15 20:20:22,121 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12655d1400 of size 1024 | |
2017-03-15 20:20:22,121 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12655d1800 of size 4718592 | |
2017-03-15 20:20:22,122 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1265a51800 of size 2048 | |
2017-03-15 20:20:22,122 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1265a52000 of size 9437184 | |
2017-03-15 20:20:22,123 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1266352000 of size 2048 | |
2017-03-15 20:20:22,123 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1266352800 of size 411041792 | |
2017-03-15 20:20:22,125 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x127eb52800 of size 16384 | |
2017-03-15 20:20:22,127 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x127eb56800 of size 67108864 | |
2017-03-15 20:20:22,127 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1282b56800 of size 16384 | |
2017-03-15 20:20:22,128 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1282b5a800 of size 32768 | |
2017-03-15 20:20:22,128 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1282b62800 of size 256 | |
2017-03-15 20:20:22,128 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1282b62900 of size 429775616 | |
2017-03-15 20:20:22,131 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x129c540400 of size 822083584 | |
2017-03-15 20:20:22,131 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12cd540400 of size 836829184 | |
2017-03-15 20:20:22,132 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x12ff350400 of size 822083584 | |
2017-03-15 20:20:22,132 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1330350400 of size 205520896 | |
2017-03-15 20:20:22,133 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x133c750400 of size 212926464 | |
2017-03-15 20:20:22,133 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1349260400 of size 411041792 | |
2017-03-15 20:20:22,134 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1361a60400 of size 829423616 | |
2017-03-15 20:20:22,135 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1393160400 of size 411041792 | |
2017-03-15 20:20:22,135 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13ab960400 of size 411041792 | |
2017-03-15 20:20:22,137 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13c4160400 of size 16384 | |
2017-03-15 20:20:22,137 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13c416c400 of size 256 | |
2017-03-15 20:20:22,138 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13c416c500 of size 411041792 | |
2017-03-15 20:20:22,138 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13dc96c500 of size 102760448 | |
2017-03-15 20:20:22,139 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13e2b6c500 of size 110231552 | |
2017-03-15 20:20:22,139 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13e948c500 of size 205520896 | |
2017-03-15 20:20:22,140 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x13f588c500 of size 220463104 | |
2017-03-15 20:20:22,140 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1402acc500 of size 205520896 | |
2017-03-15 20:20:22,140 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x140eecc500 of size 220463104 | |
2017-03-15 20:20:22,141 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x141c10c500 of size 205520896 | |
2017-03-15 20:20:22,141 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x142850c500 of size 58982400 | |
2017-03-15 20:20:22,142 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x142bd4c500 of size 102760448 | |
2017-03-15 20:20:22,143 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1431f4c500 of size 117964800 | |
2017-03-15 20:20:22,144 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1438fcc500 of size 102760448 | |
2017-03-15 20:20:22,144 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x143f1cc500 of size 117964800 | |
2017-03-15 20:20:22,145 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x144624c500 of size 102760448 | |
2017-03-15 20:20:22,145 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x144c44c500 of size 25690112 | |
2017-03-15 20:20:22,146 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x144dccc500 of size 33554432 | |
2017-03-15 20:20:22,146 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x144fccc500 of size 25690112 | |
2017-03-15 20:20:22,147 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x145154c500 of size 33554432 | |
2017-03-15 20:20:22,147 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x145354c500 of size 25690112 | |
2017-03-15 20:20:22,148 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1454dcc500 of size 33554432 | |
2017-03-15 20:20:22,148 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1456dcc500 of size 25690112 | |
2017-03-15 20:20:22,150 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x145864c500 of size 6422528 | |
2017-03-15 20:20:22,150 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x145d06c500 of size 67108864 | |
2017-03-15 20:20:22,150 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x146106c500 of size 411041792 | |
2017-03-15 20:20:22,151 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x147986c500 of size 788662016 | |
2017-03-15 20:20:22,151 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x1222acd400 of size 768 | |
2017-03-15 20:20:22,152 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x1222acd900 of size 768 | |
2017-03-15 20:20:22,152 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x1222adc100 of size 49152 | |
2017-03-15 20:20:22,153 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x1222af6700 of size 16384 | |
2017-03-15 20:20:22,153 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x1222b0a700 of size 49152 | |
2017-03-15 20:20:22,154 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x1232367300 of size 60686336 | |
2017-03-15 20:20:22,155 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x13c4164400 of size 32768 | |
2017-03-15 20:20:22,156 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x1458c6c500 of size 71303168 | |
2017-03-15 20:20:22,156 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:693] Summary of in-use Chunks by size: | |
2017-03-15 20:20:22,157 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 60 Chunks of size 256 totalling 15.0KiB | |
2017-03-15 20:20:22,157 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 11 Chunks of size 512 totalling 5.5KiB | |
2017-03-15 20:20:22,158 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 15 Chunks of size 1024 totalling 15.0KiB | |
2017-03-15 20:20:22,158 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 1280 totalling 2.5KiB | |
2017-03-15 20:20:22,159 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 31 Chunks of size 2048 totalling 62.0KiB | |
2017-03-15 20:20:22,159 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 6 Chunks of size 6912 totalling 40.5KiB | |
2017-03-15 20:20:22,160 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 12 Chunks of size 16384 totalling 192.0KiB | |
2017-03-15 20:20:22,160 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 29184 totalling 28.5KiB | |
2017-03-15 20:20:22,162 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 5 Chunks of size 32768 totalling 160.0KiB | |
2017-03-15 20:20:22,162 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 6 Chunks of size 147456 totalling 864.0KiB | |
2017-03-15 20:20:22,163 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 6 Chunks of size 294912 totalling 1.69MiB | |
2017-03-15 20:20:22,163 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 6 Chunks of size 589824 totalling 3.38MiB | |
2017-03-15 20:20:22,164 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 5 Chunks of size 1179648 totalling 5.62MiB | |
2017-03-15 20:20:22,164 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 1769472 totalling 1.69MiB | |
2017-03-15 20:20:22,165 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 10 Chunks of size 2359296 totalling 22.50MiB | |
2017-03-15 20:20:22,165 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 3538944 totalling 3.38MiB | |
2017-03-15 20:20:22,166 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 5 Chunks of size 4718592 totalling 22.50MiB | |
2017-03-15 20:20:22,166 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 5505024 totalling 5.25MiB | |
2017-03-15 20:20:22,167 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 6422528 totalling 12.25MiB | |
2017-03-15 20:20:22,168 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 25 Chunks of size 9437184 totalling 225.00MiB | |
2017-03-15 20:20:22,169 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 14155776 totalling 13.50MiB | |
2017-03-15 20:20:22,170 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 4 Chunks of size 25690112 totalling 98.00MiB | |
2017-03-15 20:20:22,170 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 3 Chunks of size 33554432 totalling 96.00MiB | |
2017-03-15 20:20:22,171 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 39226368 totalling 37.41MiB | |
2017-03-15 20:20:22,172 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 58982400 totalling 56.25MiB | |
2017-03-15 20:20:22,172 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 5 Chunks of size 67108864 totalling 320.00MiB | |
2017-03-15 20:20:22,173 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 89325568 totalling 85.19MiB | |
2017-03-15 20:20:22,174 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 90141696 totalling 85.97MiB | |
2017-03-15 20:20:22,174 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 4 Chunks of size 102760448 totalling 392.00MiB | |
2017-03-15 20:20:22,175 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 110231552 totalling 105.12MiB | |
2017-03-15 20:20:22,176 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 117964800 totalling 225.00MiB | |
2017-03-15 20:20:22,177 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 4 Chunks of size 205520896 totalling 784.00MiB | |
2017-03-15 20:20:22,177 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 212926464 totalling 203.06MiB | |
2017-03-15 20:20:22,178 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 220463104 totalling 420.50MiB | |
2017-03-15 20:20:22,179 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 8 Chunks of size 411041792 totalling 3.06GiB | |
2017-03-15 20:20:22,179 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 429775616 totalling 409.87MiB | |
2017-03-15 20:20:22,179 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 788662016 totalling 752.13MiB | |
2017-03-15 20:20:22,180 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 822083584 totalling 1.53GiB | |
2017-03-15 20:20:22,180 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 829423616 totalling 791.00MiB | |
2017-03-15 20:20:22,181 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 836829184 totalling 798.06MiB | |
2017-03-15 20:20:22,182 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:700] Sum Total of in-use chunks: 10.43GiB | |
2017-03-15 20:20:22,182 INFO - I tensorflow/core/common_runtime/bfc_allocator.cc:702] Stats: | |
2017-03-15 20:20:22,183 INFO - Limit: 11332668621 | |
2017-03-15 20:20:22,183 INFO - InUse: 11200529920 | |
2017-03-15 20:20:22,184 INFO - MaxInUse: 11332668160 | |
2017-03-15 20:20:22,184 INFO - NumAllocs: 2445 | |
2017-03-15 20:20:22,185 INFO - MaxAllocSize: 3087794176 | |
2017-03-15 20:20:22,187 INFO - | |
2017-03-15 20:20:22,187 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:274] ********************************************************xx***************************************xxx | |
2017-03-15 20:20:22,188 INFO - W tensorflow/core/common_runtime/bfc_allocator.cc:275] Ran out of memory trying to allocate 392.00MiB. See logs for memory state. | |
2017-03-15 20:20:22,189 INFO - W tensorflow/core/framework/op_kernel.cc:975] Resource exhausted: OOM when allocating tensor with shape[25088,4096] | |
2017-03-15 20:20:23,489 INFO - Traceback (most recent call last): | |
2017-03-15 20:20:23,489 INFO - File "demo1.py", line 129, in <module> | |
2017-03-15 20:20:23,501 INFO - nb_epoch=20, validation_data=valid_batches, nb_val_samples=valid_batches.nb_sample) | |
2017-03-15 20:20:23,502 INFO - File "/usr/local/lib/python2.7/site-packages/keras/models.py", line 935, in fit_generator | |
2017-03-15 20:20:23,507 INFO - initial_epoch=initial_epoch) | |
2017-03-15 20:20:23,507 INFO - File "/usr/local/lib/python2.7/site-packages/keras/engine/training.py", line 1557, in fit_generator | |
2017-03-15 20:20:23,528 INFO - class_weight=class_weight) | |
2017-03-15 20:20:23,529 INFO - File "/usr/local/lib/python2.7/site-packages/keras/engine/training.py", line 1320, in train_on_batch | |
2017-03-15 20:20:23,529 INFO - outputs = self.train_function(ins) | |
2017-03-15 20:20:23,530 INFO - File "/usr/local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 1943, in __call__ | |
2017-03-15 20:20:23,530 INFO - feed_dict=feed_dict) | |
2017-03-15 20:20:23,530 INFO - File "/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 766, in run | |
2017-03-15 20:20:23,534 INFO - run_metadata_ptr) | |
2017-03-15 20:20:23,534 INFO - File "/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 964, in _run | |
2017-03-15 20:20:23,534 INFO - feed_dict_string, options, run_metadata) | |
2017-03-15 20:20:23,536 INFO - File "/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1014, in _do_run | |
2017-03-15 20:20:23,537 INFO - target_list, options, run_metadata) | |
2017-03-15 20:20:23,537 INFO - File "/usr/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1034, in _do_call | |
2017-03-15 20:20:23,537 INFO - raise type(e)(node_def, op, message) | |
2017-03-15 20:20:23,560 INFO - tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[25088,4096] | |
2017-03-15 20:20:23,560 INFO - [[Node: Square_26 = Square[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](gradients/MatMul_grad/MatMul_1)]] | |
2017-03-15 20:20:23,561 INFO - | |
2017-03-15 20:20:23,561 INFO - Caused by op u'Square_26', defined at: | |
2017-03-15 20:20:23,562 INFO - File "demo1.py", line 129, in <module> | |
2017-03-15 20:20:23,562 INFO - nb_epoch=20, validation_data=valid_batches, nb_val_samples=valid_batches.nb_sample) | |
2017-03-15 20:20:23,563 INFO - File "/usr/local/lib/python2.7/site-packages/keras/models.py", line 935, in fit_generator | |
2017-03-15 20:20:23,563 INFO - initial_epoch=initial_epoch) | |
2017-03-15 20:20:23,564 INFO - File "/usr/local/lib/python2.7/site-packages/keras/engine/training.py", line 1454, in fit_generator | |
2017-03-15 20:20:23,564 INFO - self._make_train_function() | |
2017-03-15 20:20:23,565 INFO - File "/usr/local/lib/python2.7/site-packages/keras/engine/training.py", line 760, in _make_train_function | |
2017-03-15 20:20:23,565 INFO - self.total_loss) | |
2017-03-15 20:20:23,566 INFO - File "/usr/local/lib/python2.7/site-packages/keras/optimizers.py", line 434, in get_updates | |
2017-03-15 20:20:23,566 INFO - v_t = (self.beta_2 * v) + (1. - self.beta_2) * K.square(g) | |
2017-03-15 20:20:23,567 INFO - File "/usr/local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 1186, in square | |
2017-03-15 20:20:23,567 INFO - return tf.square(x) | |
2017-03-15 20:20:23,568 INFO - File "/usr/local/lib/python2.7/site-packages/tensorflow/python/ops/math_ops.py", line 385, in square | |
2017-03-15 20:20:23,568 INFO - return gen_math_ops.square(x, name=name) | |
2017-03-15 20:20:23,569 INFO - File "/usr/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_math_ops.py", line 2716, in square | |
2017-03-15 20:20:23,569 INFO - result = _op_def_lib.apply_op("Square", x=x, name=name) | |
2017-03-15 20:20:23,570 INFO - File "/usr/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op | |
2017-03-15 20:20:23,570 INFO - op_def=op_def) | |
2017-03-15 20:20:23,571 INFO - File "/usr/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2240, in create_op | |
2017-03-15 20:20:23,572 INFO - original_op=self._default_original_op, op_def=op_def) | |
2017-03-15 20:20:23,572 INFO - File "/usr/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1128, in __init__ | |
2017-03-15 20:20:23,573 INFO - self._traceback = _extract_stack() | |
2017-03-15 20:20:23,573 INFO - | |
2017-03-15 20:20:23,574 INFO - ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[25088,4096] | |
2017-03-15 20:20:23,574 INFO - [[Node: Square_26 = Square[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"](gradients/MatMul_grad/MatMul_1)]] | |
2017-03-15 20:20:23,575 INFO - | |
2017-03-15 20:20:23,633 INFO - E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:652] Deallocating stream with pending work | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment