Created
March 21, 2018 16:07
-
-
Save ps2/43bad496aa39da8a536d5862412d74b9 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| (venv) pete@sandbox-gpu1:~/smclocalize$ python3 bin/smclocalize_worker -c 1289 -n 100000 | |
| Width from firebase = (width:7.5, length:14.5) | |
| room_size = [14.5 7.5] | |
| room_corners = [[ 0. 0. ] | |
| [14.5 7.5]] | |
| child_entity_ids = [28348, 28349, 28350, 28352, 28354, 30007, 36212, 37757, 37758, 37760, 38394, 55514, 57651] | |
| material_entity_ids = [29, 30, 31, 32, 33, 34, 688554, 688561, 688570] | |
| teacher_entity_ids = [11028, 11029] | |
| area_entity_ids = [29, 31, 36, 49, 50, 51, 52, 53, 54] | |
| fixed_sensor_positions = [[ 7.25 3.75] | |
| [12.27 5.36] | |
| [12.95 2.57] | |
| [ 6.91 3.73] | |
| [ 7.25 3.75] | |
| [ 2.26 5.92] | |
| [10.97 1.5 ] | |
| [ 9.63 0.15] | |
| [ 3.35 3.38]] | |
| 2018-03-21 11:06:14.093756: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA | |
| 2018-03-21 11:06:14.224077: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero | |
| 2018-03-21 11:06:14.224475: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1212] Found device 0 with properties: | |
| name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.6575 | |
| pciBusID: 0000:0b:00.0 | |
| totalMemory: 10.92GiB freeMemory: 10.65GiB | |
| 2018-03-21 11:06:14.224492: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1312] Adding visible gpu devices: 0 | |
| 2018-03-21 11:06:14.399352: I tensorflow/core/common_runtime/gpu/gpu_device.cc:993] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10308 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0b:00.0, compute capability: 6.1) | |
| 2018-03-21 11:06:14.776510: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1312] Adding visible gpu devices: 0 | |
| 2018-03-21 11:06:14.776663: I tensorflow/core/common_runtime/gpu/gpu_device.cc:993] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 315 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:0b:00.0, compute capability: 6.1) | |
| Processing frame 2018-03-21 14:26:00 | |
| *** processing time: 1654ms | |
| child 28348 (18): (13.202321, 6.312410) | |
| child 28349 (28): (12.710181, 4.862849) | |
| child 28350 (26): (11.587911, 6.492370) | |
| child 28352 (17): (5.285855, 4.255816) | |
| child 28354 (16): (10.863827, 1.658994) | |
| child 30007 (14): (3.646251, 1.438483) | |
| child 36212 (19): (3.834174, 5.289225) | |
| child 37757 (20): (11.050615, 6.953112) | |
| child 37758 (11): (0.515732, 1.210853) | |
| child 37760 (12): (7.585400, 7.076092) | |
| child 38394 (15): (4.318734, 2.498019) | |
| child 55514 (21): (12.605453, 6.358153) | |
| child 57651 (13): (13.325860, 2.404974) | |
| material 29 (35): (6.030159, 5.329052) | |
| material 30 (42): (9.465969, 2.780535) | |
| material 31 (36): (9.679503, 3.397684) | |
| material 32 (37): (1.772505, 1.829466) | |
| material 33 (38): (8.051176, 3.640503) | |
| material 34 (39): (1.978853, 5.334323) | |
| material 688554 (41): (0.898999, 3.581683) | |
| material 688561 (44): (4.998163, 1.220644) | |
| material 688570 (40): (9.351588, 1.494488) | |
| teacher 11028 (30): (0.475744, 7.097616) | |
| teacher 11029 (31): (8.697465, 4.891424) | |
| ================================================================================ | |
| Processing frame 2018-03-21 14:26:10 | |
| 2018-03-21 11:06:27.976320: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 37.25GiB. Current allocation summary follows. | |
| 2018-03-21 11:06:27.976414: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (256): Total Chunks: 47, Chunks in use: 47. 11.8KiB allocated for chunks. 11.8KiB in use in bin. 572B client-requested in use in bin. | |
| 2018-03-21 11:06:27.976434: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (512): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
| 2018-03-21 11:06:27.976452: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1024): Total Chunks: 2, Chunks in use: 2. 2.5KiB allocated for chunks. 2.5KiB in use in bin. 2.0KiB client-requested in use in bin. | |
| 2018-03-21 11:06:27.976468: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2048): Total Chunks: 1, Chunks in use: 0. 3.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
| 2018-03-21 11:06:27.976485: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4096): Total Chunks: 2, Chunks in use: 2. 12.0KiB allocated for chunks. 12.0KiB in use in bin. 11.7KiB client-requested in use in bin. | |
| 2018-03-21 11:06:27.976501: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
| 2018-03-21 11:06:27.976516: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
| 2018-03-21 11:06:27.976530: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
| 2018-03-21 11:06:27.976545: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
| 2018-03-21 11:06:27.976560: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (131072): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
| 2018-03-21 11:06:27.976578: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (262144): Total Chunks: 1, Chunks in use: 1. 390.8KiB allocated for chunks. 390.8KiB in use in bin. 390.6KiB client-requested in use in bin. | |
| 2018-03-21 11:06:27.976595: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (524288): Total Chunks: 1, Chunks in use: 1. 781.2KiB allocated for chunks. 781.2KiB in use in bin. 781.2KiB client-requested in use in bin. | |
| 2018-03-21 11:06:27.976610: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
| 2018-03-21 11:06:27.976625: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
| 2018-03-21 11:06:27.976642: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (4194304): Total Chunks: 2, Chunks in use: 2. 13.73MiB allocated for chunks. 13.73MiB in use in bin. 13.73MiB client-requested in use in bin. | |
| 2018-03-21 11:06:27.976660: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (8388608): Total Chunks: 3, Chunks in use: 3. 37.77MiB allocated for chunks. 37.77MiB in use in bin. 37.76MiB client-requested in use in bin. | |
| 2018-03-21 11:06:27.976676: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (16777216): Total Chunks: 1, Chunks in use: 1. 18.31MiB allocated for chunks. 18.31MiB in use in bin. 18.31MiB client-requested in use in bin. | |
| 2018-03-21 11:06:27.976691: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
| 2018-03-21 11:06:27.976709: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (67108864): Total Chunks: 3, Chunks in use: 3. 214.00MiB allocated for chunks. 214.00MiB in use in bin. 214.00MiB client-requested in use in bin. | |
| 2018-03-21 11:06:27.976724: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. | |
| 2018-03-21 11:06:27.976740: I tensorflow/core/common_runtime/bfc_allocator.cc:630] Bin (268435456): Total Chunks: 10, Chunks in use: 9. 9.79GiB allocated for chunks. 2.51GiB in use in bin. 2.51GiB client-requested in use in bin. | |
| 2018-03-21 11:06:27.976756: I tensorflow/core/common_runtime/bfc_allocator.cc:646] Bin for 37.25GiB was 256.00MiB, Chunk State: | |
| 2018-03-21 11:06:27.976777: I tensorflow/core/common_runtime/bfc_allocator.cc:652] Size: 7.28GiB | Requested Size: 0B | in_use: 0, prev: Size: 71.33MiB | Requested Size: 71.33MiB | in_use: 1 | |
| 2018-03-21 11:06:27.976793: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f27bba00000 of size 1280 | |
| 2018-03-21 11:06:27.976805: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f27bba00500 of size 1280 | |
| 2018-03-21 11:06:27.976818: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f27bba00a00 of size 256 | |
| 2018-03-21 11:06:27.976829: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f27bba00b00 of size 256 | |
| 2018-03-21 11:06:27.976840: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f27bba00c00 of size 256 | |
| 2018-03-21 11:06:27.976851: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f27bba00d00 of size 256 | |
| 2018-03-21 11:06:27.976862: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f27bba00e00 of size 6144 | |
| 2018-03-21 11:06:27.976874: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f27bba02600 of size 299200000 | |
| 2018-03-21 11:06:27.976886: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f27cd759400 of size 299200000 | |
| 2018-03-21 11:06:27.976897: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f27df4b0200 of size 299200000 | |
| 2018-03-21 11:06:27.976908: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f27f1207000 of size 299200000 | |
| 2018-03-21 11:06:27.976920: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f2802f5de00 of size 256 | |
| 2018-03-21 11:06:27.976933: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f2802f5df00 of size 256 | |
| 2018-03-21 11:06:27.976944: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f2802f5e000 of size 256 | |
| 2018-03-21 11:06:27.976955: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f2802f5e100 of size 256 | |
| 2018-03-21 11:06:27.976967: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f2802f5e200 of size 256 | |
| 2018-03-21 11:06:27.976978: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f2802f5e300 of size 256 | |
| 2018-03-21 11:06:27.976990: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f2802f5e400 of size 256 | |
| 2018-03-21 11:06:27.977001: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f2802f5e500 of size 256 | |
| 2018-03-21 11:06:27.977013: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f2802f5e600 of size 256 | |
| 2018-03-21 11:06:27.977024: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f2802f5e700 of size 256 | |
| 2018-03-21 11:06:27.977035: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f2802f5e800 of size 256 | |
| 2018-03-21 11:06:27.977046: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f2802f5e900 of size 256 | |
| 2018-03-21 11:06:27.977057: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f2802f5ea00 of size 256 | |
| 2018-03-21 11:06:27.977069: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f2802f5eb00 of size 256 | |
| 2018-03-21 11:06:27.977080: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f2802f5ec00 of size 256 | |
| 2018-03-21 11:06:27.977100: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f2802f5ed00 of size 256 | |
| 2018-03-21 11:06:27.977113: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f2802f5ee00 of size 256 | |
| 2018-03-21 11:06:27.977125: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f2802f5ef00 of size 7200000 | |
| 2018-03-21 11:06:27.977136: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f280363cc00 of size 256 | |
| 2018-03-21 11:06:27.977147: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f280363cd00 of size 256 | |
| 2018-03-21 11:06:27.977160: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f280363ce00 of size 256 | |
| 2018-03-21 11:06:27.977172: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f280363cf00 of size 256 | |
| 2018-03-21 11:06:27.977183: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f280363d000 of size 256 | |
| 2018-03-21 11:06:27.977195: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f280363d100 of size 13200128 | |
| 2018-03-21 11:06:27.977206: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f28042d3c00 of size 6144 | |
| 2018-03-21 11:06:27.977217: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f28042d5400 of size 299200000 | |
| 2018-03-21 11:06:27.977228: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f281602c200 of size 299200000 | |
| 2018-03-21 11:06:27.977238: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f2827d83000 of size 299200000 | |
| 2018-03-21 11:06:27.977250: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f2839ad9e00 of size 299200000 | |
| 2018-03-21 11:06:27.977261: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284b830c00 of size 256 | |
| 2018-03-21 11:06:27.977273: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284b830d00 of size 256 | |
| 2018-03-21 11:06:27.977284: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284b830e00 of size 256 | |
| 2018-03-21 11:06:27.977295: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284b830f00 of size 256 | |
| 2018-03-21 11:06:27.977307: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284b831000 of size 256 | |
| 2018-03-21 11:06:27.977319: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284b831100 of size 256 | |
| 2018-03-21 11:06:27.977331: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284b831200 of size 256 | |
| 2018-03-21 11:06:27.977345: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284b831300 of size 256 | |
| 2018-03-21 11:06:27.977357: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284b831400 of size 256 | |
| 2018-03-21 11:06:27.977369: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284b831500 of size 256 | |
| 2018-03-21 11:06:27.977380: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284b831600 of size 256 | |
| 2018-03-21 11:06:27.977392: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284b831700 of size 256 | |
| 2018-03-21 11:06:27.977403: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284b831800 of size 256 | |
| 2018-03-21 11:06:27.977414: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284b831900 of size 256 | |
| 2018-03-21 11:06:27.977426: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284b831a00 of size 256 | |
| 2018-03-21 11:06:27.977438: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284b831b00 of size 256 | |
| 2018-03-21 11:06:27.977449: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284b831c00 of size 256 | |
| 2018-03-21 11:06:27.977460: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284b831d00 of size 7200000 | |
| 2018-03-21 11:06:27.977470: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284bf0fa00 of size 256 | |
| 2018-03-21 11:06:27.977481: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284bf0fb00 of size 256 | |
| 2018-03-21 11:06:27.977492: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284bf0fc00 of size 256 | |
| 2018-03-21 11:06:27.977502: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7f284bf0fd00 of size 3072 | |
| 2018-03-21 11:06:27.977513: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284bf10900 of size 13200128 | |
| 2018-03-21 11:06:27.977525: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284cba7400 of size 19200000 | |
| 2018-03-21 11:06:27.977535: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284ddf6c00 of size 256 | |
| 2018-03-21 11:06:27.977546: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284ddf6d00 of size 400128 | |
| 2018-03-21 11:06:27.977556: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284de58800 of size 13200128 | |
| 2018-03-21 11:06:27.977567: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284eaef300 of size 800000 | |
| 2018-03-21 11:06:27.977578: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f284ebb2800 of size 299200000 | |
| 2018-03-21 11:06:27.977588: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f2860909600 of size 74800128 | |
| 2018-03-21 11:06:27.977598: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f286505f200 of size 74800128 | |
| 2018-03-21 11:06:27.977607: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7f28697b4e00 of size 74800128 | |
| 2018-03-21 11:06:27.977619: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7f286df0aa00 of size 7817375488 | |
| 2018-03-21 11:06:27.977630: I tensorflow/core/common_runtime/bfc_allocator.cc:671] Summary of in-use Chunks by size: | |
| 2018-03-21 11:06:27.977644: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 47 Chunks of size 256 totalling 11.8KiB | |
| 2018-03-21 11:06:27.977657: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 1280 totalling 2.5KiB | |
| 2018-03-21 11:06:27.977668: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 6144 totalling 12.0KiB | |
| 2018-03-21 11:06:27.977679: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 400128 totalling 390.8KiB | |
| 2018-03-21 11:06:27.977691: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 800000 totalling 781.2KiB | |
| 2018-03-21 11:06:27.977704: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 7200000 totalling 13.73MiB | |
| 2018-03-21 11:06:27.977715: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 3 Chunks of size 13200128 totalling 37.77MiB | |
| 2018-03-21 11:06:27.977726: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 19200000 totalling 18.31MiB | |
| 2018-03-21 11:06:27.977738: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 3 Chunks of size 74800128 totalling 214.00MiB | |
| 2018-03-21 11:06:27.977749: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 9 Chunks of size 299200000 totalling 2.51GiB | |
| 2018-03-21 11:06:27.977760: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Sum Total of in-use chunks: 2.79GiB | |
| 2018-03-21 11:06:27.977774: I tensorflow/core/common_runtime/bfc_allocator.cc:680] Stats: | |
| Limit: 10809006490 | |
| InUse: 2991627776 | |
| MaxInUse: 5425218816 | |
| NumAllocs: 120 | |
| MaxAllocSize: 871200000 | |
| 2018-03-21 11:06:27.977796: W tensorflow/core/common_runtime/bfc_allocator.cc:279] ****************************________________________________________________________________________ | |
| 2018-03-21 11:06:27.977829: W tensorflow/core/framework/op_kernel.cc:1202] OP_REQUIRES failed at multinomial_op.cc:181 : Resource exhausted: OOM when allocating tensor with shape[1,100000,100000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc | |
| Traceback (most recent call last): | |
| File "/home/pete/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1361, in _do_call | |
| return fn(*args) | |
| File "/home/pete/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1340, in _run_fn | |
| target_list, status, run_metadata) | |
| File "/home/pete/.local/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 516, in __exit__ | |
| c_api.TF_GetCode(self.status.status)) | |
| tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1,100000,100000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc | |
| [[Node: create_next_particles/create_ancestor_indices/create_ancestor_indices/Multinomial = Multinomial[T=DT_FLOAT, output_dtype=DT_INT64, seed=0, seed2=0, _device="/job:localhost/replica:0/task:0/device:GPU:0"](create_next_particles/create_ancestor_indices/create_ancestor_indices/logits, create_next_particles/create_ancestor_indices/strided_slice)]] | |
| Current usage from device: /job:localhost/replica:0/task:0/device:GPU:0, allocator: GPU_0_bfc | |
| 285.34MiB from create_next_particles/create_y_bar_x_log_pdf/y_continuous_broadcast | |
| 71.33MiB from create_next_particles/create_y_bar_x_log_pdf/create_y_less_than_rssi_cutoff_boolean | |
| 12.59MiB from create_next_particles/create_x_bar_x_previous_sample/bernoulli_distribution_1/create_status_changed_tensor/random_uniform/RandomUniform | |
| [[Node: create_next_particles/create_x_bar_x_previous_sample/create_room_min_scaled/_37 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_36_cr...min_scaled", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]] | |
| Current usage from device: /job:localhost/replica:0/task:0/device:GPU:0, allocator: GPU_0_bfc | |
| 285.34MiB from create_next_particles/create_y_bar_x_log_pdf/y_continuous_broadcast | |
| 71.33MiB from create_next_particles/create_y_bar_x_log_pdf/create_y_less_than_rssi_cutoff_boolean | |
| 12.59MiB from create_next_particles/create_x_bar_x_previous_sample/bernoulli_distribution_1/create_status_changed_tensor/random_uniform/RandomUniform | |
| During handling of the above exception, another exception occurred: | |
| Traceback (most recent call last): | |
| File "bin/smclocalize_worker", line 207, in <module> | |
| worker.run() | |
| File "bin/smclocalize_worker", line 143, in run | |
| locations = self.process_frame(frame, frame_valid_time, delta_t, initial_frame) | |
| File "bin/smclocalize_worker", line 153, in process_frame | |
| self.state = ModelState(*self.sensor_model.generate_next_particles(*self.state, y_discrete, y_continuous, delta_t)[:3]) | |
| File "/home/pete/smclocalize/smclocalize/smc_model.py", line 373, in generate_next_particles | |
| options=tf.RunOptions(report_tensor_allocations_upon_oom=True)) | |
| File "/home/pete/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 905, in run | |
| run_metadata_ptr) | |
| File "/home/pete/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1137, in _run | |
| feed_dict_tensor, options, run_metadata) | |
| File "/home/pete/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1355, in _do_run | |
| options, run_metadata) | |
| File "/home/pete/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1374, in _do_call | |
| raise type(e)(node_def, op, message) | |
| tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1,100000,100000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc | |
| [[Node: create_next_particles/create_ancestor_indices/create_ancestor_indices/Multinomial = Multinomial[T=DT_FLOAT, output_dtype=DT_INT64, seed=0, seed2=0, _device="/job:localhost/replica:0/task:0/device:GPU:0"](create_next_particles/create_ancestor_indices/create_ancestor_indices/logits, create_next_particles/create_ancestor_indices/strided_slice)]] | |
| Current usage from device: /job:localhost/replica:0/task:0/device:GPU:0, allocator: GPU_0_bfc | |
| 285.34MiB from create_next_particles/create_y_bar_x_log_pdf/y_continuous_broadcast | |
| 71.33MiB from create_next_particles/create_y_bar_x_log_pdf/create_y_less_than_rssi_cutoff_boolean | |
| 12.59MiB from create_next_particles/create_x_bar_x_previous_sample/bernoulli_distribution_1/create_status_changed_tensor/random_uniform/RandomUniform | |
| [[Node: create_next_particles/create_x_bar_x_previous_sample/create_room_min_scaled/_37 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_36_cr...min_scaled", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]] | |
| Current usage from device: /job:localhost/replica:0/task:0/device:GPU:0, allocator: GPU_0_bfc | |
| 285.34MiB from create_next_particles/create_y_bar_x_log_pdf/y_continuous_broadcast | |
| 71.33MiB from create_next_particles/create_y_bar_x_log_pdf/create_y_less_than_rssi_cutoff_boolean | |
| 12.59MiB from create_next_particles/create_x_bar_x_previous_sample/bernoulli_distribution_1/create_status_changed_tensor/random_uniform/RandomUniform | |
| Caused by op 'create_next_particles/create_ancestor_indices/create_ancestor_indices/Multinomial', defined at: | |
| File "bin/smclocalize_worker", line 206, in <module> | |
| worker = LocationModelWorker(options.classroom_id, options.num_particles) | |
| File "bin/smclocalize_worker", line 124, in __init__ | |
| self.num_particles) | |
| File "/home/pete/smclocalize/smclocalize/sensor_model.py", line 65, in __init__ | |
| num_particles) | |
| File "/home/pete/smclocalize/smclocalize/smc_model.py", line 64, in __init__ | |
| self.t_delta_seconds_tensor) | |
| File "/home/pete/smclocalize/smclocalize/smc_model.py", line 163, in create_next_particles_tensor | |
| log_weights_previous_tensor) | |
| File "/home/pete/smclocalize/smclocalize/smc_model.py", line 267, in create_ancestor_indices_tensor | |
| name='create_ancestor_indices') | |
| File "/home/pete/.local/lib/python3.5/site-packages/tensorflow/python/ops/random_ops.py", line 355, in multinomial | |
| logits, num_samples, seed=seed1, seed2=seed2, output_dtype=output_dtype) | |
| File "/home/pete/.local/lib/python3.5/site-packages/tensorflow/python/ops/gen_random_ops.py", line 58, in multinomial | |
| seed2=seed2, output_dtype=output_dtype, name=name) | |
| File "/home/pete/.local/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper | |
| op_def=op_def) | |
| File "/home/pete/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3271, in create_op | |
| op_def=op_def) | |
| File "/home/pete/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1650, in __init__ | |
| self._traceback = self._graph._extract_stack() # pylint: disable=protected-access | |
| ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[1,100000,100000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc | |
| [[Node: create_next_particles/create_ancestor_indices/create_ancestor_indices/Multinomial = Multinomial[T=DT_FLOAT, output_dtype=DT_INT64, seed=0, seed2=0, _device="/job:localhost/replica:0/task:0/device:GPU:0"](create_next_particles/create_ancestor_indices/create_ancestor_indices/logits, create_next_particles/create_ancestor_indices/strided_slice)]] | |
| Current usage from device: /job:localhost/replica:0/task:0/device:GPU:0, allocator: GPU_0_bfc | |
| 285.34MiB from create_next_particles/create_y_bar_x_log_pdf/y_continuous_broadcast | |
| 71.33MiB from create_next_particles/create_y_bar_x_log_pdf/create_y_less_than_rssi_cutoff_boolean | |
| 12.59MiB from create_next_particles/create_x_bar_x_previous_sample/bernoulli_distribution_1/create_status_changed_tensor/random_uniform/RandomUniform | |
| [[Node: create_next_particles/create_x_bar_x_previous_sample/create_room_min_scaled/_37 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_36_cr...min_scaled", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]] | |
| Current usage from device: /job:localhost/replica:0/task:0/device:GPU:0, allocator: GPU_0_bfc | |
| 285.34MiB from create_next_particles/create_y_bar_x_log_pdf/y_continuous_broadcast | |
| 71.33MiB from create_next_particles/create_y_bar_x_log_pdf/create_y_less_than_rssi_cutoff_boolean | |
| 12.59MiB from create_next_particles/create_x_bar_x_previous_sample/bernoulli_distribution_1/create_status_changed_tensor/random_uniform/RandomUniform | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment