superpoint_coco

[05/08/2019 00:00:47 INFO] Running command TRAIN
[05/08/2019 00:00:47 INFO] Number of GPUs detected: 1
/home/hadkevich/dev/mpcode/buck-out/gen/superpoint/experiment#link-tree/tensorflow/python/util/tf_inspect.py:75: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() or inspect.getfullargspec()
  return _inspect.getargspec(target)
2019-05-08 00:01:10.469880: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2
2019-05-08 00:01:10.747003: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:897] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-05-08 00:01:10.748162: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties: 
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:03:00.0
totalMemory: 10.92GiB freeMemory: 10.72GiB
2019-05-08 00:01:10.748187: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0
2019-05-08 00:01:11.078265: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-08 00:01:11.078310: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0 
2019-05-08 00:01:11.078320: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N 
2019-05-08 00:01:11.078651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10372 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:03:00.0, compute capability: 6.1)
2019-05-08 00:01:11.258621: I tensorflow/core/common_runtime/process_util.cc:69] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
[05/08/2019 00:01:13 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:13 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:14 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:14 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:14 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:14 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:14 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:14 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:14 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:14 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:14 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:14 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:14 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:14 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:14 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:14 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:14 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:14 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:14 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:14 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:14 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:15 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:15 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:15 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:17 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:17 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:17 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:17 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:17 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:18 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:18 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:18 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:18 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:18 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:18 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:18 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:18 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:18 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:18 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:18 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:18 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:18 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:18 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:18 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:18 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:18 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:18 INFO] Scale of 0 disables regularizer.
[05/08/2019 00:01:18 INFO] Scale of 0 disables regularizer.
2019-05-08 00:01:18.668129: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0
2019-05-08 00:01:18.668180: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-05-08 00:01:18.668189: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971]      0 
2019-05-08 00:01:18.668196: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0:   N 
2019-05-08 00:01:18.668349: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10372 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:03:00.0, compute capability: 6.1)
[05/08/2019 00:01:34 INFO] Start training
2019-05-08 00:01:40.006557: I tensorflow/core/kernels/cuda_solvers.cc:159] Creating CudaSolver handles for stream 0x1894d550
[05/08/2019 00:04:10 INFO] Iter    0: loss 27.4336, precision 0.0032, recall 0.0087
[05/08/2019 00:29:40 INFO] Iter 5000: loss 6.3321, precision 0.0130, recall 0.0370
[05/08/2019 00:54:44 INFO] Iter 10000: loss 4.4181, precision 0.0223, recall 0.0621
[05/08/2019 01:18:21 INFO] Iter 15000: loss 2.5462, precision 0.0294, recall 0.0815
[05/08/2019 01:42:02 INFO] Iter 20000: loss 1.8808, precision 0.0385, recall 0.0890
[05/08/2019 02:05:40 INFO] Iter 25000: loss 1.2884, precision 0.0485, recall 0.0935
[05/08/2019 02:28:53 INFO] Iter 30000: loss 1.3949, precision 0.0546, recall 0.1011
[05/08/2019 02:51:44 INFO] Iter 35000: loss 1.6000, precision 0.0699, recall 0.1033
[05/08/2019 03:14:33 INFO] Iter 40000: loss 1.1570, precision 0.0694, recall 0.1109
[05/08/2019 03:37:18 INFO] Iter 45000: loss 0.9989, precision 0.0765, recall 0.1218
[05/08/2019 04:00:07 INFO] Iter 50000: loss 1.5302, precision 0.0793, recall 0.1162
[05/08/2019 04:22:49 INFO] Iter 55000: loss 0.8852, precision 0.0890, recall 0.1353
[05/08/2019 04:45:34 INFO] Iter 60000: loss 1.1043, precision 0.0876, recall 0.1389
[05/08/2019 05:08:22 INFO] Iter 65000: loss 1.5485, precision 0.0812, recall 0.1379
[05/08/2019 05:31:03 INFO] Iter 70000: loss 0.9836, precision 0.0973, recall 0.1509
[05/08/2019 05:53:53 INFO] Iter 75000: loss 1.3899, precision 0.0979, recall 0.1483
[05/08/2019 06:16:32 INFO] Iter 80000: loss 1.1650, precision 0.1007, recall 0.1586
[05/08/2019 06:39:19 INFO] Iter 85000: loss 1.1841, precision 0.1067, recall 0.1624
[05/08/2019 07:02:03 INFO] Iter 90000: loss 0.9233, precision 0.1070, recall 0.1637
[05/08/2019 07:24:45 INFO] Iter 95000: loss 1.5249, precision 0.1031, recall 0.1694
[05/08/2019 07:47:31 INFO] Iter 100000: loss 1.1449, precision 0.1083, recall 0.1696
[05/08/2019 08:10:17 INFO] Iter 105000: loss 1.6313, precision 0.1131, recall 0.1749
[05/08/2019 08:33:05 INFO] Iter 110000: loss 1.3084, precision 0.1059, recall 0.1759
[05/08/2019 08:55:55 INFO] Iter 115000: loss 1.7265, precision 0.1078, recall 0.1768
[05/08/2019 09:18:45 INFO] Iter 120000: loss 1.0240, precision 0.1189, recall 0.1815
[05/08/2019 09:41:27 INFO] Iter 125000: loss 1.2907, precision 0.1122, recall 0.1848
[05/08/2019 10:04:10 INFO] Iter 130000: loss 1.4065, precision 0.1223, recall 0.1834
[05/08/2019 10:26:59 INFO] Iter 135000: loss 1.2103, precision 0.1155, recall 0.1853
[05/08/2019 10:49:50 INFO] Iter 140000: loss 1.0686, precision 0.1181, recall 0.1853
[05/08/2019 11:12:35 INFO] Iter 145000: loss 1.4973, precision 0.1215, recall 0.1890
[05/08/2019 11:35:21 INFO] Iter 150000: loss 0.9042, precision 0.1232, recall 0.1873
[05/08/2019 11:58:08 INFO] Iter 155000: loss 1.0151, precision 0.1280, recall 0.1899
[05/08/2019 12:20:56 INFO] Iter 160000: loss 1.2150, precision 0.1265, recall 0.1900
[05/08/2019 12:43:41 INFO] Iter 165000: loss 1.0042, precision 0.1255, recall 0.1890
[05/08/2019 13:06:39 INFO] Training finished
[05/08/2019 13:06:40 INFO] Saving checkpoint for iteration #170000