TF estimator log

gistfile1.txt

python cifar10_main.py --data-dir=${PWD}/cifar-10-data --job-dir=./models/cifar10 --variable-strategy GPU  --num-gpus=2 --sync | tee log
INFO:tensorflow:Using config: {'_model_dir': './models/cifar10', '_save_checkpoints_secs': 600, '_num_ps_replicas': 0, '_keep_checkpoint_max': 5, '_session_config': gpu_options {
  force_gpu_compatible: true
}
allow_soft_placement: true
, '_tf_random_seed': None, '_task_type': None, '_environment': 'local', '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f6b189b4dd0>, '_tf_config': gpu_options {
  per_process_gpu_memory_fraction: 1.0
}
, '_num_worker_replicas': 0, '_task_id': 0, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_evaluation_master': '', '_keep_checkpoint_every_n_hours': 10000, '_master': '', '_log_step_count_steps': 100}
WARNING:tensorflow:From /bodylabs/src/experimental/venv/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/monitors.py:269: __init__ (from tensorflow.contrib.learn.python.learn.monitors) is deprecated and will be removed after 2016-12-05.
Instructions for updating:
Monitors are deprecated. Please use tf.train.SessionRunHook.
INFO:tensorflow:image after unit resnet/tower_0/stage/residual_v1/: (64, 16, 32, 32)
INFO:tensorflow:image after unit resnet/tower_0/stage/residual_v1_1/: (64, 16, 32, 32)
INFO:tensorflow:image after unit resnet/tower_0/stage/residual_v1_2/: (64, 16, 32, 32)
INFO:tensorflow:image after unit resnet/tower_0/stage/residual_v1_3/: (64, 16, 32, 32)
INFO:tensorflow:image after unit resnet/tower_0/stage/residual_v1_4/: (64, 16, 32, 32)
INFO:tensorflow:image after unit resnet/tower_0/stage/residual_v1_5/: (64, 16, 32, 32)
INFO:tensorflow:image after unit resnet/tower_0/stage/residual_v1_6/: (64, 16, 32, 32)
INFO:tensorflow:image after unit resnet/tower_0/stage_1/residual_v1/avg_pool/: (64, 16, 16, 16)
INFO:tensorflow:image after unit resnet/tower_0/stage_1/residual_v1/: (64, 32, 16, 16)
INFO:tensorflow:image after unit resnet/tower_0/stage_1/residual_v1_1/: (64, 32, 16, 16)
INFO:tensorflow:image after unit resnet/tower_0/stage_1/residual_v1_2/: (64, 32, 16, 16)
INFO:tensorflow:image after unit resnet/tower_0/stage_1/residual_v1_3/: (64, 32, 16, 16)
INFO:tensorflow:image after unit resnet/tower_0/stage_1/residual_v1_4/: (64, 32, 16, 16)
INFO:tensorflow:image after unit resnet/tower_0/stage_1/residual_v1_5/: (64, 32, 16, 16)
INFO:tensorflow:image after unit resnet/tower_0/stage_1/residual_v1_6/: (64, 32, 16, 16)
INFO:tensorflow:image after unit resnet/tower_0/stage_2/residual_v1/avg_pool/: (64, 32, 8, 8)
INFO:tensorflow:image after unit resnet/tower_0/stage_2/residual_v1/: (64, 64, 8, 8)
INFO:tensorflow:image after unit resnet/tower_0/stage_2/residual_v1_1/: (64, 64, 8, 8)
INFO:tensorflow:image after unit resnet/tower_0/stage_2/residual_v1_2/: (64, 64, 8, 8)
INFO:tensorflow:image after unit resnet/tower_0/stage_2/residual_v1_3/: (64, 64, 8, 8)
INFO:tensorflow:image after unit resnet/tower_0/stage_2/residual_v1_4/: (64, 64, 8, 8)
INFO:tensorflow:image after unit resnet/tower_0/stage_2/residual_v1_5/: (64, 64, 8, 8)
INFO:tensorflow:image after unit resnet/tower_0/stage_2/residual_v1_6/: (64, 64, 8, 8)
INFO:tensorflow:image after unit resnet/tower_0/global_avg_pool/: (64, 64)
INFO:tensorflow:image after unit resnet/tower_0/fully_connected/: (64, 11)
INFO:tensorflow:image after unit resnet_1/tower_1/stage/residual_v1/: (64, 16, 32, 32)
INFO:tensorflow:image after unit resnet_1/tower_1/stage/residual_v1_1/: (64, 16, 32, 32)
INFO:tensorflow:image after unit resnet_1/tower_1/stage/residual_v1_2/: (64, 16, 32, 32)
INFO:tensorflow:image after unit resnet_1/tower_1/stage/residual_v1_3/: (64, 16, 32, 32)
INFO:tensorflow:image after unit resnet_1/tower_1/stage/residual_v1_4/: (64, 16, 32, 32)
INFO:tensorflow:image after unit resnet_1/tower_1/stage/residual_v1_5/: (64, 16, 32, 32)
INFO:tensorflow:image after unit resnet_1/tower_1/stage/residual_v1_6/: (64, 16, 32, 32)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_1/residual_v1/avg_pool/: (64, 16, 16, 16)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_1/residual_v1/: (64, 32, 16, 16)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_1/residual_v1_1/: (64, 32, 16, 16)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_1/residual_v1_2/: (64, 32, 16, 16)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_1/residual_v1_3/: (64, 32, 16, 16)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_1/residual_v1_4/: (64, 32, 16, 16)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_1/residual_v1_5/: (64, 32, 16, 16)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_1/residual_v1_6/: (64, 32, 16, 16)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_2/residual_v1/avg_pool/: (64, 32, 8, 8)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_2/residual_v1/: (64, 64, 8, 8)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_2/residual_v1_1/: (64, 64, 8, 8)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_2/residual_v1_2/: (64, 64, 8, 8)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_2/residual_v1_3/: (64, 64, 8, 8)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_2/residual_v1_4/: (64, 64, 8, 8)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_2/residual_v1_5/: (64, 64, 8, 8)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_2/residual_v1_6/: (64, 64, 8, 8)
INFO:tensorflow:image after unit resnet_1/tower_1/global_avg_pool/: (64, 64)
INFO:tensorflow:image after unit resnet_1/tower_1/fully_connected/: (64, 11)
INFO:tensorflow:SyncReplicasV2: replicas_to_aggregate=1; total_num_replicas=1
INFO:tensorflow:Create CheckpointSaverHook.
2017-09-17 16:30:12.919789: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-09-17 16:30:12.919808: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-09-17 16:30:12.919812: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-09-17 16:30:12.919815: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-09-17 16:30:12.919818: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2017-09-17 16:30:13.203777: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 0 with properties: 
name: TITAN X (Pascal)
major: 6 minor: 1 memoryClockRate (GHz) 1.531
pciBusID 0000:02:00.0
Total memory: 11.90GiB
Free memory: 11.33GiB
2017-09-17 16:30:13.485436: W tensorflow/stream_executor/cuda/cuda_driver.cc:523] A non-primary context 0x619e7f0 exists before initializing the StreamExecutor. We haven't verified StreamExecutor works with that.
2017-09-17 16:30:13.486261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 1 with properties: 
name: TITAN X (Pascal)
major: 6 minor: 1 memoryClockRate (GHz) 1.531
pciBusID 0000:03:00.0
Total memory: 11.90GiB
Free memory: 11.75GiB
2017-09-17 16:30:13.487000: I tensorflow/core/common_runtime/gpu/gpu_device.cc:976] DMA: 0 1 
2017-09-17 16:30:13.487012: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 0:   Y Y 
2017-09-17 16:30:13.487016: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 1:   Y Y 
2017-09-17 16:30:13.487024: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:02:00.0)
2017-09-17 16:30:13.487032: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:1) -> (device: 1, name: TITAN X (Pascal), pci bus id: 0000:03:00.0)
INFO:tensorflow:Saving checkpoints for 1 into ./models/cifar10/model.ckpt.
INFO:tensorflow:image after unit resnet/tower_0/stage/residual_v1/: (50, 16, 32, 32)
INFO:tensorflow:image after unit resnet/tower_0/stage/residual_v1_1/: (50, 16, 32, 32)
INFO:tensorflow:image after unit resnet/tower_0/stage/residual_v1_2/: (50, 16, 32, 32)
INFO:tensorflow:image after unit resnet/tower_0/stage/residual_v1_3/: (50, 16, 32, 32)
INFO:tensorflow:image after unit resnet/tower_0/stage/residual_v1_4/: (50, 16, 32, 32)
INFO:tensorflow:image after unit resnet/tower_0/stage/residual_v1_5/: (50, 16, 32, 32)
INFO:tensorflow:image after unit resnet/tower_0/stage/residual_v1_6/: (50, 16, 32, 32)
INFO:tensorflow:image after unit resnet/tower_0/stage_1/residual_v1/avg_pool/: (50, 16, 16, 16)
INFO:tensorflow:image after unit resnet/tower_0/stage_1/residual_v1/: (50, 32, 16, 16)
INFO:tensorflow:image after unit resnet/tower_0/stage_1/residual_v1_1/: (50, 32, 16, 16)
INFO:tensorflow:image after unit resnet/tower_0/stage_1/residual_v1_2/: (50, 32, 16, 16)
INFO:tensorflow:image after unit resnet/tower_0/stage_1/residual_v1_3/: (50, 32, 16, 16)
INFO:tensorflow:image after unit resnet/tower_0/stage_1/residual_v1_4/: (50, 32, 16, 16)
INFO:tensorflow:image after unit resnet/tower_0/stage_1/residual_v1_5/: (50, 32, 16, 16)
INFO:tensorflow:image after unit resnet/tower_0/stage_1/residual_v1_6/: (50, 32, 16, 16)
INFO:tensorflow:image after unit resnet/tower_0/stage_2/residual_v1/avg_pool/: (50, 32, 8, 8)
INFO:tensorflow:image after unit resnet/tower_0/stage_2/residual_v1/: (50, 64, 8, 8)
INFO:tensorflow:image after unit resnet/tower_0/stage_2/residual_v1_1/: (50, 64, 8, 8)
INFO:tensorflow:image after unit resnet/tower_0/stage_2/residual_v1_2/: (50, 64, 8, 8)
INFO:tensorflow:image after unit resnet/tower_0/stage_2/residual_v1_3/: (50, 64, 8, 8)
INFO:tensorflow:image after unit resnet/tower_0/stage_2/residual_v1_4/: (50, 64, 8, 8)
INFO:tensorflow:image after unit resnet/tower_0/stage_2/residual_v1_5/: (50, 64, 8, 8)
INFO:tensorflow:image after unit resnet/tower_0/stage_2/residual_v1_6/: (50, 64, 8, 8)
INFO:tensorflow:image after unit resnet/tower_0/global_avg_pool/: (50, 64)
INFO:tensorflow:image after unit resnet/tower_0/fully_connected/: (50, 11)
INFO:tensorflow:image after unit resnet_1/tower_1/stage/residual_v1/: (50, 16, 32, 32)
INFO:tensorflow:image after unit resnet_1/tower_1/stage/residual_v1_1/: (50, 16, 32, 32)
INFO:tensorflow:image after unit resnet_1/tower_1/stage/residual_v1_2/: (50, 16, 32, 32)
INFO:tensorflow:image after unit resnet_1/tower_1/stage/residual_v1_3/: (50, 16, 32, 32)
INFO:tensorflow:image after unit resnet_1/tower_1/stage/residual_v1_4/: (50, 16, 32, 32)
INFO:tensorflow:image after unit resnet_1/tower_1/stage/residual_v1_5/: (50, 16, 32, 32)
INFO:tensorflow:image after unit resnet_1/tower_1/stage/residual_v1_6/: (50, 16, 32, 32)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_1/residual_v1/avg_pool/: (50, 16, 16, 16)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_1/residual_v1/: (50, 32, 16, 16)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_1/residual_v1_1/: (50, 32, 16, 16)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_1/residual_v1_2/: (50, 32, 16, 16)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_1/residual_v1_3/: (50, 32, 16, 16)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_1/residual_v1_4/: (50, 32, 16, 16)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_1/residual_v1_5/: (50, 32, 16, 16)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_1/residual_v1_6/: (50, 32, 16, 16)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_2/residual_v1/avg_pool/: (50, 32, 8, 8)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_2/residual_v1/: (50, 64, 8, 8)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_2/residual_v1_1/: (50, 64, 8, 8)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_2/residual_v1_2/: (50, 64, 8, 8)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_2/residual_v1_3/: (50, 64, 8, 8)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_2/residual_v1_4/: (50, 64, 8, 8)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_2/residual_v1_5/: (50, 64, 8, 8)
INFO:tensorflow:image after unit resnet_1/tower_1/stage_2/residual_v1_6/: (50, 64, 8, 8)
INFO:tensorflow:image after unit resnet_1/tower_1/global_avg_pool/: (50, 64)
INFO:tensorflow:image after unit resnet_1/tower_1/fully_connected/: (50, 11)
INFO:tensorflow:SyncReplicasV2: replicas_to_aggregate=1; total_num_replicas=1
INFO:tensorflow:Starting evaluation at 2017-09-17-20:30:24
2017-09-17 16:30:25.765151: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: TITAN X (Pascal), pci bus id: 0000:02:00.0)
2017-09-17 16:30:25.765176: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:1) -> (device: 1, name: TITAN X (Pascal), pci bus id: 0000:03:00.0)
INFO:tensorflow:Restoring parameters from ./models/cifar10/model.ckpt-1
INFO:tensorflow:Evaluation [1/100]
INFO:tensorflow:Evaluation [2/100]
INFO:tensorflow:Evaluation [3/100]
INFO:tensorflow:Evaluation [4/100]
INFO:tensorflow:Evaluation [5/100]
INFO:tensorflow:Evaluation [6/100]
INFO:tensorflow:Evaluation [7/100]
INFO:tensorflow:Evaluation [8/100]
INFO:tensorflow:Evaluation [9/100]
INFO:tensorflow:Evaluation [10/100]
INFO:tensorflow:Evaluation [11/100]
INFO:tensorflow:Evaluation [12/100]
INFO:tensorflow:Evaluation [13/100]
INFO:tensorflow:Evaluation [14/100]
INFO:tensorflow:Evaluation [15/100]
INFO:tensorflow:Evaluation [16/100]
INFO:tensorflow:Evaluation [17/100]
INFO:tensorflow:Evaluation [18/100]
INFO:tensorflow:Evaluation [19/100]
INFO:tensorflow:Evaluation [20/100]
INFO:tensorflow:Evaluation [21/100]
INFO:tensorflow:Evaluation [22/100]
INFO:tensorflow:Evaluation [23/100]
INFO:tensorflow:Evaluation [24/100]
INFO:tensorflow:Evaluation [25/100]
INFO:tensorflow:Evaluation [26/100]
INFO:tensorflow:Evaluation [27/100]
INFO:tensorflow:Evaluation [28/100]
INFO:tensorflow:Evaluation [29/100]
INFO:tensorflow:Evaluation [30/100]
INFO:tensorflow:Evaluation [31/100]
INFO:tensorflow:Evaluation [32/100]
INFO:tensorflow:Evaluation [33/100]
INFO:tensorflow:Evaluation [34/100]
INFO:tensorflow:Evaluation [35/100]
INFO:tensorflow:Evaluation [36/100]
INFO:tensorflow:Evaluation [37/100]
INFO:tensorflow:Evaluation [38/100]
INFO:tensorflow:Evaluation [39/100]
INFO:tensorflow:Evaluation [40/100]
INFO:tensorflow:Evaluation [41/100]
INFO:tensorflow:Evaluation [42/100]
INFO:tensorflow:Evaluation [43/100]
INFO:tensorflow:Evaluation [44/100]
INFO:tensorflow:Evaluation [45/100]
INFO:tensorflow:Evaluation [46/100]
INFO:tensorflow:Evaluation [47/100]
INFO:tensorflow:Evaluation [48/100]
INFO:tensorflow:Evaluation [49/100]
INFO:tensorflow:Evaluation [50/100]
INFO:tensorflow:Evaluation [51/100]
INFO:tensorflow:Evaluation [52/100]
INFO:tensorflow:Evaluation [53/100]
INFO:tensorflow:Evaluation [54/100]
INFO:tensorflow:Evaluation [55/100]
INFO:tensorflow:Evaluation [56/100]
INFO:tensorflow:Evaluation [57/100]
INFO:tensorflow:Evaluation [58/100]
INFO:tensorflow:Evaluation [59/100]
INFO:tensorflow:Evaluation [60/100]
INFO:tensorflow:Evaluation [61/100]
INFO:tensorflow:Evaluation [62/100]
INFO:tensorflow:Evaluation [63/100]
INFO:tensorflow:Evaluation [64/100]
INFO:tensorflow:Evaluation [65/100]
INFO:tensorflow:Evaluation [66/100]
INFO:tensorflow:Evaluation [67/100]
INFO:tensorflow:Evaluation [68/100]
INFO:tensorflow:Evaluation [69/100]
INFO:tensorflow:Evaluation [70/100]
INFO:tensorflow:Evaluation [71/100]
INFO:tensorflow:Evaluation [72/100]
INFO:tensorflow:Evaluation [73/100]
INFO:tensorflow:Evaluation [74/100]
INFO:tensorflow:Evaluation [75/100]
INFO:tensorflow:Evaluation [76/100]
INFO:tensorflow:Evaluation [77/100]
INFO:tensorflow:Evaluation [78/100]
INFO:tensorflow:Evaluation [79/100]
INFO:tensorflow:Evaluation [80/100]
INFO:tensorflow:Evaluation [81/100]
INFO:tensorflow:Evaluation [82/100]
INFO:tensorflow:Evaluation [83/100]
INFO:tensorflow:Evaluation [84/100]
INFO:tensorflow:Evaluation [85/100]
INFO:tensorflow:Evaluation [86/100]
INFO:tensorflow:Evaluation [87/100]
INFO:tensorflow:Evaluation [88/100]
INFO:tensorflow:Evaluation [89/100]
INFO:tensorflow:Evaluation [90/100]
INFO:tensorflow:Evaluation [91/100]
INFO:tensorflow:Evaluation [92/100]
INFO:tensorflow:Evaluation [93/100]
INFO:tensorflow:Evaluation [94/100]
INFO:tensorflow:Evaluation [95/100]
INFO:tensorflow:Evaluation [96/100]
INFO:tensorflow:Evaluation [97/100]
INFO:tensorflow:Evaluation [98/100]
INFO:tensorflow:Evaluation [99/100]
INFO:tensorflow:Evaluation [100/100]
INFO:tensorflow:Finished evaluation at 2017-09-17-20:30:27
INFO:tensorflow:Saving dict for global step 1: accuracy = 0.1, global_step = 1, loss = 16.0981
ERROR:tensorflow:==================================
Object was never used (type <class 'tensorflow.python.framework.ops.Tensor'>):
<tf.Tensor 'report_uninitialized_variables/boolean_mask/Gather:0' shape=(?,) dtype=string>
If you want to mark it as used call its "mark_used()" method.
It was originally created here:
['File "cifar10_main.py", line 519, in <module>\n    main(**vars(args))', 'File "cifar10_main.py", line 387, in main\n    **hparams))', 'File "/bodylabs/src/experimental/venv/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/learn_runner.py", line 209, in run\n    return _execute_schedule(experiment, schedule)', 'File "/bodylabs/src/experimental/venv/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/learn_runner.py", line 46, in _execute_schedule\n    return task()', 'File "/bodylabs/src/experimental/venv/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 501, in train_and_evaluate\n    self.train(delay_secs=0)', 'File "/bodylabs/src/experimental/venv/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 280, in train\n    hooks=self._train_monitors + extra_hooks)', 'File "/bodylabs/src/experimental/venv/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/experiment.py", line 671, in _call_train\n    hooks=hooks)', 'File "/bodylabs/src/experimental/venv/local/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 241, in train\n    loss = self._train_model(input_fn=input_fn, hooks=hooks)', 'File "/bodylabs/src/experimental/venv/local/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 686, in _train_model\n    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])', 'File "/bodylabs/src/experimental/venv/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 518, in run\n    run_metadata=run_metadata)', 'File "/bodylabs/src/experimental/venv/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 862, in run\n    run_metadata=run_metadata)', 'File "/bodylabs/src/experimental/venv/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 818, in run\n    return self._sess.run(*args, **kwargs)', 'File "/bodylabs/src/experimental/venv/local/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 980, in run\n    run_metadata=run_metadata))', 'File "/bodylabs/src/experimental/venv/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/monitors.py", line 1198, in after_run\n    induce_stop = m.step_end(self._last_step, result)', 'File "/bodylabs/src/experimental/venv/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/monitors.py", line 358, in step_end\n    return self.every_n_step_end(step, output)', 'File "/bodylabs/src/experimental/venv/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/monitors.py", line 696, in every_n_step_end\n    validation_outputs = self._evaluate_estimator()', 'File "/bodylabs/src/experimental/venv/local/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/monitors.py", line 667, in _evaluate_estimator\n    name=self.name)', 'File "/bodylabs/src/experimental/venv/local/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 292, in evaluate\n    name=name)', 'File "/bodylabs/src/experimental/venv/local/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 713, in _evaluate_model\n    features, labels, model_fn_lib.ModeKeys.EVAL)', 'File "/bodylabs/src/experimental/venv/local/lib/python2.7/site-packages/tensorflow/python/estimator/estimator.py", line 615, in _call_model_fn\n    model_fn_results = self._model_fn(features=features, **kwargs)', 'File "cifar10_main.py", line 181, in _resnet_model_fn\n    gradvars, global_step=tf.train.get_global_step())', 'File "/bodylabs/src/experimental/venv/local/lib/python2.7/site-packages/tensorflow/python/training/sync_replicas_optimizer.py", line 257, in apply_gradients\n    variables.global_variables())', 'File "/bodylabs/src/experimental/venv/local/lib/python2.7/site-packages/tensorflow/python/util/tf_should_use.py", line 175, in wrapped\n    return _add_should_use_warning(fn(*args, **kwargs))', 'File "/bodylabs/src/experimental/venv/local/lib/python2.7/site-packages/tensorflow/python/util/tf_should_use.py", line 144, in _add_should_use_warning\n    wrapped = TFShouldUseWarningWrapper(x)', 'File "/bodylabs/src/experimental/venv/local/lib/python2.7/site-packages/tensorflow/python/util/tf_should_use.py", line 101, in __init__\n    stack = [s.strip() for s in traceback.format_stack()]']
==================================
INFO:tensorflow:Validation (step 1): loss = 16.0981, global_step = 1, accuracy = 0.1
INFO:tensorflow:loss = 3.79624, step = 1
INFO:tensorflow:loss = 3.79624, learning_rate = 0.1
DEBUG:tensorflow:Skipping evaluation due to same checkpoint ./models/cifar10/model.ckpt-1 for step 2 as for step 1.