manish-kumar-garg/saver_restore.txt

## saver_restore.txt
/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
setup_tf_thread_pools() not yet called (via func get_tf_list_local_devices), calling it now.
Setup TF inter and intra global thread pools, num_threads None, session opts {'log_device_placement': False, 'device_count': {'GPU': 0}}.
2020-01-30 05:44:51.199090: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-01-30 05:44:51.806210: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-01-30 05:44:51.832949: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-01-30 05:44:51.839893: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-01-30 05:44:51.849408: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-01-30 05:44:51.850999: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55acca6007f0 executing computations on platform CUDA. Devices:
2020-01-30 05:44:51.851031: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): Tesla V100-SXM2-16GB, Compute Capability 7.0
2020-01-30 05:44:51.851041: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (1): Tesla V100-SXM2-16GB, Compute Capability 7.0
2020-01-30 05:44:51.851053: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (2): Tesla V100-SXM2-16GB, Compute Capability 7.0
2020-01-30 05:44:51.851066: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (3): Tesla V100-SXM2-16GB, Compute Capability 7.0
2020-01-30 05:44:51.872145: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2300080000 Hz
2020-01-30 05:44:51.874042: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55accacc4840 executing computations on platform Host. Devices:
2020-01-30 05:44:51.874072: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
2020-01-30 05:44:51.874170: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-01-30 05:44:51.874188: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]
Collecting TensorFlow device list...
2020-01-30 05:44:51.877328: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:00:1b.0
totalMemory: 15.78GiB freeMemory: 15.47GiB
2020-01-30 05:44:51.877391: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 1 with properties:
name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:00:1c.0
totalMemory: 15.78GiB freeMemory: 15.47GiB
2020-01-30 05:44:51.877440: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 2 with properties:
name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:00:1d.0
totalMemory: 15.78GiB freeMemory: 15.47GiB
2020-01-30 05:44:51.877485: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 3 with properties:
name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
pciBusID: 0000:00:1e.0
totalMemory: 15.78GiB freeMemory: 15.47GiB
2020-01-30 05:44:51.877538: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0, 1, 2, 3
2020-01-30 05:44:51.882975: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-01-30 05:44:51.883001: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 1 2 3
2020-01-30 05:44:51.883016: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N Y Y Y
2020-01-30 05:44:51.883025: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1:   Y N Y Y
2020-01-30 05:44:51.883032: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2:   Y Y N Y
2020-01-30 05:44:51.883040: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3:   Y Y Y N
2020-01-30 05:44:51.883188: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:0 with 15049 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1b.0, compute capability: 7.0)
2020-01-30 05:44:51.883487: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:1 with 15049 MB memory) -> physical GPU (device: 1, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1c.0, compute capability: 7.0)
2020-01-30 05:44:51.883779: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:2 with 15049 MB memory) -> physical GPU (device: 2, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1d.0, compute capability: 7.0)
2020-01-30 05:44:51.884047: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:3 with 15049 MB memory) -> physical GPU (device: 3, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
2020-01-30 05:44:53.104532: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0, 1, 2, 3
2020-01-30 05:44:53.104617: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-01-30 05:44:53.104635: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 1 2 3
2020-01-30 05:44:53.104649: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N Y Y Y
2020-01-30 05:44:53.104657: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1:   Y N Y Y
2020-01-30 05:44:53.104665: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2:   Y Y N Y
2020-01-30 05:44:53.104672: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3:   Y Y Y N
2020-01-30 05:44:53.104837: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15049 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1b.0, compute capability: 7.0)
2020-01-30 05:44:53.105031: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 15049 MB memory) -> physical GPU (device: 1, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1c.0, compute capability: 7.0)
2020-01-30 05:44:53.105228: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 15049 MB memory) -> physical GPU (device: 2, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1d.0, compute capability: 7.0)
2020-01-30 05:44:53.105436: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 15049 MB memory) -> physical GPU (device: 3, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
WARNING:tensorflow:From /home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
Traceback (most recent call last):
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [10025] rhs shape= [10026]
	 [[{{node save/Assign_1}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1276, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [10025] rhs shape= [10026]
	 [[node save/Assign_1 (defined at freeze_graph.py:15) ]]

Caused by op 'save/Assign_1', defined at:
  File "freeze_graph.py", line 15, in <module>
    saver = tf.train.import_meta_graph(input_checkpoint + '.meta',clear_devices=clear_devices)
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1435, in import_meta_graph
    meta_graph_or_file, clear_devices, import_scope, **kwargs)[0]
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1457, in _import_meta_graph_with_return_elements
    **kwargs))
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/meta_graph.py", line 806, in import_scoped_meta_graph_with_return_elements
    return_elements=return_elements)
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 442, in import_graph_def
    _ProcessNewOps(graph)
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 235, in _ProcessNewOps
    for new_op in graph._add_new_tf_operations(compute_devices=False):  # pylint: disable=protected-access
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3433, in _add_new_tf_operations
    for c_op in c_api_util.new_tf_operations(self)
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3433, in <listcomp>
    for c_op in c_api_util.new_tf_operations(self)
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3325, in _create_op_from_tf_operation
    ret = Operation(c_op, self)
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__
    self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [10025] rhs shape= [10026]
	 [[node save/Assign_1 (defined at freeze_graph.py:15) ]]


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "freeze_graph.py", line 20, in <module>
    saver.restore(sess, input_checkpoint)
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1312, in restore
    err, "a mismatch between the current graph and the graph")
tensorflow.python.framework.errors_impl.InvalidArgumentError: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Assign requires shapes of both tensors to match. lhs shape= [10025] rhs shape= [10026]
	 [[node save/Assign_1 (defined at freeze_graph.py:15) ]]

Caused by op 'save/Assign_1', defined at:
  File "freeze_graph.py", line 15, in <module>
    saver = tf.train.import_meta_graph(input_checkpoint + '.meta',clear_devices=clear_devices)
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1435, in import_meta_graph
    meta_graph_or_file, clear_devices, import_scope, **kwargs)[0]
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1457, in _import_meta_graph_with_return_elements
    **kwargs))
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/meta_graph.py", line 806, in import_scoped_meta_graph_with_return_elements
    return_elements=return_elements)
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 442, in import_graph_def
    _ProcessNewOps(graph)
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 235, in _ProcessNewOps
    for new_op in graph._add_new_tf_operations(compute_devices=False):  # pylint: disable=protected-access
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3433, in _add_new_tf_operations
    for c_op in c_api_util.new_tf_operations(self)
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3433, in <listcomp>
    for c_op in c_api_util.new_tf_operations(self)
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3325, in _create_op_from_tf_operation
    ret = Operation(c_op, self)
  File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__
    self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Assign requires shapes of both tensors to match. lhs shape= [10025] rhs shape= [10026]
	 [[node save/Assign_1 (defined at freeze_graph.py:15) ]]
	/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
	_np_qint8 = np.dtype([("qint8", np.int8, 1)])
	/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
	_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
	/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
	_np_qint16 = np.dtype([("qint16", np.int16, 1)])
	/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
	_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
	/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
	_np_qint32 = np.dtype([("qint32", np.int32, 1)])
	/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
	np_resource = np.dtype([("resource", np.ubyte, 1)])
	setup_tf_thread_pools() not yet called (via func get_tf_list_local_devices), calling it now.
	Setup TF inter and intra global thread pools, num_threads None, session opts {'log_device_placement': False, 'device_count': {'GPU': 0}}.
	2020-01-30 05:44:51.199090: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
	2020-01-30 05:44:51.806210: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
	2020-01-30 05:44:51.832949: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
	2020-01-30 05:44:51.839893: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
	2020-01-30 05:44:51.849408: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
	2020-01-30 05:44:51.850999: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55acca6007f0 executing computations on platform CUDA. Devices:
	2020-01-30 05:44:51.851031: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): Tesla V100-SXM2-16GB, Compute Capability 7.0
	2020-01-30 05:44:51.851041: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (1): Tesla V100-SXM2-16GB, Compute Capability 7.0
	2020-01-30 05:44:51.851053: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (2): Tesla V100-SXM2-16GB, Compute Capability 7.0
	2020-01-30 05:44:51.851066: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (3): Tesla V100-SXM2-16GB, Compute Capability 7.0
	2020-01-30 05:44:51.872145: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2300080000 Hz
	2020-01-30 05:44:51.874042: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55accacc4840 executing computations on platform Host. Devices:
	2020-01-30 05:44:51.874072: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
	2020-01-30 05:44:51.874170: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
	2020-01-30 05:44:51.874188: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]
	Collecting TensorFlow device list...
	2020-01-30 05:44:51.877328: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
	name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
	pciBusID: 0000:00:1b.0
	totalMemory: 15.78GiB freeMemory: 15.47GiB
	2020-01-30 05:44:51.877391: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 1 with properties:
	name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
	pciBusID: 0000:00:1c.0
	totalMemory: 15.78GiB freeMemory: 15.47GiB
	2020-01-30 05:44:51.877440: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 2 with properties:
	name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
	pciBusID: 0000:00:1d.0
	totalMemory: 15.78GiB freeMemory: 15.47GiB
	2020-01-30 05:44:51.877485: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 3 with properties:
	name: Tesla V100-SXM2-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.53
	pciBusID: 0000:00:1e.0
	totalMemory: 15.78GiB freeMemory: 15.47GiB
	2020-01-30 05:44:51.877538: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0, 1, 2, 3
	2020-01-30 05:44:51.882975: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
	2020-01-30 05:44:51.883001: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0 1 2 3
	2020-01-30 05:44:51.883016: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N Y Y Y
	2020-01-30 05:44:51.883025: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: Y N Y Y
	2020-01-30 05:44:51.883032: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: Y Y N Y
	2020-01-30 05:44:51.883040: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: Y Y Y N
	2020-01-30 05:44:51.883188: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:0 with 15049 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1b.0, compute capability: 7.0)
	2020-01-30 05:44:51.883487: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:1 with 15049 MB memory) -> physical GPU (device: 1, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1c.0, compute capability: 7.0)
	2020-01-30 05:44:51.883779: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:2 with 15049 MB memory) -> physical GPU (device: 2, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1d.0, compute capability: 7.0)
	2020-01-30 05:44:51.884047: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/device:GPU:3 with 15049 MB memory) -> physical GPU (device: 3, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
	2020-01-30 05:44:53.104532: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0, 1, 2, 3
	2020-01-30 05:44:53.104617: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
	2020-01-30 05:44:53.104635: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0 1 2 3
	2020-01-30 05:44:53.104649: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N Y Y Y
	2020-01-30 05:44:53.104657: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 1: Y N Y Y
	2020-01-30 05:44:53.104665: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 2: Y Y N Y
	2020-01-30 05:44:53.104672: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 3: Y Y Y N
	2020-01-30 05:44:53.104837: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15049 MB memory) -> physical GPU (device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1b.0, compute capability: 7.0)
	2020-01-30 05:44:53.105031: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 15049 MB memory) -> physical GPU (device: 1, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1c.0, compute capability: 7.0)
	2020-01-30 05:44:53.105228: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 15049 MB memory) -> physical GPU (device: 2, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1d.0, compute capability: 7.0)
	2020-01-30 05:44:53.105436: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 15049 MB memory) -> physical GPU (device: 3, name: Tesla V100-SXM2-16GB, pci bus id: 0000:00:1e.0, compute capability: 7.0)
	WARNING:tensorflow:From /home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
	Instructions for updating:
	Use standard file APIs to check for files with this prefix.
	Traceback (most recent call last):
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
	return fn(*args)
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
	options, feed_dict, fetch_list, target_list, run_metadata)
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
	run_metadata)
	tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [10025] rhs shape= [10026]
	[[{{node save/Assign_1}}]]

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1276, in restore
	{self.saver_def.filename_tensor_name: save_path})
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
	run_metadata_ptr)
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
	feed_dict_tensor, options, run_metadata)
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
	run_metadata)
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
	raise type(e)(node_def, op, message)
	tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [10025] rhs shape= [10026]
	[[node save/Assign_1 (defined at freeze_graph.py:15) ]]

	Caused by op 'save/Assign_1', defined at:
	File "freeze_graph.py", line 15, in <module>
	saver = tf.train.import_meta_graph(input_checkpoint + '.meta',clear_devices=clear_devices)
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1435, in import_meta_graph
	meta_graph_or_file, clear_devices, import_scope, **kwargs)[0]
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1457, in _import_meta_graph_with_return_elements
	**kwargs))
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/meta_graph.py", line 806, in import_scoped_meta_graph_with_return_elements
	return_elements=return_elements)
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
	return func(args, *kwargs)
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 442, in import_graph_def
	_ProcessNewOps(graph)
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 235, in _ProcessNewOps
	for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3433, in _add_new_tf_operations
	for c_op in c_api_util.new_tf_operations(self)
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3433, in <listcomp>
	for c_op in c_api_util.new_tf_operations(self)
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3325, in _create_op_from_tf_operation
	ret = Operation(c_op, self)
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__
	self._traceback = tf_stack.extract_stack()

	InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [10025] rhs shape= [10026]
	[[node save/Assign_1 (defined at freeze_graph.py:15) ]]


	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	File "freeze_graph.py", line 20, in <module>
	saver.restore(sess, input_checkpoint)
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1312, in restore
	err, "a mismatch between the current graph and the graph")
	tensorflow.python.framework.errors_impl.InvalidArgumentError: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

	Assign requires shapes of both tensors to match. lhs shape= [10025] rhs shape= [10026]
	[[node save/Assign_1 (defined at freeze_graph.py:15) ]]

	Caused by op 'save/Assign_1', defined at:
	File "freeze_graph.py", line 15, in <module>
	saver = tf.train.import_meta_graph(input_checkpoint + '.meta',clear_devices=clear_devices)
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1435, in import_meta_graph
	meta_graph_or_file, clear_devices, import_scope, **kwargs)[0]
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1457, in _import_meta_graph_with_return_elements
	**kwargs))
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/meta_graph.py", line 806, in import_scoped_meta_graph_with_return_elements
	return_elements=return_elements)
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
	return func(args, *kwargs)
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 442, in import_graph_def
	_ProcessNewOps(graph)
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/importer.py", line 235, in _ProcessNewOps
	for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3433, in _add_new_tf_operations
	for c_op in c_api_util.new_tf_operations(self)
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3433, in <listcomp>
	for c_op in c_api_util.new_tf_operations(self)
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3325, in _create_op_from_tf_operation
	ret = Operation(c_op, self)
	File "/home/ubuntu/tf1.13/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__
	self._traceback = tf_stack.extract_stack()

	InvalidArgumentError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

	Assign requires shapes of both tensors to match. lhs shape= [10025] rhs shape= [10026]
	[[node save/Assign_1 (defined at freeze_graph.py:15) ]]