copy_tensor_bug.out

ssh://username@192.168.1.10:22/home/username/.local/share/virtualenvs/project-1BQtdZDZ/bin/python -u /data/personal/username/deployed/project/project/bin/copy_tensor_bug.py
2020-06-10 09:49:40.096725: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-06-10 09:49:40.178485: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-06-10 09:49:40.178972: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1080 computeCapability: 6.1
coreClock: 1.86GHz coreCount: 20 deviceMemorySize: 7.93GiB deviceMemoryBandwidth: 298.32GiB/s
2020-06-10 09:49:40.179163: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-06-10 09:49:40.180759: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-06-10 09:49:40.185827: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-06-10 09:49:40.186118: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-06-10 09:49:40.187808: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-06-10 09:49:40.188764: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-06-10 09:49:40.246402: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-06-10 09:49:40.246550: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-06-10 09:49:40.247104: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-06-10 09:49:40.247558: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-06-10 09:49:40.261827: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-06-10 09:49:40.293777: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 4200375000 Hz
2020-06-10 09:49:40.294134: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4668f40 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-06-10 09:49:40.294157: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-06-10 09:49:40.605893: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-06-10 09:49:40.614139: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x46f3320 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-06-10 09:49:40.614242: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce GTX 1080, Compute Capability 6.1
2020-06-10 09:49:40.614470: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-06-10 09:49:40.615041: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce GTX 1080 computeCapability: 6.1
coreClock: 1.86GHz coreCount: 20 deviceMemorySize: 7.93GiB deviceMemoryBandwidth: 298.32GiB/s
2020-06-10 09:49:40.615238: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-06-10 09:49:40.615311: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-06-10 09:49:40.615379: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-06-10 09:49:40.615443: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-06-10 09:49:40.615505: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-06-10 09:49:40.615567: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-06-10 09:49:40.615634: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-06-10 09:49:40.615747: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-06-10 09:49:40.622028: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-06-10 09:49:40.622517: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-06-10 09:49:40.622596: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-06-10 09:49:40.623510: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-06-10 09:49:40.623573: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2020-06-10 09:49:40.623626: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2020-06-10 09:49:40.623762: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-06-10 09:49:40.624291: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-06-10 09:49:40.624788: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2020-06-10 09:49:40.624862: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7524 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1)
Epoch 1/20
2020-06-10 09:49:43.485916: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-06-10 09:49:43.510722: W tensorflow/core/framework/op_kernel.cc:1741] Unknown: InternalError: Failed copying input tensor from /job:localhost/replica:0/task:0/device:CPU:0 to /job:localhost/replica:0/task:0/device:GPU:0 in order to run AddV2: Attempted to set tensor for existing mirror. [Op:AddV2]
Traceback (most recent call last):

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/ops/script_ops.py", line 241, in __call__
    return func(device, token, args)

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/ops/script_ops.py", line 130, in __call__
    ret = self._func(*args)

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py", line 309, in wrapper
    return func(*args, **kwargs)

  File "/data/personal/username/deployed/project/project/bin/copy_tensor_bug.py", line 12, in get_node
    idx = idx + 1

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py", line 997, in binary_op_wrapper
    return func(x, y, name=name)

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py", line 1276, in _add_dispatch
    return gen_math_ops.add_v2(x, y, name=name)

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/ops/gen_math_ops.py", line 480, in add_v2
    _ops.raise_from_not_ok_status(e, name)

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)

  File "<string>", line 3, in raise_from

tensorflow.python.framework.errors_impl.InternalError: Failed copying input tensor from /job:localhost/replica:0/task:0/device:CPU:0 to /job:localhost/replica:0/task:0/device:GPU:0 in order to run AddV2: Attempted to set tensor for existing mirror. [Op:AddV2]


2020-06-10 09:49:43.511044: W tensorflow/core/framework/op_kernel.cc:1741] Unknown: InternalError: Failed copying input tensor from /job:localhost/replica:0/task:0/device:CPU:0 to /job:localhost/replica:0/task:0/device:GPU:0 in order to run AddV2: Attempted to set tensor for existing mirror. [Op:AddV2]
Traceback (most recent call last):

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/ops/script_ops.py", line 241, in __call__
    return func(device, token, args)

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/ops/script_ops.py", line 130, in __call__
    ret = self._func(*args)

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py", line 309, in wrapper
    return func(*args, **kwargs)

  File "/data/personal/username/deployed/project/project/bin/copy_tensor_bug.py", line 12, in get_node
    idx = idx + 1

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py", line 997, in binary_op_wrapper
    return func(x, y, name=name)

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py", line 1276, in _add_dispatch
    return gen_math_ops.add_v2(x, y, name=name)

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/ops/gen_math_ops.py", line 480, in add_v2
    _ops.raise_from_not_ok_status(e, name)

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)

  File "<string>", line 3, in raise_from

tensorflow.python.framework.errors_impl.InternalError: Failed copying input tensor from /job:localhost/replica:0/task:0/device:CPU:0 to /job:localhost/replica:0/task:0/device:GPU:0 in order to run AddV2: Attempted to set tensor for existing mirror. [Op:AddV2]


2020-06-10 09:49:43.512470: W tensorflow/core/framework/op_kernel.cc:1741] Unknown: InternalError: Failed copying input tensor from /job:localhost/replica:0/task:0/device:CPU:0 to /job:localhost/replica:0/task:0/device:GPU:0 in order to run AddV2: Attempted to set tensor for existing mirror. [Op:AddV2]
Traceback (most recent call last):

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/ops/script_ops.py", line 241, in __call__
    return func(device, token, args)

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/ops/script_ops.py", line 130, in __call__
    ret = self._func(*args)

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py", line 309, in wrapper
    return func(*args, **kwargs)

  File "/data/personal/username/deployed/project/project/bin/copy_tensor_bug.py", line 12, in get_node
    idx = idx + 1

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py", line 997, in binary_op_wrapper
    return func(x, y, name=name)

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py", line 1276, in _add_dispatch
    return gen_math_ops.add_v2(x, y, name=name)

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/ops/gen_math_ops.py", line 480, in add_v2
    _ops.raise_from_not_ok_status(e, name)

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)

  File "<string>", line 3, in raise_from

tensorflow.python.framework.errors_impl.InternalError: Failed copying input tensor from /job:localhost/replica:0/task:0/device:CPU:0 to /job:localhost/replica:0/task:0/device:GPU:0 in order to run AddV2: Attempted to set tensor for existing mirror. [Op:AddV2]


Traceback (most recent call last):
  File "/data/personal/username/deployed/project/project/bin/copy_tensor_bug.py", line 41, in <module>
    model.fit(dataset, verbose=True, steps_per_epoch=1000 / 20, epochs=20)
  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 66, in _method_wrapper
    return method(self, *args, **kwargs)
  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 848, in fit
    tmp_logs = train_function(iterator)
  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 580, in __call__
    result = self._call(*args, **kwds)
  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 644, in _call
    return self._stateless_fn(*args, **kwds)
  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 2420, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1665, in _filtered_call
    self.captured_inputs)
  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1746, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 598, in call
    ctx=ctx)
  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.UnknownError:  InternalError: Failed copying input tensor from /job:localhost/replica:0/task:0/device:CPU:0 to /job:localhost/replica:0/task:0/device:GPU:0 in order to run AddV2: Attempted to set tensor for existing mirror. [Op:AddV2]
Traceback (most recent call last):

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/ops/script_ops.py", line 241, in __call__
    return func(device, token, args)

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/ops/script_ops.py", line 130, in __call__
    ret = self._func(*args)

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py", line 309, in wrapper
    return func(*args, **kwargs)

  File "/data/personal/username/deployed/project/project/bin/copy_tensor_bug.py", line 12, in get_node
    idx = idx + 1

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py", line 997, in binary_op_wrapper
    return func(x, y, name=name)

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py", line 1276, in _add_dispatch
    return gen_math_ops.add_v2(x, y, name=name)

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/ops/gen_math_ops.py", line 480, in add_v2
    _ops.raise_from_not_ok_status(e, name)

  File "/home/username/.local/share/virtualenvs/project-1BQtdZDZ/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)

  File "<string>", line 3, in raise_from

tensorflow.python.framework.errors_impl.InternalError: Failed copying input tensor from /job:localhost/replica:0/task:0/device:CPU:0 to /job:localhost/replica:0/task:0/device:GPU:0 in order to run AddV2: Attempted to set tensor for existing mirror. [Op:AddV2]


   [[{{node EagerPyFunc}}]]
   [[IteratorGetNext]] [Op:__inference_train_function_679]

Function call stack:
train_function


Process finished with exit code 1