Created
August 7, 2023 23:08
-
-
Save erikr/1f27e903aab7161c9cd70478477fde98 to your computer and use it in GitHub Desktop.
2023-08-07-error.txt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
(base) root@d0f6d976e858:~/capsule# bash train.sh | |
2023-08-07 23:06:18.187369: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA | |
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. | |
2023-08-07 23:06:18.301916: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. | |
2023-08-07 23:06:18.988996: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 | |
2023-08-07 23:06:18.989054: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 | |
2023-08-07 23:06:18.989065: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. | |
/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/__init__.py:98: UserWarning: unable to load libtensorflow_io_plugins.so: unable to open file: libtensorflow_io_plugins.so, from paths: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io_plugins.so'] | |
caused by: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io_plugins.so: undefined symbol: _ZN3tsl6StatusC1EN10tensorflow5error4CodeESt17basic_string_viewIcSt11char_traitsIcEENS_18SourceLocationImplE'] | |
warnings.warn(f"unable to load libtensorflow_io_plugins.so: {e}") | |
/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/__init__.py:104: UserWarning: file system plugins are not loaded: unable to open file: libtensorflow_io.so, from paths: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io.so'] | |
caused by: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io.so: undefined symbol: _ZTVN3tsl13GcsFileSystemE'] | |
warnings.warn(f"file system plugins are not loaded: {e}") | |
2023-08-07 23:06:20 - INFO - logger:52 - Logging configuration was loaded. Log messages can be found at /results/port-to-code-ocean/log.txt. | |
2023-08-07 23:06:21.436538: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero | |
2023-08-07 23:06:21.477284: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero | |
2023-08-07 23:06:21.481801: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero | |
2023-08-07 23:06:21.483769: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA | |
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. | |
2023-08-07 23:06:21.484880: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero | |
2023-08-07 23:06:21.486630: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero | |
2023-08-07 23:06:21.488306: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero | |
2023-08-07 23:06:22.258559: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero | |
2023-08-07 23:06:22.260178: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero | |
2023-08-07 23:06:22.261557: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero | |
2023-08-07 23:06:22.262858: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13582 MB memory: -> device: 0, name: Tesla T4, pci bus id: 0000:00:1e.0, compute capability: 7.5 | |
WARNING:tensorflow:AutoGraph could not transform <function decode_tiff at 0x7fa3aaadbe20> and will run it as-is. | |
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. | |
Cause: unable to open file: libtensorflow_io.so, from paths: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io.so'] | |
caused by: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io.so: undefined symbol: _ZTVN3tsl13GcsFileSystemE'] | |
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert | |
2023-08-07 23:06:22 - WARNING - ag_logging:142 - AutoGraph could not transform <function decode_tiff at 0x7fa3aaadbe20> and will run it as-is. | |
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. | |
Cause: unable to open file: libtensorflow_io.so, from paths: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io.so'] | |
caused by: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io.so: undefined symbol: _ZTVN3tsl13GcsFileSystemE'] | |
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert | |
2023-08-07 23:06:22 - ERROR - main:46 - in user code: | |
File "/root/capsule/src/dataloader.py", line 408, in generated_load_image_function * | |
image = tfio.experimental.image.decode_tiff(image) | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/experimental/image_ops.py", line 87, in decode_tiff ** | |
return core_ops.io_decode_tiff(contents, index, name=name) | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/__init__.py", line 88, in __getattr__ | |
return getattr(self._load(), attrb) | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/__init__.py", line 84, in _load | |
self._mod = _load_library(self._library) | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/__init__.py", line 69, in _load_library | |
raise NotImplementedError( | |
NotImplementedError: unable to open file: libtensorflow_io.so, from paths: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io.so'] | |
caused by: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io.so: undefined symbol: _ZTVN3tsl13GcsFileSystemE'] | |
Traceback (most recent call last): | |
File "/root/capsule/src/main.py", line 38, in main | |
train_image_segmentation(args) | |
File "/root/capsule/src/train.py", line 43, in train_image_segmentation | |
dataset_images = create_dataset_image_segmentation( | |
File "/root/capsule/src/dataloader.py", line 51, in create_dataset_image_segmentation | |
dataset = make_dataset_images( | |
File "/root/capsule/src/dataloader.py", line 261, in make_dataset_images | |
dataset = dataset.map(load_image_func) | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 2294, in map | |
return MapDataset(self, map_func, preserve_cardinality=True, name=name) | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 5499, in __init__ | |
self._map_func = structured_function.StructuredFunctionWrapper( | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/data/ops/structured_function.py", line 263, in __init__ | |
self._function = fn_factory() | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compiler.py", line 226, in get_concrete_function | |
concrete_function = self._get_concrete_function_garbage_collected( | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compiler.py", line 192, in _get_concrete_function_garbage_collected | |
concrete_function, _ = self._maybe_define_concrete_function(args, kwargs) | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compiler.py", line 157, in _maybe_define_concrete_function | |
return self._maybe_define_function(args, kwargs) | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compiler.py", line 360, in _maybe_define_function | |
concrete_function = self._create_concrete_function(args, kwargs) | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compiler.py", line 284, in _create_concrete_function | |
func_graph_module.func_graph_from_py_func( | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/framework/func_graph.py", line 1283, in func_graph_from_py_func | |
func_outputs = python_func(*func_args, **func_kwargs) | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/data/ops/structured_function.py", line 240, in wrapped_fn | |
ret = wrapper_helper(*args) | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/data/ops/structured_function.py", line 171, in wrapper_helper | |
ret = autograph.tf_convert(self._func, ag_ctx)(*nested_args) | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/autograph/impl/api.py", line 692, in wrapper | |
raise e.ag_error_metadata.to_exception(e) | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/autograph/impl/api.py", line 689, in wrapper | |
return converted_call(f, args, kwargs, options=options) | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/autograph/impl/api.py", line 439, in converted_call | |
result = converted_f(*effective_args, **kwargs) | |
File "/tmp/__autograph_generated_filedzvipift.py", line 48, in tf__generated_load_image_function | |
ag__.if_stmt(ag__.ld(extension) == '.tif', if_body_1, else_body_1, get_state_1, set_state_1, ('image',), 1) | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1363, in if_stmt | |
_py_if_stmt(cond, body, orelse) | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1416, in _py_if_stmt | |
return body() if cond else orelse() | |
File "/tmp/__autograph_generated_filedzvipift.py", line 24, in if_body_1 | |
image = ag__.converted_call(ag__.ld(tfio).experimental.image.decode_tiff, (ag__.ld(image),), None, fscope) | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/autograph/impl/api.py", line 434, in converted_call | |
return _fall_back_unconverted(f, args, kwargs, options, e) | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/autograph/impl/api.py", line 484, in _fall_back_unconverted | |
return _call_unconverted(f, args, kwargs, options) | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/autograph/impl/api.py", line 459, in _call_unconverted | |
return f(*args) | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/experimental/image_ops.py", line 87, in decode_tiff | |
return core_ops.io_decode_tiff(contents, index, name=name) | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/__init__.py", line 88, in __getattr__ | |
return getattr(self._load(), attrb) | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/__init__.py", line 84, in _load | |
self._mod = _load_library(self._library) | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/__init__.py", line 69, in _load_library | |
raise NotImplementedError( | |
NotImplementedError: in user code: | |
File "/root/capsule/src/dataloader.py", line 408, in generated_load_image_function * | |
image = tfio.experimental.image.decode_tiff(image) | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/experimental/image_ops.py", line 87, in decode_tiff ** | |
return core_ops.io_decode_tiff(contents, index, name=name) | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/__init__.py", line 88, in __getattr__ | |
return getattr(self._load(), attrb) | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/__init__.py", line 84, in _load | |
self._mod = _load_library(self._library) | |
File "/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/__init__.py", line 69, in _load_library | |
raise NotImplementedError( | |
NotImplementedError: unable to open file: libtensorflow_io.so, from paths: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io.so'] | |
caused by: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io.so: undefined symbol: _ZTVN3tsl13GcsFileSystemE'] | |
2023-08-07 23:06:22 - INFO - main:51 - Completed mode train_image_segmentation in 2 sec |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment