2023-08-14-error.txt

error.txt

(base) root@dcc597189383:~/capsule# python src/main.py \
--mode train_image_segmentation \
--images /data/cedars-sinai-histopathology/ileum-image-regions \
--masks /data/cedars-sinai-histopathology/ileum-image-regions-masks-png \
--epochs 3 \
--sample_size 2 \
--batch_size 2 \
--buffer_size 2 \
--random_seed 2023 \
--learning_rate 0.08 \
--patch_size 1024 \
--plot_extension jpeg \
--results /results/port-to-code-ocean
2023-08-14 14:40:05.879601: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-08-14 14:40:05.992701: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-08-14 14:40:06.687101: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2023-08-14 14:40:06.687180: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
2023-08-14 14:40:06.687191: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/__init__.py:98: UserWarning: unable to load libtensorflow_io_plugins.so: unable to open file: libtensorflow_io_plugins.so, from paths: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io_plugins.so']
caused by: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io_plugins.so: undefined symbol: _ZN3tsl6StatusC1EN10tensorflow5error4CodeESt17basic_string_viewIcSt11char_traitsIcEENS_18SourceLocationImplE']
  warnings.warn(f"unable to load libtensorflow_io_plugins.so: {e}")
/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/__init__.py:104: UserWarning: file system plugins are not loaded: unable to open file: libtensorflow_io.so, from paths: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io.so']
caused by: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io.so: undefined symbol: _ZTVN3tsl13GcsFileSystemE']
  warnings.warn(f"file system plugins are not loaded: {e}")
2023-08-14 14:40:08 - INFO - logger:52 - Logging configuration was loaded. Log messages can be found at /results/port-to-code-ocean/log.txt.
2023-08-14 14:40:08.810830: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-08-14 14:40:08.852891: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-08-14 14:40:08.854858: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-08-14 14:40:08.856890: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-08-14 14:40:08.858011: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-08-14 14:40:08.859852: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-08-14 14:40:08.861629: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-08-14 14:40:09.640603: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-08-14 14:40:09.642526: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-08-14 14:40:09.644190: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-08-14 14:40:09.645756: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1613] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13595 MB memory:  -> device: 0, name: Tesla T4, pci bus id: 0000:00:1e.0, compute capability: 7.5
WARNING:tensorflow:AutoGraph could not transform <function decode_tiff at 0x7f30996ebe20> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: unable to open file: libtensorflow_io.so, from paths: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io.so']
caused by: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io.so: undefined symbol: _ZTVN3tsl13GcsFileSystemE']
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
2023-08-14 14:40:10 - WARNING - ag_logging:142 - AutoGraph could not transform <function decode_tiff at 0x7f30996ebe20> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: unable to open file: libtensorflow_io.so, from paths: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io.so']
caused by: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io.so: undefined symbol: _ZTVN3tsl13GcsFileSystemE']
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
2023-08-14 14:40:10 - ERROR - main:46 - in user code:

    File "/root/capsule/src/dataloader.py", line 408, in generated_load_image_function  *
        image = tfio.experimental.image.decode_tiff(image)
    File "/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/experimental/image_ops.py", line 87, in decode_tiff  **
        return core_ops.io_decode_tiff(contents, index, name=name)
    File "/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/__init__.py", line 88, in __getattr__
        return getattr(self._load(), attrb)
    File "/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/__init__.py", line 84, in _load
        self._mod = _load_library(self._library)
    File "/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/__init__.py", line 69, in _load_library
        raise NotImplementedError(

    NotImplementedError: unable to open file: libtensorflow_io.so, from paths: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io.so']
    caused by: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io.so: undefined symbol: _ZTVN3tsl13GcsFileSystemE']
Traceback (most recent call last):
  File "/root/capsule/src/main.py", line 38, in main
    train_image_segmentation(args)
  File "/root/capsule/src/train.py", line 43, in train_image_segmentation
    dataset_images = create_dataset_image_segmentation(
  File "/root/capsule/src/dataloader.py", line 51, in create_dataset_image_segmentation
    dataset = make_dataset_images(
  File "/root/capsule/src/dataloader.py", line 261, in make_dataset_images
    dataset = dataset.map(load_image_func)
  File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 2294, in map
    return MapDataset(self, map_func, preserve_cardinality=True, name=name)
  File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 5499, in __init__
    self._map_func = structured_function.StructuredFunctionWrapper(
  File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/data/ops/structured_function.py", line 263, in __init__
    self._function = fn_factory()
  File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compiler.py", line 226, in get_concrete_function
    concrete_function = self._get_concrete_function_garbage_collected(
  File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compiler.py", line 192, in _get_concrete_function_garbage_collected
    concrete_function, _ = self._maybe_define_concrete_function(args, kwargs)
  File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compiler.py", line 157, in _maybe_define_concrete_function
    return self._maybe_define_function(args, kwargs)
  File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compiler.py", line 360, in _maybe_define_function
    concrete_function = self._create_concrete_function(args, kwargs)
  File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compiler.py", line 284, in _create_concrete_function
    func_graph_module.func_graph_from_py_func(
  File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/framework/func_graph.py", line 1283, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/data/ops/structured_function.py", line 240, in wrapped_fn
    ret = wrapper_helper(*args)
  File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/data/ops/structured_function.py", line 171, in wrapper_helper
    ret = autograph.tf_convert(self._func, ag_ctx)(*nested_args)
  File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/autograph/impl/api.py", line 692, in wrapper
    raise e.ag_error_metadata.to_exception(e)
  File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/autograph/impl/api.py", line 689, in wrapper
    return converted_call(f, args, kwargs, options=options)
  File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/autograph/impl/api.py", line 439, in converted_call
    result = converted_f(*effective_args, **kwargs)
  File "/tmp/__autograph_generated_file8ombhe2l.py", line 48, in tf__generated_load_image_function
    ag__.if_stmt(ag__.ld(extension) == '.tif', if_body_1, else_body_1, get_state_1, set_state_1, ('image',), 1)
  File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1363, in if_stmt
    _py_if_stmt(cond, body, orelse)
  File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1416, in _py_if_stmt
    return body() if cond else orelse()
  File "/tmp/__autograph_generated_file8ombhe2l.py", line 24, in if_body_1
    image = ag__.converted_call(ag__.ld(tfio).experimental.image.decode_tiff, (ag__.ld(image),), None, fscope)
  File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/autograph/impl/api.py", line 434, in converted_call
    return _fall_back_unconverted(f, args, kwargs, options, e)
  File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/autograph/impl/api.py", line 484, in _fall_back_unconverted
    return _call_unconverted(f, args, kwargs, options)
  File "/opt/conda/lib/python3.10/site-packages/tensorflow/python/autograph/impl/api.py", line 459, in _call_unconverted
    return f(*args)
  File "/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/experimental/image_ops.py", line 87, in decode_tiff
    return core_ops.io_decode_tiff(contents, index, name=name)
  File "/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/__init__.py", line 88, in __getattr__
    return getattr(self._load(), attrb)
  File "/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/__init__.py", line 84, in _load
    self._mod = _load_library(self._library)
  File "/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/__init__.py", line 69, in _load_library
    raise NotImplementedError(
NotImplementedError: in user code:

    File "/root/capsule/src/dataloader.py", line 408, in generated_load_image_function  *
        image = tfio.experimental.image.decode_tiff(image)
    File "/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/experimental/image_ops.py", line 87, in decode_tiff  **
        return core_ops.io_decode_tiff(contents, index, name=name)
    File "/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/__init__.py", line 88, in __getattr__
        return getattr(self._load(), attrb)
    File "/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/__init__.py", line 84, in _load
        self._mod = _load_library(self._library)
    File "/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/__init__.py", line 69, in _load_library
        raise NotImplementedError(

    NotImplementedError: unable to open file: libtensorflow_io.so, from paths: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io.so']
    caused by: ['/opt/conda/lib/python3.10/site-packages/tensorflow_io/python/ops/libtensorflow_io.so: undefined symbol: _ZTVN3tsl13GcsFileSystemE']

2023-08-14 14:40:10 - INFO - main:51 - Completed mode train_image_segmentation in 2 sec
(base) root@dcc597189383:~/capsule#