Skip to content

Instantly share code, notes, and snippets.

@FlorinAndrei

FlorinAndrei/error.txt

Created December 27, 2023 21:31
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save FlorinAndrei/c4b9fed2e2bf63998e521526bb8b49e2 to your computer and use it in GitHub Desktop.
Save FlorinAndrei/c4b9fed2e2bf63998e521526bb8b49e2 to your computer and use it in GitHub Desktop.
Download ZIP
ray error
Raw
error.txt
2023-12-27 13:26:43,487 ERROR worker.py:405 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): The worker died unexpectedly while executing this task. Check python-core-worker-*.log files for more information.
2023-12-27 13:26:43,558 ERROR worker.py:405 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): The worker died unexpectedly while executing this task. Check python-core-worker-*.log files for more information.
2023-12-27 13:26:43,644 ERROR worker.py:405 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): The worker died unexpectedly while executing this task. Check python-core-worker-*.log files for more information.
2023-12-27 13:26:43,721 ERROR worker.py:405 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): The worker died unexpectedly while executing this task. Check python-core-worker-*.log files for more information.
(bundle_reservation_check_func pid=137883) Traceback (most recent call last):
(bundle_reservation_check_func pid=137883) File "python/ray/_raylet.pyx", line 1788, in ray._raylet.execute_task
(bundle_reservation_check_func pid=137883) File "python/ray/_raylet.pyx", line 1790, in ray._raylet.execute_task
(bundle_reservation_check_func pid=137883) File "/home/florin/.local/lib/python3.11/site-packages/ray/_private/worker.py", line 790, in deserialize_objects
(bundle_reservation_check_func pid=137883) context = self.get_serialization_context()
(bundle_reservation_check_func pid=137883) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=137883) File "/home/florin/.local/lib/python3.11/site-packages/ray/_private/worker.py", line 678, in get_serialization_context
(bundle_reservation_check_func pid=137883) context_map[job_id] = serialization.SerializationContext(self)
(bundle_reservation_check_func pid=137883) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=137883) File "/home/florin/.local/lib/python3.11/site-packages/ray/_private/serialization.py", line 153, in __init__
(bundle_reservation_check_func pid=137883) serialization_addons.apply(self)
(bundle_reservation_check_func pid=137883) File "/home/florin/.local/lib/python3.11/site-packages/ray/util/serialization_addons.py", line 29, in apply
(bundle_reservation_check_func pid=137883) from ray._private.pydantic_compat import register_pydantic_serializers
(bundle_reservation_check_func pid=137883) File "/home/florin/.local/lib/python3.11/site-packages/ray/_private/pydantic_compat.py", line 53, in <module>
(bundle_reservation_check_func pid=137883) from pydantic.v1 import (
(bundle_reservation_check_func pid=137883) File "/home/florin/.local/lib/python3.11/site-packages/pydantic/v1/__init__.py", line 2, in <module>
(bundle_reservation_check_func pid=137883) from . import dataclasses
(bundle_reservation_check_func pid=137883) File "/home/florin/.local/lib/python3.11/site-packages/pydantic/v1/dataclasses.py", line 45, in <module>
(bundle_reservation_check_func pid=137883) from .error_wrappers import ValidationError
(bundle_reservation_check_func pid=137883) File "/home/florin/.local/lib/python3.11/site-packages/pydantic/v1/error_wrappers.py", line 4, in <module>
(bundle_reservation_check_func pid=137883) from .json import pydantic_encoder
(bundle_reservation_check_func pid=137883) File "/home/florin/.local/lib/python3.11/site-packages/pydantic/v1/json.py", line 13, in <module>
(bundle_reservation_check_func pid=137883) from .networks import NameEmail
(bundle_reservation_check_func pid=137883) File "/home/florin/.local/lib/python3.11/site-packages/pydantic/v1/networks.py", line 70, in <module>
(bundle_reservation_check_func pid=137883) NetworkType = Union[str, bytes, int, Tuple[Union[str, bytes, int], Union[str, int]]]
(bundle_reservation_check_func pid=137883) ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=137883) File "/usr/lib/python3.11/typing.py", line 362, in inner
(bundle_reservation_check_func pid=137883) return cached(*args, **kwds)
(bundle_reservation_check_func pid=137883) ^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=137883) File "/home/florin/.local/lib/python3.11/site-packages/ray/_private/worker.py", line 841, in sigterm_handler
(bundle_reservation_check_func pid=137883) raise_sys_exit_with_custom_error_message(
(bundle_reservation_check_func pid=137883) File "python/ray/_raylet.pyx", line 846, in ray._raylet.raise_sys_exit_with_custom_error_message
(bundle_reservation_check_func pid=137883) SystemExit: 1
(bundle_reservation_check_func pid=137883)
(bundle_reservation_check_func pid=137883) During handling of the above exception, another exception occurred:
(bundle_reservation_check_func pid=137883)
(bundle_reservation_check_func pid=137883) Traceback (most recent call last):
(bundle_reservation_check_func pid=137883) File "python/ray/_raylet.pyx", line 2102, in ray._raylet.execute_task_with_cancellation_handler
(bundle_reservation_check_func pid=137883) File "python/ray/_raylet.pyx", line 1756, in ray._raylet.execute_task
(bundle_reservation_check_func pid=137883) File "python/ray/_raylet.pyx", line 1757, in ray._raylet.execute_task
(bundle_reservation_check_func pid=137883) File "python/ray/_raylet.pyx", line 1760, in ray._raylet.execute_task
(bundle_reservation_check_func pid=137883) File "python/ray/includes/libcoreworker.pxi", line 33, in ray._raylet.ProfileEvent.__exit__
(bundle_reservation_check_func pid=137883) File "/usr/lib/python3.11/traceback.py", line 184, in format_exc
(bundle_reservation_check_func pid=137883) return "".join(format_exception(*sys.exc_info(), limit=limit, chain=chain))
(bundle_reservation_check_func pid=137883) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=137883) File "/usr/lib/python3.11/traceback.py", line 139, in format_exception
(bundle_reservation_check_func pid=137883) te = TracebackException(type(value), value, tb, limit=limit, compact=True)
(bundle_reservation_check_func pid=137883) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=137883) File "/usr/lib/python3.11/traceback.py", line 728, in __init__
(bundle_reservation_check_func pid=137883) self.stack = StackSummary._extract_from_extended_frame_gen(
(bundle_reservation_check_func pid=137883) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=137883) File "/usr/lib/python3.11/traceback.py", line 433, in _extract_from_extended_frame_gen
(bundle_reservation_check_func pid=137883) f.line
(bundle_reservation_check_func pid=137883) File "/usr/lib/python3.11/traceback.py", line 318, in line
(bundle_reservation_check_func pid=137883) self._line = linecache.getline(self.filename, self.lineno)
(bundle_reservation_check_func pid=137883) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=137883) File "/usr/lib/python3.11/linecache.py", line 30, in getline
(bundle_reservation_check_func pid=137883) lines = getlines(filename, module_globals)
(bundle_reservation_check_func pid=137883) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=137883) File "/usr/lib/python3.11/linecache.py", line 46, in getlines
(bundle_reservation_check_func pid=137883) return updatecache(filename, module_globals)
(bundle_reservation_check_func pid=137883) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=137883) File "/usr/lib/python3.11/linecache.py", line 93, in updatecache
(bundle_reservation_check_func pid=137883) stat = os.stat(fullname)
(bundle_reservation_check_func pid=137883) ^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=137883) KeyboardInterrupt
(bundle_reservation_check_func pid=137883)
(bundle_reservation_check_func pid=137883) During handling of the above exception, another exception occurred:
(bundle_reservation_check_func pid=137883)
(bundle_reservation_check_func pid=137883) Traceback (most recent call last):
(bundle_reservation_check_func pid=137883) File "python/ray/_raylet.pyx", line 2206, in ray._raylet.task_execution_handler
(bundle_reservation_check_func pid=137883) File "python/ray/_raylet.pyx", line 2137, in ray._raylet.execute_task_with_cancellation_handler
(bundle_reservation_check_func pid=137883) File "python/ray/_raylet.pyx", line 1090, in ray._raylet.store_task_errors
(bundle_reservation_check_func pid=137883) File "/home/florin/.local/lib/python3.11/site-packages/ray/_private/utils.py", line 178, in push_error_to_driver
(bundle_reservation_check_func pid=137883) worker.core_worker.push_error(job_id, error_type, message, time.time())
(bundle_reservation_check_func pid=137883) File "python/ray/_raylet.pyx", line 4701, in ray._raylet.CoreWorker.push_error
(bundle_reservation_check_func pid=137883) File "python/ray/_raylet.pyx", line 589, in ray._raylet.check_status
(bundle_reservation_check_func pid=137883) ray.exceptions.RaySystemError: System error: Broken pipe
(bundle_reservation_check_func pid=137883)
(bundle_reservation_check_func pid=137883) During handling of the above exception, another exception occurred:
(bundle_reservation_check_func pid=137883)
(bundle_reservation_check_func pid=137883) Traceback (most recent call last):
(bundle_reservation_check_func pid=137883) File "python/ray/_raylet.pyx", line 2245, in ray._raylet.task_execution_handler
(bundle_reservation_check_func pid=137883) File "/home/florin/.local/lib/python3.11/site-packages/ray/_private/utils.py", line 178, in push_error_to_driver
(bundle_reservation_check_func pid=137883) worker.core_worker.push_error(job_id, error_type, message, time.time())
(bundle_reservation_check_func pid=137883) File "python/ray/_raylet.pyx", line 4701, in ray._raylet.CoreWorker.push_error
(bundle_reservation_check_func pid=137883) File "python/ray/_raylet.pyx", line 589, in ray._raylet.check_status
(bundle_reservation_check_func pid=137883) ray.exceptions.RaySystemError: System error: Broken pipe
(bundle_reservation_check_func pid=137883) Exception ignored in: 'ray._raylet.task_execution_handler'
(bundle_reservation_check_func pid=137883) Traceback (most recent call last):
(bundle_reservation_check_func pid=137883) File "python/ray/_raylet.pyx", line 2245, in ray._raylet.task_execution_handler
(bundle_reservation_check_func pid=137883) File "/home/florin/.local/lib/python3.11/site-packages/ray/_private/utils.py", line 178, in push_error_to_driver
(bundle_reservation_check_func pid=137883) worker.core_worker.push_error(job_id, error_type, message, time.time())
(bundle_reservation_check_func pid=137883) File "python/ray/_raylet.pyx", line 4701, in ray._raylet.CoreWorker.push_error
(bundle_reservation_check_func pid=137883) File "python/ray/_raylet.pyx", line 589, in ray._raylet.check_status
(bundle_reservation_check_func pid=137883) ray.exceptions.RaySystemError: System error: Broken pipe
2023-12-27 13:26:45,762 ERROR worker.py:405 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): The worker died unexpectedly while executing this task. Check python-core-worker-*.log files for more information.
2023-12-27 13:26:45,773 ERROR worker.py:405 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): The worker died unexpectedly while executing this task. Check python-core-worker-*.log files for more information.
2023-12-27 13:26:45,788 ERROR worker.py:405 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): The worker died unexpectedly while executing this task. Check python-core-worker-*.log files for more information.
2023-12-27 13:26:45,839 ERROR worker.py:405 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): The worker died unexpectedly while executing this task. Check python-core-worker-*.log files for more information.
2023-12-27 13:26:45,879 ERROR worker.py:405 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): The worker died unexpectedly while executing this task. Check python-core-worker-*.log files for more information.
2023-12-27 13:26:45,886 ERROR worker.py:405 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): The worker died unexpectedly while executing this task. Check python-core-worker-*.log files for more information.
2023-12-27 13:26:45,969 ERROR worker.py:405 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): The worker died unexpectedly while executing this task. Check python-core-worker-*.log files for more information.
(bundle_reservation_check_func pid=139958) Exception ignored in: <module 'collections.abc' from '/usr/lib/python3.11/collections/abc.py'>
(bundle_reservation_check_func pid=139958) File "<string>", line 0, in <module>
(bundle_reservation_check_func pid=139958) <frozen importlib._bootstrap>:241: RuntimeWarning: Cython module failed to patch module with custom type
(bundle_reservation_check_func pid=139927) from .validators import constr_length_validator, str_validator
(bundle_reservation_check_func pid=139927) ) -> Callable[[Any], Dict[str, Any]]:
(bundle_reservation_check_func pid=139927) ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=139927) File "/usr/lib/python3.11/typing.py", line 1637, in __getitem__
(bundle_reservation_check_func pid=139927) return self.__getitem_inner__(params)
(bundle_reservation_check_func pid=139927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=139927) File "/usr/lib/python3.11/typing.py", line 1650, in __getitem_inner__
(bundle_reservation_check_func pid=139927) return self.copy_with(params)
(bundle_reservation_check_func pid=139927) ^^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=139927) File "/usr/lib/python3.11/typing.py", line 1621, in copy_with
(bundle_reservation_check_func pid=139927) return _CallableGenericAlias(self.__origin__, params,
(bundle_reservation_check_func pid=139927) self.__args__ = tuple(... if a is _TypingEllipsis else
(bundle_reservation_check_func pid=139927) ^^^^^^^^^^^^^
(bundle_reservation_check_func pid=139927) File "/usr/lib/python3.11/typing.py", line 1307, in __setattr__
(bundle_reservation_check_func pid=139927) super().__setattr__(attr, val)
(bundle_reservation_check_func pid=139927) FileNotFoundError: [Errno 2] No such file or directory: 'python/ray/_raylet.pyx'
(bundle_reservation_check_func pid=139927) if lazycache(filename, module_globals):
(bundle_reservation_check_func pid=139927) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=139927) File "/usr/lib/python3.11/linecache.py", line 147, in lazycache
(bundle_reservation_check_func pid=139927) def lazycache(filename, module_globals):
2023-12-27 13:26:46,011 ERROR worker.py:405 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): The worker died unexpectedly while executing this task. Check python-core-worker-*.log files for more information.
2023-12-27 13:26:46,031 ERROR worker.py:405 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): The worker died unexpectedly while executing this task. Check python-core-worker-*.log files for more information.
(bundle_reservation_check_func pid=139958) File "/home/florin/.local/lib/python3.11/site-packages/ray/_private/utils.py", line 1852, in __exit__
(bundle_reservation_check_func pid=139958) raise KeyboardInterrupt
(bundle_reservation_check_func pid=140861) from pkg_resources import packaging
(bundle_reservation_check_func pid=140861) __import__('pkg_resources.extern.packaging.requirements')
(bundle_reservation_check_func pid=140861) from ._parser import parse_requirement as _parse_requirement
(bundle_reservation_check_func pid=140861) from ._tokenizer import DEFAULT_RULES, Tokenizer
(bundle_reservation_check_func pid=140861) "SPECIFIER": re.compile(
(bundle_reservation_check_func pid=140861) ^^^^^^^^^^^
(bundle_reservation_check_func pid=140861) File "/usr/lib/python3.11/re/__init__.py", line 227, in compile
(bundle_reservation_check_func pid=140861) return _compile(pattern, flags)
(bundle_reservation_check_func pid=140861) ^^^^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=140861) File "/usr/lib/python3.11/re/__init__.py", line 294, in _compile
(bundle_reservation_check_func pid=140861) p = _compiler.compile(pattern, flags)
(bundle_reservation_check_func pid=140861) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=140861) File "/usr/lib/python3.11/re/_compiler.py", line 745, in compile
(bundle_reservation_check_func pid=140861) p = _parser.parse(p, flags)
(bundle_reservation_check_func pid=140861) ^^^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=140861) File "/usr/lib/python3.11/re/_parser.py", line 989, in parse
(bundle_reservation_check_func pid=140861) p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
(bundle_reservation_check_func pid=140861) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=140861) File "/usr/lib/python3.11/re/_parser.py", line 464, in _parse_sub
(bundle_reservation_check_func pid=140861) itemsappend(_parse(source, state, verbose, nested + 1,
(bundle_reservation_check_func pid=140861) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=140861) File "/usr/lib/python3.11/re/_parser.py", line 872, in _parse
(bundle_reservation_check_func pid=140861) p = _parse_sub(source, state, sub_verbose, nested + 1)
(bundle_reservation_check_func pid=140861) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=140861) File "/usr/lib/python3.11/re/_parser.py", line 464, in _parse_sub
(bundle_reservation_check_func pid=140861) itemsappend(_parse(source, state, verbose, nested + 1,
(bundle_reservation_check_func pid=140861) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=140861) File "/usr/lib/python3.11/re/_parser.py", line 872, in _parse
(bundle_reservation_check_func pid=140861) p = _parse_sub(source, state, sub_verbose, nested + 1)
(bundle_reservation_check_func pid=140861) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=140861) File "/usr/lib/python3.11/re/_parser.py", line 464, in _parse_sub
(bundle_reservation_check_func pid=140861) itemsappend(_parse(source, state, verbose, nested + 1,
(bundle_reservation_check_func pid=140861) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=140861) File "/usr/lib/python3.11/re/_parser.py", line 872, in _parse
(bundle_reservation_check_func pid=140861) p = _parse_sub(source, state, sub_verbose, nested + 1)
(bundle_reservation_check_func pid=140861) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=140861) File "/usr/lib/python3.11/re/_parser.py", line 464, in _parse_sub
(bundle_reservation_check_func pid=140861) itemsappend(_parse(source, state, verbose, nested + 1,
(bundle_reservation_check_func pid=140861) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=140861) File "/usr/lib/python3.11/re/_parser.py", line 539, in _parse
(bundle_reservation_check_func pid=140861) continue
2023-12-27 13:26:47,920 ERROR worker.py:405 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): The worker died unexpectedly while executing this task. Check python-core-worker-*.log files for more information.
2023-12-27 13:26:48,006 ERROR worker.py:405 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): The worker died unexpectedly while executing this task. Check python-core-worker-*.log files for more information.
2023-12-27 13:26:48,141 ERROR worker.py:405 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): The worker died unexpectedly while executing this task. Check python-core-worker-*.log files for more information.
2023-12-27 13:26:48,505 ERROR worker.py:405 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): The worker died unexpectedly while executing this task. Check python-core-worker-*.log files for more information.
2023-12-27 13:26:50,456 INFO timeout.py:54 -- Reached timeout of 10 seconds. Stopping all trials.
2023-12-27 13:26:50,459 ERROR worker.py:405 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): The worker died unexpectedly while executing this task. Check python-core-worker-*.log files for more information.
2023-12-27 13:26:50,473 ERROR worker.py:405 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): The worker died unexpectedly while executing this task. Check python-core-worker-*.log files for more information.
2023-12-27 13:26:50,485 ERROR worker.py:405 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): The worker died unexpectedly while executing this task. Check python-core-worker-*.log files for more information.
(bundle_reservation_check_func pid=144218) File "/usr/lib/python3.11/contextlib.py", line 299, in helper
(bundle_reservation_check_func pid=144218) @wraps(func)
(bundle_reservation_check_func pid=144218) Traceback (most recent call last): [repeated 21x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/ray-logging.html#log-deduplication for more options.)
(bundle_reservation_check_func pid=144218) File "python/ray/_raylet.pyx", line 1756, in ray._raylet.execute_task [repeated 16x across cluster]
(bundle_reservation_check_func pid=140861) File "/home/florin/.local/lib/python3.11/site-packages/ray/_private/worker.py", line 790, in deserialize_objects [repeated 2x across cluster]
(bundle_reservation_check_func pid=140861) context = self.get_serialization_context() [repeated 2x across cluster]
(bundle_reservation_check_func pid=140861) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [repeated 2x across cluster]
(bundle_reservation_check_func pid=140861) File "/home/florin/.local/lib/python3.11/site-packages/ray/_private/worker.py", line 678, in get_serialization_context [repeated 2x across cluster]
(bundle_reservation_check_func pid=140861) context_map[job_id] = serialization.SerializationContext(self) [repeated 2x across cluster]
(bundle_reservation_check_func pid=140861) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [repeated 2x across cluster]
(bundle_reservation_check_func pid=144218) File "/usr/lib/python3.11/traceback.py", line 728, in __init__ [repeated 6x across cluster]
(bundle_reservation_check_func pid=140861) serialization_addons.apply(self) [repeated 2x across cluster]
(bundle_reservation_check_func pid=140861) File "/home/florin/.local/lib/python3.11/site-packages/ray/util/serialization_addons.py", line 29, in apply [repeated 2x across cluster]
(bundle_reservation_check_func pid=140861) from ray._private.pydantic_compat import register_pydantic_serializers [repeated 2x across cluster]
(bundle_reservation_check_func pid=140861) File "/home/florin/.local/lib/python3.11/site-packages/pkg_resources/_vendor/packaging/_tokenizer.py", line 74, in <module> [repeated 12x across cluster]
(bundle_reservation_check_func pid=139927) from pydantic.v1 import (
(bundle_reservation_check_func pid=139927) from . import dataclasses
(bundle_reservation_check_func pid=139927) from .error_wrappers import ValidationError
(bundle_reservation_check_func pid=139927) from .json import pydantic_encoder
(bundle_reservation_check_func pid=139927) from .networks import NameEmail
(bundle_reservation_check_func pid=139927) File "/usr/lib/python3.11/typing.py", line 362, in inner
(bundle_reservation_check_func pid=139927) return cached(*args, **kwds)
(bundle_reservation_check_func pid=139927) ^^^^^^^^^^^^^^^^^^^^^
(bundle_reservation_check_func pid=144218) File "/home/florin/.local/lib/python3.11/site-packages/ray/_private/worker.py", line 841, in sigterm_handler [repeated 4x across cluster]
(bundle_reservation_check_func pid=144218) raise_sys_exit_with_custom_error_message( [repeated 4x across cluster]
(bundle_reservation_check_func pid=144218) File "python/ray/_raylet.pyx", line 846, in ray._raylet.raise_sys_exit_with_custom_error_message [repeated 4x across cluster]
(bundle_reservation_check_func pid=144218) SystemExit: 1 [repeated 4x across cluster]
(bundle_reservation_check_func pid=144218)  [repeated 26x across cluster]
(bundle_reservation_check_func pid=144218) During handling of the above exception, another exception occurred: [repeated 12x across cluster]
(bundle_reservation_check_func pid=144218) File "python/ray/_raylet.pyx", line 2137, in ray._raylet.execute_task_with_cancellation_handler [repeated 8x across cluster]
(bundle_reservation_check_func pid=144218) File "python/ray/includes/libcoreworker.pxi", line 33, in ray._raylet.ProfileEvent.__exit__ [repeated 3x across cluster]
(bundle_reservation_check_func pid=144218) File "/usr/lib/python3.11/traceback.py", line 184, in format_exc [repeated 3x across cluster]
(bundle_reservation_check_func pid=144218) return "".join(format_exception(*sys.exc_info(), limit=limit, chain=chain)) [repeated 3x across cluster]
(bundle_reservation_check_func pid=144218) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [repeated 3x across cluster]
(bundle_reservation_check_func pid=144218) File "/usr/lib/python3.11/traceback.py", line 139, in format_exception [repeated 3x across cluster]
(bundle_reservation_check_func pid=144218) te = TracebackException(type(value), value, tb, limit=limit, compact=True) [repeated 3x across cluster]
(bundle_reservation_check_func pid=144218) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [repeated 3x across cluster]
(bundle_reservation_check_func pid=144218) self.stack = StackSummary._extract_from_extended_frame_gen( [repeated 3x across cluster]
(bundle_reservation_check_func pid=144218) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [repeated 4x across cluster]
(bundle_reservation_check_func pid=144218) File "/usr/lib/python3.11/traceback.py", line 433, in _extract_from_extended_frame_gen [repeated 3x across cluster]
(bundle_reservation_check_func pid=144218) f.line [repeated 3x across cluster]
(bundle_reservation_check_func pid=144218) File "/usr/lib/python3.11/traceback.py", line 318, in line [repeated 3x across cluster]
(bundle_reservation_check_func pid=144218) self._line = linecache.getline(self.filename, self.lineno) [repeated 3x across cluster]
(bundle_reservation_check_func pid=144218) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [repeated 3x across cluster]
(bundle_reservation_check_func pid=144218) File "/usr/lib/python3.11/linecache.py", line 30, in getline [repeated 3x across cluster]
(bundle_reservation_check_func pid=144218) lines = getlines(filename, module_globals) [repeated 3x across cluster]
(bundle_reservation_check_func pid=144218) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [repeated 3x across cluster]
(bundle_reservation_check_func pid=144218) File "/usr/lib/python3.11/linecache.py", line 46, in getlines [repeated 3x across cluster]
(bundle_reservation_check_func pid=144218) return updatecache(filename, module_globals) [repeated 3x across cluster]
(bundle_reservation_check_func pid=144218) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [repeated 3x across cluster]
(bundle_reservation_check_func pid=144218) File "/usr/lib/python3.11/linecache.py", line 93, in updatecache [repeated 4x across cluster]
(bundle_reservation_check_func pid=144218) stat = os.stat(fullname) [repeated 3x across cluster]
(bundle_reservation_check_func pid=144218) ^^^^^^^^^^^^^^^^^ [repeated 3x across cluster]
(bundle_reservation_check_func pid=144218) KeyboardInterrupt [repeated 4x across cluster]
(bundle_reservation_check_func pid=144218) File "python/ray/_raylet.pyx", line 2245, in ray._raylet.task_execution_handler [repeated 12x across cluster]
(bundle_reservation_check_func pid=144218) File "python/ray/_raylet.pyx", line 1090, in ray._raylet.store_task_errors [repeated 4x across cluster]
(bundle_reservation_check_func pid=144218) File "/home/florin/.local/lib/python3.11/site-packages/ray/_private/utils.py", line 178, in push_error_to_driver [repeated 12x across cluster]
(bundle_reservation_check_func pid=144218) worker.core_worker.push_error(job_id, error_type, message, time.time()) [repeated 12x across cluster]
(bundle_reservation_check_func pid=144218) File "python/ray/_raylet.pyx", line 4701, in ray._raylet.CoreWorker.push_error [repeated 12x across cluster]
(bundle_reservation_check_func pid=144218) File "python/ray/_raylet.pyx", line 589, in ray._raylet.check_status [repeated 12x across cluster]
(bundle_reservation_check_func pid=144218) ray.exceptions.RaySystemError: System error: Broken pipe [repeated 12x across cluster]
(bundle_reservation_check_func pid=144218) Exception ignored in: 'ray._raylet.task_execution_handler' [repeated 4x across cluster]
2023-12-27 13:26:51,675 WARNING experiment_analysis.py:193 -- Failed to fetch metrics for 8 trial(s):
- rastrigin_9efc5_01220: FileNotFoundError('Could not fetch metrics for rastrigin_9efc5_01220: both result.json and progress.csv were not found at /home/florin/ray_results/rastrigin_2023-12-27_13-26-36/rastrigin_9efc5_01220_1220_x=0.4530,y=-4.2148_2023-12-27_13-26-50')
- rastrigin_9efc5_01228: FileNotFoundError('Could not fetch metrics for rastrigin_9efc5_01228: both result.json and progress.csv were not found at /home/florin/ray_results/rastrigin_2023-12-27_13-26-36/rastrigin_9efc5_01228_1228_x=-1.6663,y=-3.2884_2023-12-27_13-26-50')
- rastrigin_9efc5_01230: FileNotFoundError('Could not fetch metrics for rastrigin_9efc5_01230: both result.json and progress.csv were not found at /home/florin/ray_results/rastrigin_2023-12-27_13-26-36/rastrigin_9efc5_01230_1230_x=-3.7400,y=-3.7037_2023-12-27_13-26-50')
- rastrigin_9efc5_01231: FileNotFoundError('Could not fetch metrics for rastrigin_9efc5_01231: both result.json and progress.csv were not found at /home/florin/ray_results/rastrigin_2023-12-27_13-26-36/rastrigin_9efc5_01231_1231_x=3.9602,y=2.8401_2023-12-27_13-26-50')
- rastrigin_9efc5_01232: FileNotFoundError('Could not fetch metrics for rastrigin_9efc5_01232: both result.json and progress.csv were not found at /home/florin/ray_results/rastrigin_2023-12-27_13-26-36/rastrigin_9efc5_01232_1232_x=3.4354,y=-0.2609_2023-12-27_13-26-50')
- rastrigin_9efc5_01233: FileNotFoundError('Could not fetch metrics for rastrigin_9efc5_01233: both result.json and progress.csv were not found at /home/florin/ray_results/rastrigin_2023-12-27_13-26-36/rastrigin_9efc5_01233_1233_x=-3.1780,y=-3.1743_2023-12-27_13-26-50')
- rastrigin_9efc5_01235: FileNotFoundError('Could not fetch metrics for rastrigin_9efc5_01235: both result.json and progress.csv were not found at /home/florin/ray_results/rastrigin_2023-12-27_13-26-36/rastrigin_9efc5_01235_1235_x=4.8021,y=2.0680_2023-12-27_13-26-50')
- rastrigin_9efc5_01237: FileNotFoundError('Could not fetch metrics for rastrigin_9efc5_01237: both result.json and progress.csv were not found at /home/florin/ray_results/rastrigin_2023-12-27_13-26-36/rastrigin_9efc5_01237_1237_x=-4.6047,y=-1.5357_2023-12-27_13-26-50')
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment