Skip to content

Instantly share code, notes, and snippets.

@xwjiang2010

xwjiang2010/gist:ee1c4bb129c12a04979616bfd9a18c44

Created August 5, 2022 17:27
Show Gist options
  • Save xwjiang2010/ee1c4bb129c12a04979616bfd9a18c44 to your computer and use it in GitHub Desktop.
Save xwjiang2010/ee1c4bb129c12a04979616bfd9a18c44 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
(base) root@70b3d47b1c72:/ray# pytest -s python/ray/tune/tests/test_cluster.py
Test session starts (platform: linux, Python 3.7.9, pytest 7.0.1, pytest-sugar 0.9.5)
rootdir: /ray/python
plugins: anyio-3.6.1, asyncio-0.16.0, docker-tools-3.1.3, forked-1.4.0, lazy-fixture-0.6.3, rerunfailures-10.2, shutil-1.7.0, sugar-0.9.5, timeout-2.1.0, virtualenv-1.7.0, remotedata-0.3.2, typeguard-2.13.3
collecting ... 2022-08-05 10:19:12,277 INFO worker.py:1312 -- Connecting to existing Ray cluster at address: 172.18.0.3:64540...
2022-08-05 10:19:12,282 INFO worker.py:1487 -- Connected to Ray cluster. View the dashboard at http://127.0.0.1:8265.
2022-08-05 10:19:12,299 INFO cluster_utils.py:162 -- RayContext(dashboard_url='127.0.0.1:8265', python_version='3.7.9', ray_version='2.0.0rc0', ray_commit='{{RAY_COMMIT_SHA}}', address_info={'node_ip_address': '172.18.0.3', 'raylet_ip_address': '172.18.0.3', 'redis_address': None, 'object_store_address': '/tmp/ray/session_2022-08-05_10-19-10_016654_12497/sockets/plasma_store', 'raylet_socket_name': '/tmp/ray/session_2022-08-05_10-19-10_016654_12497/sockets/raylet', 'webui_url': '127.0.0.1:8265', 'session_dir': '/tmp/ray/session_2022-08-05_10-19-10_016654_12497', 'metrics_export_port': 65221, 'gcs_address': '172.18.0.3:64540', 'address': '172.18.0.3:64540', 'dashboard_agent_listen_port': 52365, 'node_id': '4632103a1fb0c4481a7de148079081ee2da3982042fc534330d3c834'})
(pid=12693) /opt/miniconda/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
(pid=12693) import imp
(pid=12693) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:23: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
(pid=12693) 'nearest': pil_image.NEAREST,
(pid=12693) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:24: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
(pid=12693) 'bilinear': pil_image.BILINEAR,
(pid=12693) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:25: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
(pid=12693) 'bicubic': pil_image.BICUBIC,
(pid=12693) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:28: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.
(pid=12693) if hasattr(pil_image, 'HAMMING'):
(pid=12693) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:30: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.
(pid=12693) if hasattr(pil_image, 'BOX'):
(pid=12693) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:33: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
(pid=12693) if hasattr(pil_image, 'LANCZOS'):
(_MockTrainer pid=12693) 2022-08-05 10:19:21,746 WARNING util.py:65 -- Install gputil for GPU system monitoring.
2022-08-05 10:19:22,853 WARNING worker.py:1799 -- Raylet is terminated: ip=172.18.0.3, id=9671f6805c80b0b752c0f1cfca10b936014126708c02f3d86b4b612c. Termination is unexpected. Possible reasons include: (1) SIGKILL by the user or system OOM killer, (2) Invalid memory access from Raylet causing SIGSEGV or SIGBUS, (3) Other termination signals. Last 20 lines of the Raylet logs:
[state-dump] RayletWorkerPool.deadline_timer.kill_idle_workers - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
[state-dump] InternalPubSubGcsService.grpc_client.GcsSubscriberPoll - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
[state-dump] DebugString() time ms: 0
[state-dump]
[state-dump]
[2022-08-05 10:19:12,183 I 12561 12561] (raylet) accessor.cc:608: Received notification for node id = 4632103a1fb0c4481a7de148079081ee2da3982042fc534330d3c834, IsAlive = 1
[2022-08-05 10:19:12,286 I 12561 12561] (raylet) node_manager.cc:599: New job has started. Job id 01000000 Driver pid 12497 is dead: 0 driver address: 172.18.0.3
[2022-08-05 10:19:12,286 I 12561 12561] (raylet) worker_pool.cc:636: Job 01000000 already started in worker pool.
[2022-08-05 10:19:12,291 I 12561 12585] (raylet) object_store.cc:35: Object store current usage 8e-09 / 0.157286 GB.
[2022-08-05 10:19:13,301 I 12561 12561] (raylet) agent_manager.cc:40: HandleRegisterAgent, ip: 172.18.0.3, port: 43530, id: 424238335
[2022-08-05 10:19:15,747 I 12561 12561] (raylet) worker_pool.cc:447: Started worker process with pid 12667, the token is 0
[2022-08-05 10:19:16,734 I 12561 12561] (raylet) worker_pool.cc:447: Started worker process with pid 12693, the token is 1
[2022-08-05 10:19:18,582 I 12561 12561] (raylet) node_manager.cc:1429: NodeManager::DisconnectClient, disconnect_type=1, has creation task exception = 0
[2022-08-05 10:19:21,788 I 12561 12561] (raylet) accessor.cc:608: Received notification for node id = 9671f6805c80b0b752c0f1cfca10b936014126708c02f3d86b4b612c, IsAlive = 1
[2022-08-05 10:19:21,982 I 12561 12561] (raylet) accessor.cc:608: Received notification for node id = 9671f6805c80b0b752c0f1cfca10b936014126708c02f3d86b4b612c, IsAlive = 0
[2022-08-05 10:19:22,042 I 12561 12561] (raylet) accessor.cc:608: Received notification for node id = 8b89994286dd5aa089bad2f9c2035b928a3dfadeb70c8f83ee072b7b, IsAlive = 1
[2022-08-05 10:19:22,168 I 12561 12561] (raylet) accessor.cc:608: Received notification for node id = 2f0e0fa48664f42e49cb6cc75712fb61faccda6060412118a7816930, IsAlive = 1
[2022-08-05 10:19:22,303 I 12561 12561] (raylet) accessor.cc:608: Received notification for node id = f7ce221ce3dc8d0673df7b14633c9b96ba640d599d88562d20ef7457, IsAlive = 1
[2022-08-05 10:19:22,441 I 12561 12561] (raylet) accessor.cc:608: Received notification for node id = 8fd6d924a59f5b6b27de61d6b59b6fd9030ce1c51eab8cafb5dc04c9, IsAlive = 1
[2022-08-05 10:19:22,567 I 12561 12561] (raylet) accessor.cc:608: Received notification for node id = d9b93020c9c10da52425517f78e996c84d34a2c617400fc7a60d276a, IsAlive = 1
ray/tune/tests/test_cluster.py ✓ 8% ▊ 2022-08-05 10:19:33,110 INFO worker.py:1312 -- Connecting to existing Ray cluster at address: 172.18.0.3:48759...
2022-08-05 10:19:33,115 INFO worker.py:1487 -- Connected to Ray cluster. View the dashboard at http://127.0.0.1:8265.
2022-08-05 10:19:33,136 INFO cluster_utils.py:162 -- RayContext(dashboard_url='127.0.0.1:8265', python_version='3.7.9', ray_version='2.0.0rc0', ray_commit='{{RAY_COMMIT_SHA}}', address_info={'node_ip_address': '172.18.0.3', 'raylet_ip_address': '172.18.0.3', 'redis_address': None, 'object_store_address': '/tmp/ray/session_2022-08-05_10-19-30_806453_12497/sockets/plasma_store', 'raylet_socket_name': '/tmp/ray/session_2022-08-05_10-19-30_806453_12497/sockets/raylet', 'webui_url': '127.0.0.1:8265', 'session_dir': '/tmp/ray/session_2022-08-05_10-19-30_806453_12497', 'metrics_export_port': 55910, 'gcs_address': '172.18.0.3:48759', 'address': '172.18.0.3:48759', 'dashboard_agent_listen_port': 52365, 'node_id': '74856010aedc5c541aef788ca4f80ff99e221e0eee92984a650d3849'})
(pid=13275) /opt/miniconda/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
(pid=13275) import imp
(pid=13275) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:23: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
(pid=13275) 'nearest': pil_image.NEAREST,
(pid=13275) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:24: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
(pid=13275) 'bilinear': pil_image.BILINEAR,
(pid=13275) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:25: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
(pid=13275) 'bicubic': pil_image.BICUBIC,
(pid=13275) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:28: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.
(pid=13275) if hasattr(pil_image, 'HAMMING'):
(pid=13275) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:30: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.
(pid=13275) if hasattr(pil_image, 'BOX'):
(pid=13275) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:33: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
(pid=13275) if hasattr(pil_image, 'LANCZOS'):
(_MockTrainer pid=13275) 2022-08-05 10:19:39,408 WARNING util.py:65 -- Install gputil for GPU system monitoring.
2022-08-05 10:19:39,629 ERROR trial_runner.py:980 -- Trial __fake_c5db72e2: Error processing event.
ray.tune.error._TuneNoNextExecutorEventError: Traceback (most recent call last):
File "/ray/python/ray/tune/execution/ray_trial_executor.py", line 989, in get_next_executor_event
future_result = ray.get(ready_future)
File "/ray/python/ray/_private/client_mode_hook.py", line 105, in wrapper
return func(*args, **kwargs)
File "/ray/python/ray/_private/worker.py", line 2247, in get
raise value
ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.
class_name: _MockTrainer
actor_id: fd6b2acd8d5dd90a07c4448901000000
pid: 13275
namespace: 8afd4549-8050-4bea-9471-abebc3fb8482
ip: 172.18.0.3
The actor is dead because its node has died. Node Id: 2cb95d9b9ef3761bdfee4e7915bd3fa33e14ca22df47c31f3e39c083
ray/tune/tests/test_cluster.py ✓✓ 15% █▋ 2022-08-05 10:19:44,498 INFO worker.py:1312 -- Connecting to existing Ray cluster at address: 172.18.0.3:58142...
2022-08-05 10:19:44,503 INFO worker.py:1487 -- Connected to Ray cluster. View the dashboard at http://127.0.0.1:8265.
2022-08-05 10:19:44,522 INFO cluster_utils.py:162 -- RayContext(dashboard_url='127.0.0.1:8265', python_version='3.7.9', ray_version='2.0.0rc0', ray_commit='{{RAY_COMMIT_SHA}}', address_info={'node_ip_address': '172.18.0.3', 'raylet_ip_address': '172.18.0.3', 'redis_address': None, 'object_store_address': '/tmp/ray/session_2022-08-05_10-19-42_195581_12497/sockets/plasma_store', 'raylet_socket_name': '/tmp/ray/session_2022-08-05_10-19-42_195581_12497/sockets/raylet', 'webui_url': '127.0.0.1:8265', 'session_dir': '/tmp/ray/session_2022-08-05_10-19-42_195581_12497', 'metrics_export_port': 62159, 'gcs_address': '172.18.0.3:58142', 'address': '172.18.0.3:58142', 'dashboard_agent_listen_port': 52365, 'node_id': '84f7ea887a6e3398b60ff9e510e548bdeae35d98b8186ccaa1397638'})
(pid=13541) /opt/miniconda/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
(pid=13541) import imp
(pid=13541) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:23: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
(pid=13541) 'nearest': pil_image.NEAREST,
(pid=13541) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:24: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
(pid=13541) 'bilinear': pil_image.BILINEAR,
(pid=13541) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:25: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
(pid=13541) 'bicubic': pil_image.BICUBIC,
(pid=13541) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:28: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.
(pid=13541) if hasattr(pil_image, 'HAMMING'):
(pid=13541) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:30: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.
(pid=13541) if hasattr(pil_image, 'BOX'):
(pid=13541) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:33: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
(pid=13541) if hasattr(pil_image, 'LANCZOS'):
(_MockTrainer pid=13541) 2022-08-05 10:19:50,796 WARNING util.py:65 -- Install gputil for GPU system monitoring.
2022-08-05 10:19:51,096 ERROR trial_runner.py:980 -- Trial __fake_cca51aba: Error processing event.
ray.tune.error._TuneNoNextExecutorEventError: Traceback (most recent call last):
File "/ray/python/ray/tune/execution/ray_trial_executor.py", line 989, in get_next_executor_event
future_result = ray.get(ready_future)
File "/ray/python/ray/_private/client_mode_hook.py", line 105, in wrapper
return func(*args, **kwargs)
File "/ray/python/ray/_private/worker.py", line 2247, in get
raise value
ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.
class_name: _MockTrainer
actor_id: f99bcd0e3db9829b0eda691e01000000
pid: 13541
namespace: c5e40118-8e4f-4345-bad9-452604697145
ip: 172.18.0.3
The actor is dead because its node has died. Node Id: 621c0a0c28c318c5d2f07d2af44e68bfe00fd96f46b5f318585c3d70
2022-08-05 10:19:51,098 INFO trial_runner.py:1323 -- Trial __fake_cca51aba: Attempting to restore trial state from last checkpoint.
2022-08-05 10:19:51,102 ERROR ray_trial_executor.py:104 -- An exception occurred when trying to stop the Ray actor:Traceback (most recent call last):
File "/ray/python/ray/tune/execution/ray_trial_executor.py", line 94, in _post_stop_cleanup
ray.get(future, timeout=0)
File "/ray/python/ray/_private/client_mode_hook.py", line 105, in wrapper
return func(*args, **kwargs)
File "/ray/python/ray/_private/worker.py", line 2247, in get
raise value
ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.
class_name: _MockTrainer
actor_id: f99bcd0e3db9829b0eda691e01000000
pid: 13541
namespace: c5e40118-8e4f-4345-bad9-452604697145
ip: 172.18.0.3
The actor is dead because its node has died. Node Id: 621c0a0c28c318c5d2f07d2af44e68bfe00fd96f46b5f318585c3d70
2022-08-05 10:19:51,479 WARNING worker.py:1799 -- Raylet is terminated: ip=172.18.0.3, id=621c0a0c28c318c5d2f07d2af44e68bfe00fd96f46b5f318585c3d70. Termination is unexpected. Possible reasons include: (1) SIGKILL by the user or system OOM killer, (2) Invalid memory access from Raylet causing SIGSEGV or SIGBUS, (3) Other termination signals. Last 20 lines of the Raylet logs:
[state-dump] NodeManager.deadline_timer.flush_free_objects - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
[state-dump] NodeManager.deadline_timer.record_metrics - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
[state-dump] InternalPubSubGcsService.grpc_client.GcsSubscriberCommandBatch - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
[state-dump] InternalPubSubGcsService.grpc_client.GcsSubscriberPoll - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
[state-dump] RayletWorkerPool.deadline_timer.kill_idle_workers - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
[state-dump] NodeInfoGcsService.grpc_client.GetInternalConfig - 1 total (0 active), CPU time: mean = 10.781 ms, total = 10.781 ms
[state-dump] NodeManager.deadline_timer.debug_state_dump - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
[state-dump] NodeManagerService.grpc_server.RequestResourceReport - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
[state-dump] NodeInfoGcsService.grpc_client.RegisterNode - 1 total (0 active), CPU time: mean = 435.950 us, total = 435.950 us
[state-dump] DebugString() time ms: 0
[state-dump]
[state-dump]
[2022-08-05 10:19:44,395 I 13385 13385] (raylet) accessor.cc:608: Received notification for node id = 84f7ea887a6e3398b60ff9e510e548bdeae35d98b8186ccaa1397638, IsAlive = 1
[2022-08-05 10:19:44,508 I 13385 13385] (raylet) node_manager.cc:599: New job has started. Job id 01000000 Driver pid 12497 is dead: 0 driver address: 172.18.0.3
[2022-08-05 10:19:44,508 I 13385 13385] (raylet) worker_pool.cc:636: Job 01000000 already started in worker pool.
[2022-08-05 10:19:44,514 I 13385 13402] (raylet) object_store.cc:35: Object store current usage 8e-09 / 0.157286 GB.
[2022-08-05 10:19:44,561 I 13385 13385] (raylet) accessor.cc:608: Received notification for node id = 621c0a0c28c318c5d2f07d2af44e68bfe00fd96f46b5f318585c3d70, IsAlive = 1
[2022-08-05 10:19:45,539 I 13385 13385] (raylet) agent_manager.cc:40: HandleRegisterAgent, ip: 172.18.0.3, port: 63549, id: 424238335
[2022-08-05 10:19:50,959 I 13385 13385] (raylet) accessor.cc:608: Received notification for node id = 621c0a0c28c318c5d2f07d2af44e68bfe00fd96f46b5f318585c3d70, IsAlive = 0
[2022-08-05 10:19:51,003 I 13385 13385] (raylet) accessor.cc:608: Received notification for node id = 462b1fe2222b03a1466e723e6acdcb2d5adb9822ed52d6b13dd51168, IsAlive = 1
(pid=13656) /opt/miniconda/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
(pid=13656) import imp
(pid=13656) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:23: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
(pid=13656) 'nearest': pil_image.NEAREST,
(pid=13656) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:24: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
(pid=13656) 'bilinear': pil_image.BILINEAR,
(pid=13656) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:25: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
(pid=13656) 'bicubic': pil_image.BICUBIC,
(pid=13656) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:28: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.
(pid=13656) if hasattr(pil_image, 'HAMMING'):
(pid=13656) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:30: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.
(pid=13656) if hasattr(pil_image, 'BOX'):
(pid=13656) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:33: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
(pid=13656) if hasattr(pil_image, 'LANCZOS'):
ray/tune/tests/test_cluster.py ✓✓✓ 23% ██▍ 2022-08-05 10:20:01,982 INFO worker.py:1312 -- Connecting to existing Ray cluster at address: 172.18.0.3:62294...
2022-08-05 10:20:01,987 INFO worker.py:1487 -- Connected to Ray cluster. View the dashboard at http://127.0.0.1:8265.
2022-08-05 10:20:02,005 INFO cluster_utils.py:162 -- RayContext(dashboard_url='127.0.0.1:8265', python_version='3.7.9', ray_version='2.0.0rc0', ray_commit='{{RAY_COMMIT_SHA}}', address_info={'node_ip_address': '172.18.0.3', 'raylet_ip_address': '172.18.0.3', 'redis_address': None, 'object_store_address': '/tmp/ray/session_2022-08-05_10-19-59_573575_12497/sockets/plasma_store', 'raylet_socket_name': '/tmp/ray/session_2022-08-05_10-19-59_573575_12497/sockets/raylet', 'webui_url': '127.0.0.1:8265', 'session_dir': '/tmp/ray/session_2022-08-05_10-19-59_573575_12497', 'metrics_export_port': 44870, 'gcs_address': '172.18.0.3:62294', 'address': '172.18.0.3:62294', 'dashboard_agent_listen_port': 52365, 'node_id': 'fc06183b9db6f7b18c4e81a0aee35e8af807073defc436d3cab3a7af'})
(pid=13922) /opt/miniconda/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
(pid=13922) import imp
(pid=13922) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:23: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
(pid=13922) 'nearest': pil_image.NEAREST,
(pid=13922) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:24: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
(pid=13922) 'bilinear': pil_image.BILINEAR,
(pid=13922) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:25: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
(pid=13922) 'bicubic': pil_image.BICUBIC,
(pid=13922) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:28: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.
(pid=13922) if hasattr(pil_image, 'HAMMING'):
(pid=13922) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:30: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.
(pid=13922) if hasattr(pil_image, 'BOX'):
(pid=13922) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:33: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
(pid=13922) if hasattr(pil_image, 'LANCZOS'):
(_MockTrainer pid=13922) 2022-08-05 10:20:08,319 WARNING util.py:65 -- Install gputil for GPU system monitoring.
2022-08-05 10:20:08,669 ERROR trial_runner.py:980 -- Trial __fake_d7105bea: Error processing event.
ray.tune.error._TuneNoNextExecutorEventError: Traceback (most recent call last):
File "/ray/python/ray/tune/execution/ray_trial_executor.py", line 989, in get_next_executor_event
future_result = ray.get(ready_future)
File "/ray/python/ray/_private/client_mode_hook.py", line 105, in wrapper
return func(*args, **kwargs)
File "/ray/python/ray/_private/worker.py", line 2247, in get
raise value
ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.
class_name: _MockTrainer
actor_id: ab6f28255b64399a2662b9a501000000
pid: 13922
namespace: a67948d7-aa7d-4178-8b36-8a44e0bbd375
ip: 172.18.0.3
The actor is dead because its node has died. Node Id: 4cb1eabd95bd85f500eb047825bb4daf66ca5d41e9d3ed473203b2b4
2022-08-05 10:20:08,670 INFO trial_runner.py:1323 -- Trial __fake_d7105bea: Attempting to restore trial state from last checkpoint.
2022-08-05 10:20:08,942 WARNING worker.py:1799 -- Raylet is terminated: ip=172.18.0.3, id=4cb1eabd95bd85f500eb047825bb4daf66ca5d41e9d3ed473203b2b4. Termination is unexpected. Possible reasons include: (1) SIGKILL by the user or system OOM killer, (2) Invalid memory access from Raylet causing SIGSEGV or SIGBUS, (3) Other termination signals. Last 20 lines of the Raylet logs:
[state-dump] NodeManagerService.grpc_server.RequestResourceReport - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
[state-dump] NodeInfoGcsService.grpc_client.GetInternalConfig - 1 total (0 active), CPU time: mean = 11.554 ms, total = 11.554 ms
[state-dump] InternalPubSubGcsService.grpc_client.GcsSubscriberPoll - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
[state-dump] NodeManager.deadline_timer.record_metrics - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
[state-dump] InternalPubSubGcsService.grpc_client.GcsSubscriberCommandBatch - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
[state-dump] RayletWorkerPool.deadline_timer.kill_idle_workers - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
[state-dump] NodeManager.deadline_timer.debug_state_dump - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
[state-dump] NodeManager.deadline_timer.flush_free_objects - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
[state-dump] DebugString() time ms: 0
[state-dump]
[state-dump]
[2022-08-05 10:20:01,879 I 13767 13767] (raylet) accessor.cc:608: Received notification for node id = fc06183b9db6f7b18c4e81a0aee35e8af807073defc436d3cab3a7af, IsAlive = 1
[2022-08-05 10:20:01,992 I 13767 13767] (raylet) node_manager.cc:599: New job has started. Job id 01000000 Driver pid 12497 is dead: 0 driver address: 172.18.0.3
[2022-08-05 10:20:01,992 I 13767 13767] (raylet) worker_pool.cc:636: Job 01000000 already started in worker pool.
[2022-08-05 10:20:01,997 I 13767 13784] (raylet) object_store.cc:35: Object store current usage 8e-09 / 0.157286 GB.
[2022-08-05 10:20:02,050 I 13767 13767] (raylet) accessor.cc:608: Received notification for node id = 4cb1eabd95bd85f500eb047825bb4daf66ca5d41e9d3ed473203b2b4, IsAlive = 1
[2022-08-05 10:20:03,035 I 13767 13767] (raylet) agent_manager.cc:40: HandleRegisterAgent, ip: 172.18.0.3, port: 50415, id: 424238335
[2022-08-05 10:20:08,362 I 13767 13767] (raylet) accessor.cc:608: Received notification for node id = 9cc6d602534ccc0a990dc01bb1c55dd9d66b8310712336a01c556622, IsAlive = 1
[2022-08-05 10:20:08,643 I 13767 13767] (raylet) accessor.cc:608: Received notification for node id = 4cb1eabd95bd85f500eb047825bb4daf66ca5d41e9d3ed473203b2b4, IsAlive = 0
[2022-08-05 10:20:08,671 I 13767 13767] (raylet) worker_pool.cc:447: Started worker process with pid 14004, the token is 0
(pid=14067) /opt/miniconda/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
(pid=14067) import imp
(pid=14067) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:23: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
(pid=14067) 'nearest': pil_image.NEAREST,
(pid=14067) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:24: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
(pid=14067) 'bilinear': pil_image.BILINEAR,
(pid=14067) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:25: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
(pid=14067) 'bicubic': pil_image.BICUBIC,
(pid=14067) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:28: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.
(pid=14067) if hasattr(pil_image, 'HAMMING'):
(pid=14067) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:30: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.
(pid=14067) if hasattr(pil_image, 'BOX'):
(pid=14067) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:33: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
(pid=14067) if hasattr(pil_image, 'LANCZOS'):
(_MockTrainer pid=14067) 2022-08-05 10:20:15,548 WARNING util.py:65 -- Install gputil for GPU system monitoring.
2022-08-05 10:20:16,522 WARNING util.py:220 -- The `process_trial_save` operation took 0.967 s, which may be a performance bottleneck.
(pid=14172) /opt/miniconda/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
(pid=14172) import imp
(pid=14172) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:23: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
(pid=14172) 'nearest': pil_image.NEAREST,
(pid=14172) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:24: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
(pid=14172) 'bilinear': pil_image.BILINEAR,
(pid=14172) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:25: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
(pid=14172) 'bicubic': pil_image.BICUBIC,
(pid=14172) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:28: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.
(pid=14172) if hasattr(pil_image, 'HAMMING'):
(pid=14172) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:30: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.
(pid=14172) if hasattr(pil_image, 'BOX'):
(pid=14172) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:33: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
(pid=14172) if hasattr(pil_image, 'LANCZOS'):
(_MockTrainer pid=14172) 2022-08-05 10:20:23,605 WARNING util.py:65 -- Install gputil for GPU system monitoring.
2022-08-05 10:20:24,574 WARNING util.py:220 -- The `process_trial_save` operation took 0.962 s, which may be a performance bottleneck.
2022-08-05 10:20:24,823 ERROR trial_runner.py:980 -- Trial __fake_dfa0a1ca: Error processing event.
ray.tune.error._TuneNoNextExecutorEventError: Traceback (most recent call last):
File "/ray/python/ray/tune/execution/ray_trial_executor.py", line 989, in get_next_executor_event
future_result = ray.get(ready_future)
File "/ray/python/ray/_private/client_mode_hook.py", line 105, in wrapper
return func(*args, **kwargs)
File "/ray/python/ray/_private/worker.py", line 2247, in get
raise value
ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.
class_name: _MockTrainer
actor_id: c7a15e58aa64a0c223f9b8bf01000000
pid: 14172
namespace: a67948d7-aa7d-4178-8b36-8a44e0bbd375
ip: 172.18.0.3
The actor is dead because its node has died. Node Id: 9cc6d602534ccc0a990dc01bb1c55dd9d66b8310712336a01c556622
2022-08-05 10:20:24,824 INFO trial_runner.py:1323 -- Trial __fake_dfa0a1ca: Attempting to restore trial state from last checkpoint.
2022-08-05 10:20:25,269 WARNING worker.py:1799 -- Raylet is terminated: ip=172.18.0.3, id=9cc6d602534ccc0a990dc01bb1c55dd9d66b8310712336a01c556622. Termination is unexpected. Possible reasons include: (1) SIGKILL by the user or system OOM killer, (2) Invalid memory access from Raylet causing SIGSEGV or SIGBUS, (3) Other termination signals. Last 20 lines of the Raylet logs:
[state-dump] NodeManager.deadline_timer.debug_state_dump - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
[state-dump] NodeManager.deadline_timer.flush_free_objects - 1 total (1 active), CPU time: mean = 0.000 s, total = 0.000 s
[state-dump] DebugString() time ms: 0
[state-dump]
[state-dump]
[2022-08-05 10:20:01,879 I 13767 13767] (raylet) accessor.cc:608: Received notification for node id = fc06183b9db6f7b18c4e81a0aee35e8af807073defc436d3cab3a7af, IsAlive = 1
[2022-08-05 10:20:01,992 I 13767 13767] (raylet) node_manager.cc:599: New job has started. Job id 01000000 Driver pid 12497 is dead: 0 driver address: 172.18.0.3
[2022-08-05 10:20:01,992 I 13767 13767] (raylet) worker_pool.cc:636: Job 01000000 already started in worker pool.
[2022-08-05 10:20:01,997 I 13767 13784] (raylet) object_store.cc:35: Object store current usage 8e-09 / 0.157286 GB.
[2022-08-05 10:20:02,050 I 13767 13767] (raylet) accessor.cc:608: Received notification for node id = 4cb1eabd95bd85f500eb047825bb4daf66ca5d41e9d3ed473203b2b4, IsAlive = 1
[2022-08-05 10:20:03,035 I 13767 13767] (raylet) agent_manager.cc:40: HandleRegisterAgent, ip: 172.18.0.3, port: 50415, id: 424238335
[2022-08-05 10:20:08,362 I 13767 13767] (raylet) accessor.cc:608: Received notification for node id = 9cc6d602534ccc0a990dc01bb1c55dd9d66b8310712336a01c556622, IsAlive = 1
[2022-08-05 10:20:08,643 I 13767 13767] (raylet) accessor.cc:608: Received notification for node id = 4cb1eabd95bd85f500eb047825bb4daf66ca5d41e9d3ed473203b2b4, IsAlive = 0
[2022-08-05 10:20:08,671 I 13767 13767] (raylet) worker_pool.cc:447: Started worker process with pid 14004, the token is 0
[2022-08-05 10:20:10,679 I 13767 13767] (raylet) node_manager.cc:1429: NodeManager::DisconnectClient, disconnect_type=1, has creation task exception = 0
[2022-08-05 10:20:15,557 I 13767 13767] (raylet) worker_pool.cc:447: Started worker process with pid 14112, the token is 1
[2022-08-05 10:20:17,678 I 13767 13767] (raylet) node_manager.cc:1429: NodeManager::DisconnectClient, disconnect_type=1, has creation task exception = 0
[2022-08-05 10:20:23,611 I 13767 13767] (raylet) worker_pool.cc:447: Started worker process with pid 14217, the token is 2
[2022-08-05 10:20:24,617 I 13767 13767] (raylet) accessor.cc:608: Received notification for node id = a7ba0994298865e2d941cb8d4a96c1c26ae8707f2b779b4ca0d9fe19, IsAlive = 1
[2022-08-05 10:20:24,767 I 13767 13767] (raylet) accessor.cc:608: Received notification for node id = 9cc6d602534ccc0a990dc01bb1c55dd9d66b8310712336a01c556622, IsAlive = 0
(pid=14319) /opt/miniconda/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
(pid=14319) import imp
(pid=14319) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:23: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
(pid=14319) 'nearest': pil_image.NEAREST,
(pid=14319) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:24: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
(pid=14319) 'bilinear': pil_image.BILINEAR,
(pid=14319) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:25: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
(pid=14319) 'bicubic': pil_image.BICUBIC,
(pid=14319) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:28: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.
(pid=14319) if hasattr(pil_image, 'HAMMING'):
(pid=14319) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:30: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.
(pid=14319) if hasattr(pil_image, 'BOX'):
(pid=14319) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:33: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
(pid=14319) if hasattr(pil_image, 'LANCZOS'):
(_MockTrainer pid=14319) 2022-08-05 10:20:31,532 WARNING util.py:65 -- Install gputil for GPU system monitoring.
(_MockTrainer pid=14319) 2022-08-05 10:20:31,537 INFO trainable.py:669 -- Restored on 172.18.0.3 from checkpoint: /tmp/checkpoint_tmp_lzh0n_y8
(_MockTrainer pid=14319) 2022-08-05 10:20:31,538 INFO trainable.py:677 -- Current state after restoring: {'_iteration': 2, '_timesteps_total': 20, '_time_total': 1.3113021850585938e-05, '_episodes_total': None}
2022-08-05 10:20:32,500 WARNING util.py:220 -- The `process_trial_save` operation took 0.957 s, which may be a performance bottleneck.
(pid=14423) /opt/miniconda/lib/python3.7/site-packages/tensorflow/python/autograph/impl/api.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
(pid=14423) import imp
(pid=14423) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:23: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
(pid=14423) 'nearest': pil_image.NEAREST,
(pid=14423) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:24: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
(pid=14423) 'bilinear': pil_image.BILINEAR,
(pid=14423) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:25: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
(pid=14423) 'bicubic': pil_image.BICUBIC,
(pid=14423) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:28: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.
(pid=14423) if hasattr(pil_image, 'HAMMING'):
(pid=14423) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:30: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.
(pid=14423) if hasattr(pil_image, 'BOX'):
(pid=14423) /opt/miniconda/lib/python3.7/site-packages/keras_preprocessing/image/utils.py:33: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
(pid=14423) if hasattr(pil_image, 'LANCZOS'):
(_MockTrainer pid=14423) 2022-08-05 10:20:39,558 WARNING util.py:65 -- Install gputil for GPU system monitoring.
2022-08-05 10:20:39,754 ERROR trial_runner.py:980 -- Trial __fake_e9255678: Error processing event.
ray.tune.error._TuneNoNextExecutorEventError: Traceback (most recent call last):
File "/ray/python/ray/tune/execution/ray_trial_executor.py", line 989, in get_next_executor_event
future_result = ray.get(ready_future)
File "/ray/python/ray/_private/client_mode_hook.py", line 105, in wrapper
return func(*args, **kwargs)
File "/ray/python/ray/_private/worker.py", line 2247, in get
raise value
ray.exceptions.RayActorError: The actor died unexpectedly before finishing this task.
class_name: _MockTrainer
actor_id: 6034476ab8918e80915b5f6501000000
pid: 14423
namespace: a67948d7-aa7d-4178-8b36-8a44e0bbd375
ip: 172.18.0.3
The actor is dead because its node has died. Node Id: a7ba0994298865e2d941cb8d4a96c1c26ae8707f2b779b4ca0d9fe19
ray/tune/tests/test_cluster.py ✓✓✓✓ 31% ███▏ 2022-08-05 10:20:44,709 INFO worker.py:1312 -- Connecting to existing Ray cluster at address: 172.18.0.3:61620...
2022-08-05 10:20:44,714 INFO worker.py:1487 -- Connected to Ray cluster. View the dashboard at http://127.0.0.1:8265.
2022-08-05 10:20:44,734 INFO cluster_utils.py:162 -- RayContext(dashboard_url='127.0.0.1:8265', python_version='3.7.9', ray_version='2.0.0rc0', ray_commit='{{RAY_COMMIT_SHA}}', address_info={'node_ip_address': '172.18.0.3', 'raylet_ip_address': '172.18.0.3', 'redis_address': None, 'object_store_address': '/tmp/ray/session_2022-08-05_10-20-42_504278_12497/sockets/plasma_store', 'raylet_socket_name': '/tmp/ray/session_2022-08-05_10-20-42_504278_12497/sockets/raylet', 'webui_url': '127.0.0.1:8265', 'session_dir': '/tmp/ray/session_2022-08-05_10-20-42_504278_12497', 'metrics_export_port': 47840, 'gcs_address': '172.18.0.3:61620', 'address': '172.18.0.3:61620', 'dashboard_agent_listen_port': 52365, 'node_id': '616ce4c7a560472926148c623d6f8a2605ec0e1cd5d4b37abc8d5598'})
*** SIGSEGV received at time=1659720044 on cpu 2 ***
PC: @ 0x7fa200000000 (unknown) dnnl::impl::cpu::x64::jit_avx_f32_copy_an_kern::generate()
@ 0x7fa2ae984420 3728 (unknown)
@ 0x7fa2aa23a660 64 ray::core::CoreWorkerMemoryStore::Get()
@ 0x7fa2aa23a849 208 ray::core::CoreWorkerMemoryStore::Get()
@ 0x7fa2aa1d2b4a 352 ray::core::CoreWorker::Get()
@ 0x7fa2aa0a6056 224 __pyx_pw_3ray_7_raylet_10CoreWorker_31get_objects()
@ 0x55b61015f914 (unknown) _PyMethodDef_RawFastCallKeywords
@ 0x7fa2aa0a5dc0 (unknown) (unknown)
[2022-08-05 10:20:44,802 E 12497 14504] logging.cc:361: *** SIGSEGV received at time=1659720044 on cpu 2 ***
[2022-08-05 10:20:44,802 E 12497 14504] logging.cc:361: PC: @ 0x7fa200000000 (unknown) dnnl::impl::cpu::x64::jit_avx_f32_copy_an_kern::generate()
[2022-08-05 10:20:44,802 E 12497 14504] logging.cc:361: @ 0x7fa2ae984420 3728 (unknown)
[2022-08-05 10:20:44,802 E 12497 14504] logging.cc:361: @ 0x7fa2aa23a660 64 ray::core::CoreWorkerMemoryStore::Get()
[2022-08-05 10:20:44,802 E 12497 14504] logging.cc:361: @ 0x7fa2aa23a849 208 ray::core::CoreWorkerMemoryStore::Get()
[2022-08-05 10:20:44,802 E 12497 14504] logging.cc:361: @ 0x7fa2aa1d2b4a 352 ray::core::CoreWorker::Get()
[2022-08-05 10:20:44,802 E 12497 14504] logging.cc:361: @ 0x7fa2aa0a6056 224 __pyx_pw_3ray_7_raylet_10CoreWorker_31get_objects()
[2022-08-05 10:20:44,802 E 12497 14504] logging.cc:361: @ 0x55b61015f914 (unknown) _PyMethodDef_RawFastCallKeywords
[2022-08-05 10:20:44,805 E 12497 14504] logging.cc:361: @ 0x7fa2aa0a5dc0 (unknown) (unknown)
Segmentation fault
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment