Created
April 29, 2021 10:23
-
-
Save Seraphli/02021a2717adc3c60d04615df3b42fbb to your computer and use it in GitHub Desktop.
tail /tmp/ray/session_latest/logs/*
This file has been truncated, but you can view the full file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ==> /tmp/ray/session_latest/logs/dashboard_agent.log <== | |
| 2021-04-29 17:01:31,397 INFO agent.py:72 -- Parent pid is 2395 | |
| 2021-04-29 17:01:31,398 INFO agent.py:76 -- Dashboard agent grpc address: 10.133.100.138:61628 | |
| 2021-04-29 17:01:31,400 INFO utils.py:202 -- Get all modules by type: DashboardAgentModule | |
| 2021-04-29 17:01:31,764 INFO agent.py:90 -- Loading DashboardAgentModule: <class 'ray.new_dashboard.modules.log.log_agent.LogAgent'> | |
| 2021-04-29 17:01:31,764 INFO agent.py:90 -- Loading DashboardAgentModule: <class 'ray.new_dashboard.modules.reporter.reporter_agent.ReporterAgent'> | |
| 2021-04-29 17:01:31,769 INFO agent.py:95 -- Loaded 2 modules. | |
| 2021-04-29 17:01:31,770 INFO agent.py:163 -- Dashboard agent http address: 10.133.100.138:19257 | |
| 2021-04-29 17:01:31,770 INFO agent.py:171 -- <ResourceRoute [GET] <StaticResource /logs -> PosixPath('/tmp/ray/session_2021-04-29_17-01-29_656359_2340/logs')> -> <bound method StaticResource._handle of <StaticResource /logs -> PosixPath('/tmp/ray/session_2021-04-29_17-01-29_656359_2340/logs')>> | |
| 2021-04-29 17:01:31,770 INFO agent.py:171 -- <ResourceRoute [OPTIONS] <StaticResource /logs -> PosixPath('/tmp/ray/session_2021-04-29_17-01-29_656359_2340/logs')> -> <bound method _PreflightHandler._preflight_handler of <aiohttp_cors.cors_config._CorsConfigImpl object at 0x7fdd898c2340>> | |
| 2021-04-29 17:01:31,770 INFO agent.py:172 -- Registered 2 routes. | |
| ==> /tmp/ray/session_latest/logs/dashboard.log <== | |
| raise _create_rpc_error(self._cython_call._initial_metadata, | |
| grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with: | |
| status = StatusCode.DEADLINE_EXCEEDED | |
| details = "Deadline Exceeded" | |
| debug_error_string = "{"created":"@1619691527.743034972","description":"Error received from peer ipv4:10.134.60.23:29553","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"Deadline Exceeded","grpc_status":4}" | |
| > | |
| 2021-04-29 18:18:48,272 INFO stats_collector_head.py:296 -- Received a log for 10.134.61.130 and 2344 | |
| 2021-04-29 18:18:48,647 INFO stats_collector_head.py:296 -- Received a log for 10.134.70.227 and 3304 | |
| 2021-04-29 18:18:48,828 INFO stats_collector_head.py:296 -- Received a log for 10.134.61.130 and 3260 | |
| 2021-04-29 18:18:49,235 INFO stats_collector_head.py:296 -- Received a log for 10.134.112.222 and 2302 | |
| ==> /tmp/ray/session_latest/logs/gcs_server.err <== | |
| ==> /tmp/ray/session_latest/logs/gcs_server.out <== | |
| - total commands queued: 0 | |
| DefaultTaskInfoHandler: {AddTask request count: 1599, GetTask request count: 0, AddTaskLease request count: 0, GetTaskLease request count: 0, AttemptTaskReconstruction request count: 0} | |
| [2021-04-29 18:18:29,952 I 2352 2352] gcs_server.cc:379: GcsNodeManager: {RegisterNode request count: 11, UnregisterNode request count: 0, GetAllNodeInfo request count: 3750, GetInternalConfig request count: 11} | |
| GcsActorManager: {RegisterActor request count: 862, CreateActor request count: 862, GetActorInfo request count: 12036, GetNamedActorInfo request count: 1523, KillActor request count: 0, Registered actors count: 862, Destroyed actors count: 0, Named actors count: 1, Unresolved actors count: 0, Pending actors count: 0, Created actors count: 11} | |
| GcsObjectManager: {GetObjectLocations request count: 0, GetAllObjectLocations request count: 0, AddObjectLocation request count: 0, RemoveObjectLocation request count: 0, Object count: 0} | |
| GcsPlacementGroupManager: {CreatePlacementGroup request count: 0, RemovePlacementGroup request count: 0, GetPlacementGroup request count: 0, GetAllPlacementGroup request count: 0, WaitPlacementGroupUntilReady request count: 0, Registered placement groups count: 0, Named placement group count: 0, Pending placement groups count: 0} | |
| GcsPubSub: | |
| - num channels subscribed to: 0 | |
| - total commands queued: 0 | |
| DefaultTaskInfoHandler: {AddTask request count: 1599, GetTask request count: 0, AddTaskLease request count: 0, GetTaskLease request count: 0, AttemptTaskReconstruction request count: 0} | |
| ==> /tmp/ray/session_latest/logs/log_monitor.log <== | |
| 2021-04-29 17:04:09,933 INFO log_monitor.py:162 -- Beginning to track file worker-3093572d3b5257c1efa43aadf2556a993e5df199364e514421779c91-01000000-18599.err | |
| 2021-04-29 17:04:09,933 INFO log_monitor.py:162 -- Beginning to track file worker-f76f5cd3ec56570992ad5d73c464769ae10ee2072c62afb8abbb688a-01000000-18605.out | |
| 2021-04-29 17:04:09,934 INFO log_monitor.py:162 -- Beginning to track file worker-3093572d3b5257c1efa43aadf2556a993e5df199364e514421779c91-01000000-18599.out | |
| 2021-04-29 17:04:09,936 INFO log_monitor.py:162 -- Beginning to track file worker-97de0e58634e306a1a34738bb12aec8564ba83cfeec1a97d283fbf9e-01000000-18593.err | |
| 2021-04-29 17:04:09,936 INFO log_monitor.py:162 -- Beginning to track file worker-0c482bb33815c518a2ed7d795842bf9899d44829770fa0a9e3296dc8-01000000-18581.err | |
| 2021-04-29 17:04:09,936 INFO log_monitor.py:162 -- Beginning to track file worker-e97008a83f19e36a0fd82fdc5b91eec9f008b370d8a429c91be7d3d4-01000000-18592.out | |
| 2021-04-29 17:04:09,937 INFO log_monitor.py:162 -- Beginning to track file worker-e97008a83f19e36a0fd82fdc5b91eec9f008b370d8a429c91be7d3d4-01000000-18592.err | |
| 2021-04-29 17:04:09,938 INFO log_monitor.py:162 -- Beginning to track file worker-0c482bb33815c518a2ed7d795842bf9899d44829770fa0a9e3296dc8-01000000-18581.out | |
| 2021-04-29 17:04:09,938 INFO log_monitor.py:162 -- Beginning to track file worker-1391830e9cdbceaad733fecac50f901dac985238d6bf09d9f0566126-01000000-18613.err | |
| 2021-04-29 17:04:09,938 INFO log_monitor.py:162 -- Beginning to track file worker-99d7c411d3dbb4eac06859444829c24e5a0f27d3174e77650fec6d3b-01000000-18631.out | |
| ==> /tmp/ray/session_latest/logs/monitor.err <== | |
| ==> /tmp/ray/session_latest/logs/monitor.log <== | |
| 2021-04-29 17:01:30,310 INFO monitor.py:361 -- Starting monitor using ray installation: /usr/local/lib/python3.8/site-packages/ray/__init__.py | |
| 2021-04-29 17:01:30,310 INFO monitor.py:362 -- Ray version: 1.3.0 | |
| 2021-04-29 17:01:30,310 INFO monitor.py:363 -- Ray commit: 9f45548488c4fa288f3cecb556801f97958eae8b | |
| 2021-04-29 17:01:30,310 INFO monitor.py:364 -- Monitor started with command: ['/usr/local/lib/python3.8/site-packages/ray/_private/monitor.py', '--logs-dir=/tmp/ray/session_2021-04-29_17-01-29_656359_2340/logs', '--redis-address=10.133.100.138:9000', '--logging-rotate-bytes=536870912', '--logging-rotate-backup-count=5', '--redis-password=5241590000000000'] | |
| 2021-04-29 17:01:30,315 INFO monitor.py:122 -- Monitor: Started | |
| ==> /tmp/ray/session_latest/logs/monitor.out <== | |
| ==> /tmp/ray/session_latest/logs/old <== | |
| ==> /tmp/ray/session_latest/logs/plasma_store.err <== | |
| ==> /tmp/ray/session_latest/logs/plasma_store.out <== | |
| [2021-04-29 17:01:30,898 I 2394 2394] store_exec.cc:81: The Plasma Store is started with the '-z' flag, and it will run idle as a placeholder. | |
| ==> /tmp/ray/session_latest/logs/python-core-driver-01000000ffffffffffffffffffffffffffffffffffffffffffffffff_3930.log <== | |
| [2021-04-29 17:03:21,436 E 3930 3930] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:21,436 E 3930 3930] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:21,454 I 3930 3983] direct_task_transport.cc:189: Task queue empty, canceling lease request | |
| [2021-04-29 17:03:21,747 I 3930 3983] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 4df1e90f84c18be653ea651801000000, ip address: 10.133.100.138, port: 9148, worker_id: f187dcfc8ba7434ef61b09d24021b869b93bca29b01c19d7f9c1384c, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:21,749 E 3930 3930] actor_manager.cc:93: Actor handle already exists 4df1e90f84c18be653ea651801000000 | |
| [2021-04-29 17:03:35,643 E 3930 3930] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:35,643 E 3930 3930] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:35,662 I 3930 3983] direct_task_transport.cc:189: Task queue empty, canceling lease request | |
| [2021-04-29 17:03:35,959 I 3930 3983] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 5fcfdec7dbd20ae44c215a0801000000, ip address: 10.133.100.138, port: 9151, worker_id: 59619cb9e9aa958d17251b8cd812f8bbc6aa1c308048565cbd6c7219, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:35,961 E 3930 3930] actor_manager.cc:93: Actor handle already exists 5fcfdec7dbd20ae44c215a0801000000 | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-0013ad47cb5db38eda749638d7879ff7696511eb2f60185df5adb986_7448.log <== | |
| [2021-04-29 17:03:58,147 I 7448 8184] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,148 E 7448 7448] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,149 I 7448 8184] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,706 E 7448 7448] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:58,786 I 7448 7448] direct_task_transport.cc:47: Creating actor via GCS actor id = : cfd8c4f4c83e00c41193cb7901000000 | |
| [2021-04-29 17:03:58,804 I 7448 8184] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: cfd8c4f4c83e00c41193cb7901000000, ip address: 10.133.95.40, port: 9136, worker_id: c35530e14c73c2b4cf6f5f4b7c18536e9503ce3f00246ac4b33bdc78, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,477 I 7448 8184] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: cfd8c4f4c83e00c41193cb7901000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,549 I 7448 8184] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: cfd8c4f4c83e00c41193cb7901000000, ip address: 10.133.95.40, port: 9136, worker_id: c35530e14c73c2b4cf6f5f4b7c18536e9503ce3f00246ac4b33bdc78, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:06,284 E 7448 7448] actor_manager.cc:93: Actor handle already exists cfd8c4f4c83e00c41193cb7901000000 | |
| [2021-04-29 17:04:12,041 I 7448 7448] core_worker.cc:2064: Finished executing task 9a3f2e912832626cffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-017e00a98456a3609d3548ec2d95f7e0b7b40809575071223ce6304e_17728.log <== | |
| [2021-04-29 17:04:09,144 I 17728 19149] service_based_accessor.cc:579: Received notification for node id = ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, IsAlive = 1 | |
| [2021-04-29 17:04:09,144 I 17728 19149] service_based_accessor.cc:579: Received notification for node id = d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, IsAlive = 1 | |
| [2021-04-29 17:04:09,144 I 17728 19149] service_based_accessor.cc:579: Received notification for node id = b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, IsAlive = 1 | |
| [2021-04-29 17:04:09,144 I 17728 19149] service_based_accessor.cc:579: Received notification for node id = 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, IsAlive = 1 | |
| [2021-04-29 17:04:09,144 I 17728 19149] service_based_accessor.cc:579: Received notification for node id = 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, IsAlive = 1 | |
| [2021-04-29 17:04:10,142 I 17728 19149] core_worker.cc:717: Exit signal received, this process will exit after all outstanding tasks have finished, exit_type=INTENDED_EXIT | |
| [2021-04-29 17:04:10,142 I 17728 17728] raylet_client.cc:129: RayletClient::Disconnect, exit_type=INTENDED_EXIT, has creation_task_exception_pb_bytes=0 | |
| [2021-04-29 17:04:10,142 I 17728 17728] core_worker.cc:308: Removed worker 017e00a98456a3609d3548ec2d95f7e0b7b40809575071223ce6304e | |
| [2021-04-29 17:04:10,146 I 17728 17728] core_worker.cc:181: Destructing CoreWorkerProcess. pid: 17728 | |
| [2021-04-29 17:04:10,146 I 17728 17728] io_service_pool.cc:47: IOServicePool is stopped. | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-022061b03a8293555699fc7a8ffe57b73767357e5a0e895032f94384_5287.log <== | |
| [2021-04-29 17:03:59,207 E 5287 5287] actor_manager.cc:93: Actor handle already exists c10133c2e064940f30de8d0f01000000 | |
| [2021-04-29 17:04:01,385 I 5287 5287] core_worker.cc:2064: Finished executing task 7e58f81f439024f4ffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:02,000 E 5287 5287] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:02,000 E 5287 5287] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:02,064 I 5287 5287] direct_task_transport.cc:47: Creating actor via GCS actor id = : 5ab38e0c08dc3c1493f2735101000000 | |
| [2021-04-29 17:04:02,081 I 5287 6148] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 5ab38e0c08dc3c1493f2735101000000, ip address: 10.134.112.222, port: 9115, worker_id: e814e4a61917a7e2369dcb653e531d5271a539fda85b24cb4e1a5d5e, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,542 I 5287 6148] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 5ab38e0c08dc3c1493f2735101000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,391 I 5287 6148] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 5ab38e0c08dc3c1493f2735101000000, ip address: 10.134.112.222, port: 9115, worker_id: e814e4a61917a7e2369dcb653e531d5271a539fda85b24cb4e1a5d5e, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,129 E 5287 5287] actor_manager.cc:93: Actor handle already exists 5ab38e0c08dc3c1493f2735101000000 | |
| [2021-04-29 17:04:29,468 I 5287 5287] core_worker.cc:2064: Finished executing task 78d5adfefad361ebffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-022d7ac09eb51b6d3311f8bf8130f79a20486f0eb4e17c5f6032bf45_5385.log <== | |
| [2021-04-29 17:03:46,174 E 5385 5385] actor_manager.cc:93: Actor handle already exists ff125b7bd2b8fb7149bc306301000000 | |
| [2021-04-29 17:03:46,275 I 5385 5385] core_worker.cc:2064: Finished executing task ffffffffffffffffac1bcbf6934a6751aa1bab9101000000, status=OK | |
| [2021-04-29 17:03:46,275 I 5385 5385] direct_actor_transport.cc:497: Actor creation task finished, task_id: ffffffffffffffffac1bcbf6934a6751aa1bab9101000000, actor_id: ac1bcbf6934a6751aa1bab9101000000 | |
| [2021-04-29 17:03:46,277 I 5385 6998] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: ac1bcbf6934a6751aa1bab9101000000, ip address: 10.133.100.138, port: 9240, worker_id: 022d7ac09eb51b6d3311f8bf8130f79a20486f0eb4e17c5f6032bf45, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:46,278 I 5385 5385] core_worker.cc:2064: Finished executing task c683ef94f369c490ac1bcbf6934a6751aa1bab9101000000, status=OK | |
| [2021-04-29 17:03:46,282 I 5385 5385] core_worker.cc:2064: Finished executing task 6b7d6ead684e8ec9ac1bcbf6934a6751aa1bab9101000000, status=OK | |
| [2021-04-29 17:03:46,298 I 5385 5385] core_worker.cc:2064: Finished executing task 9878094def831b9cac1bcbf6934a6751aa1bab9101000000, status=OK | |
| [2021-04-29 17:03:46,595 I 5385 5385] core_worker.cc:2064: Finished executing task fc1543d5785b5cd5ac1bcbf6934a6751aa1bab9101000000, status=OK | |
| [2021-04-29 17:03:46,606 I 5385 5385] core_worker.cc:2064: Finished executing task 6010904978e2be1fac1bcbf6934a6751aa1bab9101000000, status=OK | |
| [2021-04-29 17:03:51,408 I 5385 5385] core_worker.cc:2064: Finished executing task bd59523528bc6cc6ac1bcbf6934a6751aa1bab9101000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-0256fcb2128e48e54bc852d699a5428ff890ee8c00733ca76194867f_4064.log <== | |
| [2021-04-29 17:03:44,991 I 4064 4064] core_worker.cc:2064: Finished executing task afa703c941641edc1ff7c757488400a158dcf08e01000000, status=OK | |
| [2021-04-29 17:04:48,375 I 4064 4064] core_worker.cc:2064: Finished executing task e93eb87c5a445da61ff7c757488400a158dcf08e01000000, status=OK | |
| [2021-04-29 17:04:50,646 I 4064 4064] core_worker.cc:2064: Finished executing task 210c6f961463bde61ff7c757488400a158dcf08e01000000, status=OK | |
| [2021-04-29 17:04:50,697 I 4064 4064] core_worker.cc:2064: Finished executing task 94d140f3082131cb1ff7c757488400a158dcf08e01000000, status=OK | |
| [2021-04-29 17:04:51,934 I 4064 4064] core_worker.cc:2064: Finished executing task 0ba18de27689da741ff7c757488400a158dcf08e01000000, status=OK | |
| [2021-04-29 17:04:52,194 I 4064 4064] core_worker.cc:2064: Finished executing task be6c604852db71a11ff7c757488400a158dcf08e01000000, status=OK | |
| [2021-04-29 17:04:54,184 I 4064 4064] core_worker.cc:2064: Finished executing task 0a5ea5250528a9e31ff7c757488400a158dcf08e01000000, status=OK | |
| [2021-04-29 17:04:55,929 I 4064 4064] core_worker.cc:2064: Finished executing task ab935b23245e37571ff7c757488400a158dcf08e01000000, status=OK | |
| [2021-04-29 17:04:56,531 I 4064 4064] core_worker.cc:2064: Finished executing task 06fcb1f49c43ff181ff7c757488400a158dcf08e01000000, status=OK | |
| [2021-04-29 17:04:58,060 I 4064 4064] core_worker.cc:2064: Finished executing task c33eab3a8d456e511ff7c757488400a158dcf08e01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-02795b6c5446bd404f482c004e966a35a4e143e155b50db94fac4492_15896.log <== | |
| [2021-04-29 17:04:07,911 I 15896 16770] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,151 E 15896 15896] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,200 I 15896 16770] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,925 E 15896 15896] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:10,994 I 15896 15896] direct_task_transport.cc:47: Creating actor via GCS actor id = : 5182a1aa6b99acc9355d231801000000 | |
| [2021-04-29 17:04:11,026 I 15896 16770] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 5182a1aa6b99acc9355d231801000000, ip address: 10.134.60.23, port: 9201, worker_id: 42882c3d710e744fef6da7f41b4206fa3514a8cd5db337916c51cfc1, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,368 I 15896 16770] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 5182a1aa6b99acc9355d231801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:23,922 I 15896 16770] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 5182a1aa6b99acc9355d231801000000, ip address: 10.134.60.23, port: 9201, worker_id: 42882c3d710e744fef6da7f41b4206fa3514a8cd5db337916c51cfc1, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:26,941 E 15896 15896] actor_manager.cc:93: Actor handle already exists 5182a1aa6b99acc9355d231801000000 | |
| [2021-04-29 17:04:29,773 I 15896 15896] core_worker.cc:2064: Finished executing task e7b389b9c2a4df46ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-02f135e38cf5d4f0986ac165d1c9d3971638f7f896169d5d1504558a_14325.log <== | |
| [2021-04-29 17:04:05,949 I 14325 15789] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,984 E 14325 14325] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:06,007 I 14325 15789] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,799 E 14325 14325] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:08,915 I 14325 14325] direct_task_transport.cc:47: Creating actor via GCS actor id = : db8d681b5b2cc5be10fecdd801000000 | |
| [2021-04-29 17:04:08,968 I 14325 15789] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: db8d681b5b2cc5be10fecdd801000000, ip address: 10.134.60.23, port: 9179, worker_id: 13840852d0bd7fc583c20a31e0f1eb2e71338ac6271cfbb5457c83de, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:20,042 I 14325 15789] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: db8d681b5b2cc5be10fecdd801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,401 I 14325 15789] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: db8d681b5b2cc5be10fecdd801000000, ip address: 10.134.60.23, port: 9179, worker_id: 13840852d0bd7fc583c20a31e0f1eb2e71338ac6271cfbb5457c83de, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,554 E 14325 14325] actor_manager.cc:93: Actor handle already exists db8d681b5b2cc5be10fecdd801000000 | |
| [2021-04-29 17:04:29,534 I 14325 14325] core_worker.cc:2064: Finished executing task 45c19ac3423dacdeffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-03290fba63712c0509edc177ff3400ce27fb5634d8bfbb7fdc0e38b0_12974.log <== | |
| [2021-04-29 17:04:05,089 I 12974 14517] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,117 E 12974 12974] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,137 I 12974 14517] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:07,012 E 12974 12974] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:07,150 I 12974 12974] direct_task_transport.cc:47: Creating actor via GCS actor id = : 443c1013bbda88e345c2aee901000000 | |
| [2021-04-29 17:04:07,199 I 12974 14517] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 443c1013bbda88e345c2aee901000000, ip address: 10.134.112.222, port: 21101, worker_id: NIL_ID, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:16,289 I 12974 14517] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 443c1013bbda88e345c2aee901000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,398 I 12974 14517] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 443c1013bbda88e345c2aee901000000, ip address: 10.134.112.222, port: 9157, worker_id: 3aeaa5d68c4d82a6dd1b8309fd61a549c3135bcd6e69b8f1b01fe165, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,379 E 12974 12974] actor_manager.cc:93: Actor handle already exists 443c1013bbda88e345c2aee901000000 | |
| [2021-04-29 17:04:29,518 I 12974 12974] core_worker.cc:2064: Finished executing task 4c723af4fac450b7ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-033780ec16b16eab8fdae05f8bff916f301a3e9189b98049a4b6924e_14337.log <== | |
| [2021-04-29 17:04:05,947 I 14337 15759] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,984 E 14337 14337] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:06,007 I 14337 15759] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,779 E 14337 14337] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:08,875 I 14337 14337] direct_task_transport.cc:47: Creating actor via GCS actor id = : daffca0c5d9fb3fc7277334e01000000 | |
| [2021-04-29 17:04:08,923 I 14337 15759] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: daffca0c5d9fb3fc7277334e01000000, ip address: 10.134.61.130, port: 9226, worker_id: 512a43c6bc104d244ee0cb3ec8c08c0d2efc2c1133c4dc98fd4cf874, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:19,564 I 14337 15759] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: daffca0c5d9fb3fc7277334e01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,681 I 14337 15759] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: daffca0c5d9fb3fc7277334e01000000, ip address: 10.134.61.130, port: 9226, worker_id: 512a43c6bc104d244ee0cb3ec8c08c0d2efc2c1133c4dc98fd4cf874, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:24,457 E 14337 14337] actor_manager.cc:93: Actor handle already exists daffca0c5d9fb3fc7277334e01000000 | |
| [2021-04-29 17:04:29,592 I 14337 14337] core_worker.cc:2064: Finished executing task 6b751ba610d9d4c0ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-033b0b92191c38120f7109fc75e093894db7d97a48438eb0cf620f58_18036.log <== | |
| [2021-04-29 17:04:09,621 I 18036 19759] service_based_accessor.cc:579: Received notification for node id = ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, IsAlive = 1 | |
| [2021-04-29 17:04:09,621 I 18036 19759] service_based_accessor.cc:579: Received notification for node id = d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, IsAlive = 1 | |
| [2021-04-29 17:04:09,621 I 18036 19759] service_based_accessor.cc:579: Received notification for node id = b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, IsAlive = 1 | |
| [2021-04-29 17:04:09,621 I 18036 19759] service_based_accessor.cc:579: Received notification for node id = 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, IsAlive = 1 | |
| [2021-04-29 17:04:09,621 I 18036 19759] service_based_accessor.cc:579: Received notification for node id = 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, IsAlive = 1 | |
| [2021-04-29 17:04:10,733 I 18036 19759] core_worker.cc:717: Exit signal received, this process will exit after all outstanding tasks have finished, exit_type=INTENDED_EXIT | |
| [2021-04-29 17:04:10,733 I 18036 18036] raylet_client.cc:129: RayletClient::Disconnect, exit_type=INTENDED_EXIT, has creation_task_exception_pb_bytes=0 | |
| [2021-04-29 17:04:10,734 I 18036 18036] core_worker.cc:308: Removed worker 033b0b92191c38120f7109fc75e093894db7d97a48438eb0cf620f58 | |
| [2021-04-29 17:04:10,736 I 18036 18036] core_worker.cc:181: Destructing CoreWorkerProcess. pid: 18036 | |
| [2021-04-29 17:04:10,736 I 18036 18036] io_service_pool.cc:47: IOServicePool is stopped. | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-03460bf70c169da22bddb6e1a2bf23702e5d0d5a430854755eb20b3d_12115.log <== | |
| [2021-04-29 17:04:03,452 I 12115 12639] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,479 E 12115 12115] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:03,491 I 12115 12639] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,873 E 12115 12115] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,970 I 12115 12115] direct_task_transport.cc:47: Creating actor via GCS actor id = : 978026690762b1d28ba112dc01000000 | |
| [2021-04-29 17:04:05,983 I 12115 12639] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 978026690762b1d28ba112dc01000000, ip address: 10.134.142.44, port: 9153, worker_id: 9029ee66d3a78f112281ba2cc181ad4bad4237078d3e919f97b8955d, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,519 I 12115 12639] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 978026690762b1d28ba112dc01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,395 I 12115 12639] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 978026690762b1d28ba112dc01000000, ip address: 10.134.142.44, port: 9153, worker_id: 9029ee66d3a78f112281ba2cc181ad4bad4237078d3e919f97b8955d, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,257 E 12115 12115] actor_manager.cc:93: Actor handle already exists 978026690762b1d28ba112dc01000000 | |
| [2021-04-29 17:04:29,485 I 12115 12115] core_worker.cc:2064: Finished executing task 9d2d0237e646fa11ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-037fe00546f76e5aaf254b029cd5e73b7f18c23b76d97bd2a7fe41a5_16065.log <== | |
| [2021-04-29 17:04:08,970 I 16065 18110] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,088 E 16065 16065] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,114 I 16065 18110] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,206 E 16065 16065] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,301 I 16065 16065] direct_task_transport.cc:47: Creating actor via GCS actor id = : 8bdb45f08c4f20f6aa730aa101000000 | |
| [2021-04-29 17:04:11,328 I 16065 18110] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 8bdb45f08c4f20f6aa730aa101000000, ip address: 10.134.142.44, port: 9209, worker_id: d3f175a4695b5686e617385adc148a75458f4a303ac00a23c61daa41, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,377 I 16065 18110] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 8bdb45f08c4f20f6aa730aa101000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:27,492 I 16065 18110] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8bdb45f08c4f20f6aa730aa101000000, ip address: 10.134.142.44, port: 9209, worker_id: d3f175a4695b5686e617385adc148a75458f4a303ac00a23c61daa41, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,884 E 16065 16065] actor_manager.cc:93: Actor handle already exists 8bdb45f08c4f20f6aa730aa101000000 | |
| [2021-04-29 17:04:29,851 I 16065 16065] core_worker.cc:2064: Finished executing task 471820677f2628cdffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-0385806d292cb18ed49d141f6db021771481aa1bd8fd40fe797d41fb_12925.log <== | |
| [2021-04-29 17:04:05,102 I 12925 14497] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,131 E 12925 12925] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,150 I 12925 14497] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:07,013 E 12925 12925] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:07,150 I 12925 12925] direct_task_transport.cc:47: Creating actor via GCS actor id = : 902cabe28a97273fddda5c7601000000 | |
| [2021-04-29 17:04:07,199 I 12925 14497] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 902cabe28a97273fddda5c7601000000, ip address: 10.134.61.130, port: 9230, worker_id: 3071af190d8153756fc4572acea926e9d980bf9739e4d824e620c2fe, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:16,358 I 12925 14497] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 902cabe28a97273fddda5c7601000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,398 I 12925 14497] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 902cabe28a97273fddda5c7601000000, ip address: 10.134.61.130, port: 9230, worker_id: 3071af190d8153756fc4572acea926e9d980bf9739e4d824e620c2fe, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,396 E 12925 12925] actor_manager.cc:93: Actor handle already exists 902cabe28a97273fddda5c7601000000 | |
| [2021-04-29 17:04:29,517 I 12925 12925] core_worker.cc:2064: Finished executing task bc56f23700e50375ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-040ac8d8a076e2abee9699210b547637788099bd249ffd54bc2135d5_17836.log <== | |
| [2021-04-29 17:04:09,335 I 17836 19379] service_based_accessor.cc:579: Received notification for node id = ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, IsAlive = 1 | |
| [2021-04-29 17:04:09,335 I 17836 19379] service_based_accessor.cc:579: Received notification for node id = d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, IsAlive = 1 | |
| [2021-04-29 17:04:09,335 I 17836 19379] service_based_accessor.cc:579: Received notification for node id = b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, IsAlive = 1 | |
| [2021-04-29 17:04:09,335 I 17836 19379] service_based_accessor.cc:579: Received notification for node id = 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, IsAlive = 1 | |
| [2021-04-29 17:04:09,335 I 17836 19379] service_based_accessor.cc:579: Received notification for node id = 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, IsAlive = 1 | |
| [2021-04-29 17:04:10,338 I 17836 19379] core_worker.cc:717: Exit signal received, this process will exit after all outstanding tasks have finished, exit_type=INTENDED_EXIT | |
| [2021-04-29 17:04:10,338 I 17836 17836] raylet_client.cc:129: RayletClient::Disconnect, exit_type=INTENDED_EXIT, has creation_task_exception_pb_bytes=0 | |
| [2021-04-29 17:04:10,338 I 17836 17836] core_worker.cc:308: Removed worker 040ac8d8a076e2abee9699210b547637788099bd249ffd54bc2135d5 | |
| [2021-04-29 17:04:10,340 I 17836 17836] core_worker.cc:181: Destructing CoreWorkerProcess. pid: 17836 | |
| [2021-04-29 17:04:10,340 I 17836 17836] io_service_pool.cc:47: IOServicePool is stopped. | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-04189715074da0bbb4997d5782e334e26696470e4726b1a689fbd51c_17840.log <== | |
| [2021-04-29 17:04:09,844 I 17840 19612] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,919 E 17840 17840] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,943 I 17840 19612] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,582 E 17840 17840] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,702 I 17840 17840] direct_task_transport.cc:47: Creating actor via GCS actor id = : d767f80214b90c53c3e509a101000000 | |
| [2021-04-29 17:04:11,715 I 17840 19612] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: d767f80214b90c53c3e509a101000000, ip address: 10.134.61.25, port: 9185, worker_id: 103fcec220e6f736771833ab30509b5fcf8d807cc918ec056279e67b, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,385 I 17840 19612] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: d767f80214b90c53c3e509a101000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:27,006 I 17840 19612] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: d767f80214b90c53c3e509a101000000, ip address: 10.134.61.25, port: 9185, worker_id: 103fcec220e6f736771833ab30509b5fcf8d807cc918ec056279e67b, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,747 E 17840 17840] actor_manager.cc:93: Actor handle already exists d767f80214b90c53c3e509a101000000 | |
| [2021-04-29 17:04:29,845 I 17840 17840] core_worker.cc:2064: Finished executing task 27183076499dd950ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-0418b9c53f2bea3de049d4c376c3971048f5d8ac8da35a73e19483c1_18592.log <== | |
| [2021-04-29 17:04:10,142 I 18592 20046] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,294 E 18592 18592] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,313 I 18592 20046] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,623 E 18592 18592] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,751 I 18592 18592] direct_task_transport.cc:47: Creating actor via GCS actor id = : c1cb51b9bb8f9468ca8a76fc01000000 | |
| [2021-04-29 17:04:11,779 I 18592 20046] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: c1cb51b9bb8f9468ca8a76fc01000000, ip address: 10.134.70.227, port: 9198, worker_id: 79df031c006b044b023feeda8c3e93051530026d2d96cf8ff6d9df81, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,389 I 18592 20046] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: c1cb51b9bb8f9468ca8a76fc01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:28,434 I 18592 20046] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: c1cb51b9bb8f9468ca8a76fc01000000, ip address: 10.134.70.227, port: 9198, worker_id: 79df031c006b044b023feeda8c3e93051530026d2d96cf8ff6d9df81, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:29,364 E 18592 18592] actor_manager.cc:93: Actor handle already exists c1cb51b9bb8f9468ca8a76fc01000000 | |
| [2021-04-29 17:04:29,866 I 18592 18592] core_worker.cc:2064: Finished executing task e5bc1aed567a297bffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-0440a94a5a7e325663c1f244dcfd597bb60af8403a832a4bc6f3e741_12907.log <== | |
| [2021-04-29 17:04:05,026 I 12907 14048] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,083 E 12907 12907] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,101 I 12907 14048] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,949 E 12907 12907] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:07,116 I 12907 12907] direct_task_transport.cc:47: Creating actor via GCS actor id = : cf83ea349c16dd8f89bf9b9101000000 | |
| [2021-04-29 17:04:07,158 I 12907 14048] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: cf83ea349c16dd8f89bf9b9101000000, ip address: 10.134.142.16, port: 22629, worker_id: NIL_ID, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:16,154 I 12907 14048] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: cf83ea349c16dd8f89bf9b9101000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,398 I 12907 14048] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: cf83ea349c16dd8f89bf9b9101000000, ip address: 10.134.142.16, port: 9157, worker_id: 15a56db4366d56827eeaa4f505c16572ddcabb90b7483598644ec718, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,387 E 12907 12907] actor_manager.cc:93: Actor handle already exists cf83ea349c16dd8f89bf9b9101000000 | |
| [2021-04-29 17:04:29,521 I 12907 12907] core_worker.cc:2064: Finished executing task 854f01968d915accffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-048472fc66edb81954d41d062df9a94a4d1471746535bc3df2219166_9736.log <== | |
| [2021-04-29 17:04:01,277 I 9736 10695] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,301 E 9736 9736] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,310 I 9736 10695] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,924 E 9736 9736] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:01,985 I 9736 9736] direct_task_transport.cc:47: Creating actor via GCS actor id = : 64824d1124f9ec443214f21801000000 | |
| [2021-04-29 17:04:01,994 I 9736 10695] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 64824d1124f9ec443214f21801000000, ip address: 10.133.95.40, port: 9113, worker_id: 9b044549559f1c50b258fbadedeca532a30c6494de82bd5ad74c8ced, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,529 I 9736 10695] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 64824d1124f9ec443214f21801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,390 I 9736 10695] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 64824d1124f9ec443214f21801000000, ip address: 10.133.95.40, port: 9113, worker_id: 9b044549559f1c50b258fbadedeca532a30c6494de82bd5ad74c8ced, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,065 E 9736 9736] actor_manager.cc:93: Actor handle already exists 64824d1124f9ec443214f21801000000 | |
| [2021-04-29 17:04:29,451 I 9736 9736] core_worker.cc:2064: Finished executing task f72367ccc078980effffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-066715dd61d5e832670b539720a9739d24cb84f88f185f890543b212_7469.log <== | |
| [2021-04-29 17:03:58,239 I 7469 8309] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,242 E 7469 7469] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,243 I 7469 8309] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,712 E 7469 7469] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:58,801 I 7469 7469] direct_task_transport.cc:47: Creating actor via GCS actor id = : a64afafead302710871296ea01000000 | |
| [2021-04-29 17:03:58,819 I 7469 8309] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: a64afafead302710871296ea01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,667 I 7469 8309] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: a64afafead302710871296ea01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,774 I 7469 8309] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: a64afafead302710871296ea01000000, ip address: 10.134.60.23, port: 9137, worker_id: 9264203f5ed875035bae66c169dcb32a0084aa66d179ec1d8cfd202c, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:06,354 E 7469 7469] actor_manager.cc:93: Actor handle already exists a64afafead302710871296ea01000000 | |
| [2021-04-29 17:04:12,095 I 7469 7469] core_worker.cc:2064: Finished executing task 99c5289bc6b91b62ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-07390ab6657c848dcead7513f6497e85f8763013b22ac5111835731e_7587.log <== | |
| [2021-04-29 17:03:58,772 I 7587 9169] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,810 E 7587 7587] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,818 I 7587 9169] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,501 E 7587 7587] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,548 I 7587 7587] direct_task_transport.cc:47: Creating actor via GCS actor id = : 9ebd7edee9a9cc5b1a37ecf701000000 | |
| [2021-04-29 17:03:59,557 I 7587 9169] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 9ebd7edee9a9cc5b1a37ecf701000000, ip address: 10.133.95.40, port: 9131, worker_id: 204dfeb11082d602df54ff25dcfb53ee618ddfbf6b2f7e38bafb76a7, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,398 I 7587 9169] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 9ebd7edee9a9cc5b1a37ecf701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:13,608 I 7587 9169] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 9ebd7edee9a9cc5b1a37ecf701000000, ip address: 10.133.95.40, port: 9131, worker_id: 204dfeb11082d602df54ff25dcfb53ee618ddfbf6b2f7e38bafb76a7, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:15,345 E 7587 7587] actor_manager.cc:93: Actor handle already exists 9ebd7edee9a9cc5b1a37ecf701000000 | |
| [2021-04-29 17:04:22,249 I 7587 7587] core_worker.cc:2064: Finished executing task 19388ce60c5d488affffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-07551c1c3aafd8ddad02cc9feb8700ab949f0c1c7f9048c4bba2b865_17825.log <== | |
| [2021-04-29 17:04:09,621 I 17825 19330] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,792 E 17825 17825] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,819 I 17825 19330] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,577 E 17825 17825] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,704 I 17825 17825] direct_task_transport.cc:47: Creating actor via GCS actor id = : d0fcaca5d7a0ac8093b52a5a01000000 | |
| [2021-04-29 17:04:11,756 I 17825 19330] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: d0fcaca5d7a0ac8093b52a5a01000000, ip address: 10.133.95.40, port: 9180, worker_id: 347920f79ad493c9db0d772b83b84c2272a8708fbe3545f6a015fcec, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,386 I 17825 19330] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: d0fcaca5d7a0ac8093b52a5a01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:27,462 I 17825 19330] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: d0fcaca5d7a0ac8093b52a5a01000000, ip address: 10.133.95.40, port: 9180, worker_id: 347920f79ad493c9db0d772b83b84c2272a8708fbe3545f6a015fcec, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,877 E 17825 17825] actor_manager.cc:93: Actor handle already exists d0fcaca5d7a0ac8093b52a5a01000000 | |
| [2021-04-29 17:04:29,855 I 17825 17825] core_worker.cc:2064: Finished executing task 4d4dcb086417e0cbffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-07b5dded858a291c780cb256a9cfebf793546e9c0a83dc5884525cb6_12476.log <== | |
| [2021-04-29 17:04:04,540 I 12476 13213] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,598 E 12476 12476] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:04,615 I 12476 13213] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,511 E 12476 12476] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:06,618 I 12476 12476] direct_task_transport.cc:47: Creating actor via GCS actor id = : d05fa3acfb52a0ad9cb3afbb01000000 | |
| [2021-04-29 17:04:06,680 I 12476 13213] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: d05fa3acfb52a0ad9cb3afbb01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:12,732 I 12476 13213] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: d05fa3acfb52a0ad9cb3afbb01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,396 I 12476 13213] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: d05fa3acfb52a0ad9cb3afbb01000000, ip address: 10.134.61.130, port: 9231, worker_id: 4b79bd72cc116f3c93d1b2080a7452a319f9814107fb06775c93ebdd, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,314 E 12476 12476] actor_manager.cc:93: Actor handle already exists d05fa3acfb52a0ad9cb3afbb01000000 | |
| [2021-04-29 17:04:29,503 I 12476 12476] core_worker.cc:2064: Finished executing task c1d0facafb58b796ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-07e3595fdf3c3d4a43039518f66f5f4d06d01d53bc34790c8fdbb153_15979.log <== | |
| [2021-04-29 17:04:08,415 I 15979 17613] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,506 E 15979 15979] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,538 I 15979 17613] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,055 E 15979 15979] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,165 I 15979 15979] direct_task_transport.cc:47: Creating actor via GCS actor id = : b623feeb6b04ab65105ce84901000000 | |
| [2021-04-29 17:04:11,194 I 15979 17613] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: b623feeb6b04ab65105ce84901000000, ip address: 10.133.95.40, port: 9201, worker_id: b405a4dc3f267d15554b2747c1b2b568d2699915af195b629306303a, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,376 I 15979 17613] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: b623feeb6b04ab65105ce84901000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,741 I 15979 17613] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: b623feeb6b04ab65105ce84901000000, ip address: 10.133.95.40, port: 9201, worker_id: b405a4dc3f267d15554b2747c1b2b568d2699915af195b629306303a, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:27,581 E 15979 15979] actor_manager.cc:93: Actor handle already exists b623feeb6b04ab65105ce84901000000 | |
| [2021-04-29 17:04:29,806 I 15979 15979] core_worker.cc:2064: Finished executing task 8296953a26b24454ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-07f4c71550cb00f17be428e8c392a1657a8c5bad6a02cd951f7ba6b3_5353.log <== | |
| [2021-04-29 17:04:02,489 E 5353 5353] actor_manager.cc:93: Actor handle already exists 9696e6cac03365bd0d84ef4101000000 | |
| [2021-04-29 17:04:10,308 I 5353 5353] core_worker.cc:2064: Finished executing task 21b37591090b6684ffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:10,318 E 5353 5353] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,318 E 5353 5353] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,402 I 5353 5353] direct_task_transport.cc:47: Creating actor via GCS actor id = : 59828f981bed53bb9b2de4da01000000 | |
| [2021-04-29 17:04:10,447 I 5353 6705] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 59828f981bed53bb9b2de4da01000000, ip address: 10.133.95.40, port: 9205, worker_id: dcdac7fc9d8c440e40685d81c43f2cdd9f62ee476bb15de223f37d1c, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,366 I 5353 6705] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 59828f981bed53bb9b2de4da01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:22,799 I 5353 6705] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 59828f981bed53bb9b2de4da01000000, ip address: 10.133.95.40, port: 9205, worker_id: dcdac7fc9d8c440e40685d81c43f2cdd9f62ee476bb15de223f37d1c, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:25,479 E 5353 5353] actor_manager.cc:93: Actor handle already exists 59828f981bed53bb9b2de4da01000000 | |
| [2021-04-29 17:04:29,742 I 5353 5353] core_worker.cc:2064: Finished executing task b2b9848e005c09f1ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-086dc5c19751ef6bdcf63e60ff181d8530f6947588a28929fc1859c4_9661.log <== | |
| [2021-04-29 17:04:00,925 I 9661 10031] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:00,963 E 9661 9661] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:00,975 I 9661 10031] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,762 E 9661 9661] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:01,835 I 9661 9661] direct_task_transport.cc:47: Creating actor via GCS actor id = : 846e850675ac4071c8ba88f401000000 | |
| [2021-04-29 17:04:01,842 I 9661 10031] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 846e850675ac4071c8ba88f401000000, ip address: 10.134.112.222, port: 9120, worker_id: 45e46a4cde277d8cdcebcb7859908b41852cf17bc800774e20fa8fce, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,318 I 9661 10031] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 846e850675ac4071c8ba88f401000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,389 I 9661 10031] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 846e850675ac4071c8ba88f401000000, ip address: 10.134.112.222, port: 9120, worker_id: 45e46a4cde277d8cdcebcb7859908b41852cf17bc800774e20fa8fce, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,071 E 9661 9661] actor_manager.cc:93: Actor handle already exists 846e850675ac4071c8ba88f401000000 | |
| [2021-04-29 17:04:29,460 I 9661 9661] core_worker.cc:2064: Finished executing task be66ce44f5688dc2ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-09ae33bda4277181721861219a00478d5ce01f7a95c8e38123b628f0_4037.log <== | |
| [2021-04-29 17:04:02,270 E 4037 4037] actor_manager.cc:93: Actor handle already exists 5b8a193e99a5948224efe1c401000000 | |
| [2021-04-29 17:04:10,000 I 4037 4037] core_worker.cc:2064: Finished executing task 9c340df0d07fc3d6ffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:10,019 E 4037 4037] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,019 E 4037 4037] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,105 I 4037 4037] direct_task_transport.cc:47: Creating actor via GCS actor id = : 54d588a7b916aec29c43de3301000000 | |
| [2021-04-29 17:04:10,143 I 4037 4533] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 54d588a7b916aec29c43de3301000000, ip address: 10.134.60.23, port: 9204, worker_id: 7f2b098d51b08bf9f7a4dd7056812f1091b5ee31211ec5b33a76263d, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,361 I 4037 4533] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 54d588a7b916aec29c43de3301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,633 I 4037 4533] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 54d588a7b916aec29c43de3301000000, ip address: 10.134.60.23, port: 9204, worker_id: 7f2b098d51b08bf9f7a4dd7056812f1091b5ee31211ec5b33a76263d, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:24,231 E 4037 4037] actor_manager.cc:93: Actor handle already exists 54d588a7b916aec29c43de3301000000 | |
| [2021-04-29 17:04:29,572 I 4037 4037] core_worker.cc:2064: Finished executing task 08e22ac34a0e610dffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-09bd2ffd9b024d817e42d19c757da81c5730a69285c6e8900d2d364b_9756.log <== | |
| [2021-04-29 17:04:01,393 I 9756 10832] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,416 E 9756 9756] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,421 I 9756 10832] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,010 E 9756 9756] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:02,075 I 9756 9756] direct_task_transport.cc:47: Creating actor via GCS actor id = : d75f0c53368a2c5ac89fb97a01000000 | |
| [2021-04-29 17:04:02,080 I 9756 10832] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: d75f0c53368a2c5ac89fb97a01000000, ip address: 10.134.142.44, port: 9105, worker_id: c8c216f473bda220478e0c65ba291d61848dad38dca129d93f75c3bf, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,543 I 9756 10832] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: d75f0c53368a2c5ac89fb97a01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,391 I 9756 10832] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: d75f0c53368a2c5ac89fb97a01000000, ip address: 10.134.142.44, port: 9105, worker_id: c8c216f473bda220478e0c65ba291d61848dad38dca129d93f75c3bf, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,118 E 9756 9756] actor_manager.cc:93: Actor handle already exists d75f0c53368a2c5ac89fb97a01000000 | |
| [2021-04-29 17:04:29,461 I 9756 9756] core_worker.cc:2064: Finished executing task 5bb8668bfa1fa182ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-09dfdb9db6440e073127fe2db4ec5b941401a8e65bf8de923dc8ae80_9776.log <== | |
| [2021-04-29 17:04:01,493 I 9776 11183] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,514 E 9776 9776] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,522 I 9776 11183] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,102 E 9776 9776] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:02,154 I 9776 9776] direct_task_transport.cc:47: Creating actor via GCS actor id = : 180916becb95f6ab553f38cf01000000 | |
| [2021-04-29 17:04:02,169 I 9776 11183] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 180916becb95f6ab553f38cf01000000, ip address: 10.134.142.16, port: 9115, worker_id: fb751a309898c88e0e14e149b7d04c923d7258b2047125391affc3a5, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,545 I 9776 11183] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 180916becb95f6ab553f38cf01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,391 I 9776 11183] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 180916becb95f6ab553f38cf01000000, ip address: 10.134.142.16, port: 9115, worker_id: fb751a309898c88e0e14e149b7d04c923d7258b2047125391affc3a5, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,116 E 9776 9776] actor_manager.cc:93: Actor handle already exists 180916becb95f6ab553f38cf01000000 | |
| [2021-04-29 17:04:29,468 I 9776 9776] core_worker.cc:2064: Finished executing task 05855996d9c96aeeffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-0af12ee804159637899ef71d95a6ccef44b33b83022133cfd2fb4469_12085.log <== | |
| [2021-04-29 17:04:03,372 I 12085 12457] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,402 E 12085 12085] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:03,409 I 12085 12457] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,866 E 12085 12085] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,957 I 12085 12085] direct_task_transport.cc:47: Creating actor via GCS actor id = : 2f484e006baf743a761d710901000000 | |
| [2021-04-29 17:04:05,966 I 12085 12457] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 2f484e006baf743a761d710901000000, ip address: 10.134.61.25, port: 17901, worker_id: NIL_ID, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,641 I 12085 12457] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 2f484e006baf743a761d710901000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,394 I 12085 12457] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 2f484e006baf743a761d710901000000, ip address: 10.134.61.25, port: 9104, worker_id: 4f786f308f33ed2fc633237266720c562ee5af9f0c841c5078e6e6b4, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,190 E 12085 12085] actor_manager.cc:93: Actor handle already exists 2f484e006baf743a761d710901000000 | |
| [2021-04-29 17:04:29,474 I 12085 12085] core_worker.cc:2064: Finished executing task 77ce85d97ce81168ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-0b464a0731e9c06105b93e6a7a1eecc11e2c2f791bf460bba740b0f0_15870.log <== | |
| [2021-04-29 17:04:07,727 I 15870 16519] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:07,789 E 15870 15870] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:07,849 I 15870 16519] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,496 E 15870 15870] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:10,566 I 15870 15870] direct_task_transport.cc:47: Creating actor via GCS actor id = : 9cd2d9661a9282c29d8fddb601000000 | |
| [2021-04-29 17:04:10,591 I 15870 16519] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 9cd2d9661a9282c29d8fddb601000000, ip address: 10.134.70.227, port: 9214, worker_id: 2a68958361980ed5dd02e5b6989a8e77a46bcb05b2304b594575020e, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,367 I 15870 16519] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 9cd2d9661a9282c29d8fddb601000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,676 I 15870 16519] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 9cd2d9661a9282c29d8fddb601000000, ip address: 10.134.70.227, port: 9214, worker_id: 2a68958361980ed5dd02e5b6989a8e77a46bcb05b2304b594575020e, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:27,512 E 15870 15870] actor_manager.cc:93: Actor handle already exists 9cd2d9661a9282c29d8fddb601000000 | |
| [2021-04-29 17:04:29,803 I 15870 15870] core_worker.cc:2064: Finished executing task 9d66d10169c39cc7ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-0b65ef26de0b2b3b3d872949648c73574396023069611d382dbebf4a_9791.log <== | |
| [2021-04-29 17:04:02,258 I 9791 11372] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,279 E 9791 9791] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:02,286 I 9791 11372] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,409 E 9791 9791] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,488 I 9791 9791] direct_task_transport.cc:47: Creating actor via GCS actor id = : 6da20d2ded8e6a2ce02387cf01000000 | |
| [2021-04-29 17:04:05,516 I 9791 11372] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 6da20d2ded8e6a2ce02387cf01000000, ip address: 10.134.142.16, port: 9108, worker_id: 5b173c8f3b16b98352df68aa44be0ea7e353b703185647b6a3759de2, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,360 I 9791 11372] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 6da20d2ded8e6a2ce02387cf01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,394 I 9791 11372] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 6da20d2ded8e6a2ce02387cf01000000, ip address: 10.134.142.16, port: 9108, worker_id: 5b173c8f3b16b98352df68aa44be0ea7e353b703185647b6a3759de2, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,213 E 9791 9791] actor_manager.cc:93: Actor handle already exists 6da20d2ded8e6a2ce02387cf01000000 | |
| [2021-04-29 17:04:29,478 I 9791 9791] core_worker.cc:2064: Finished executing task f023ff7c966d0fe1ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-0bdcb45c6d762b94ec17dbe5ed33d48cfd8f89a1934cc22470cbb26e_12944.log <== | |
| [2021-04-29 17:04:05,072 I 12944 14468] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,103 E 12944 12944] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,127 I 12944 14468] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,957 E 12944 12944] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:07,166 I 12944 12944] direct_task_transport.cc:47: Creating actor via GCS actor id = : 0b3f9baec1b3198928a8790e01000000 | |
| [2021-04-29 17:04:07,198 I 12944 14468] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 0b3f9baec1b3198928a8790e01000000, ip address: 10.134.142.16, port: 22629, worker_id: NIL_ID, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:16,495 I 12944 14468] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 0b3f9baec1b3198928a8790e01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,398 I 12944 14468] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 0b3f9baec1b3198928a8790e01000000, ip address: 10.134.142.16, port: 9158, worker_id: ba27a0fbdf7ca4b98e2ad966fc8e2a016379f88d9ecd2e24d5155fe4, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,377 E 12944 12944] actor_manager.cc:93: Actor handle already exists 0b3f9baec1b3198928a8790e01000000 | |
| [2021-04-29 17:04:29,518 I 12944 12944] core_worker.cc:2064: Finished executing task 0a9fd6689e283472ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-0be9548f53a2731048d1fc5352e991b646a3bd351f01d2d13442c4e6_7414.log <== | |
| [2021-04-29 17:03:58,015 I 7414 7655] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,051 E 7414 7414] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,060 I 7414 7655] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,676 E 7414 7414] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:58,726 I 7414 7414] direct_task_transport.cc:47: Creating actor via GCS actor id = : 4e1c4fef516ca683b27504c101000000 | |
| [2021-04-29 17:03:58,745 I 7414 7655] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 4e1c4fef516ca683b27504c101000000, ip address: 10.134.142.44, port: 9130, worker_id: 6ede63a8a861c6b15e2f23b734b11e56fa7f3bd93e895acb8135b12c, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,928 I 7414 7655] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 4e1c4fef516ca683b27504c101000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,549 I 7414 7655] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 4e1c4fef516ca683b27504c101000000, ip address: 10.134.142.44, port: 9130, worker_id: 6ede63a8a861c6b15e2f23b734b11e56fa7f3bd93e895acb8135b12c, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:06,107 E 7414 7414] actor_manager.cc:93: Actor handle already exists 4e1c4fef516ca683b27504c101000000 | |
| [2021-04-29 17:04:11,941 I 7414 7414] core_worker.cc:2064: Finished executing task 7e64b65ef84340c9ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-0c39a6f13ef6962965b98be227a2c045d92bd94d954d2a8144f2eac2_4042.log <== | |
| [2021-04-29 17:03:47,134 E 4042 4042] actor_manager.cc:93: Actor handle already exists ff125b7bd2b8fb7149bc306301000000 | |
| [2021-04-29 17:03:47,236 I 4042 4042] core_worker.cc:2064: Finished executing task ffffffffffffffffb76540fe48bba987c71139b801000000, status=OK | |
| [2021-04-29 17:03:47,236 I 4042 4042] direct_actor_transport.cc:497: Actor creation task finished, task_id: ffffffffffffffffb76540fe48bba987c71139b801000000, actor_id: b76540fe48bba987c71139b801000000 | |
| [2021-04-29 17:03:47,241 I 4042 4416] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: b76540fe48bba987c71139b801000000, ip address: 10.133.100.138, port: 9117, worker_id: 0c39a6f13ef6962965b98be227a2c045d92bd94d954d2a8144f2eac2, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:47,242 I 4042 4042] core_worker.cc:2064: Finished executing task 12bcdbc42c0e775db76540fe48bba987c71139b801000000, status=OK | |
| [2021-04-29 17:03:47,278 I 4042 4042] core_worker.cc:2064: Finished executing task e0b6b05163631730b76540fe48bba987c71139b801000000, status=OK | |
| [2021-04-29 17:03:47,416 I 4042 4042] core_worker.cc:2064: Finished executing task c742cb27a017cf65b76540fe48bba987c71139b801000000, status=OK | |
| [2021-04-29 17:03:47,716 I 4042 4042] core_worker.cc:2064: Finished executing task 72c7527ea33742beb76540fe48bba987c71139b801000000, status=OK | |
| [2021-04-29 17:03:47,722 I 4042 4042] core_worker.cc:2064: Finished executing task 5bde02d069be4b14b76540fe48bba987c71139b801000000, status=OK | |
| [2021-04-29 17:03:51,486 I 4042 4042] core_worker.cc:2064: Finished executing task 8919f41e903337f4b76540fe48bba987c71139b801000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-0c887182c8371cd916253c7f9340ff6a986cd44d3078d55b244d6fb4_7580.log <== | |
| [2021-04-29 17:03:58,471 I 7580 9323] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,472 E 7580 7580] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,473 I 7580 9323] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,837 E 7580 7580] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:58,912 I 7580 7580] direct_task_transport.cc:47: Creating actor via GCS actor id = : cd28e3d1cc775c147edc66c301000000 | |
| [2021-04-29 17:03:58,916 I 7580 9323] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: cd28e3d1cc775c147edc66c301000000, ip address: 10.133.95.40, port: 9133, worker_id: 11e7ddc105c7371c85e63bc4219c6d572937298721f0379d81c158f1, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:00,541 I 7580 9323] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: cd28e3d1cc775c147edc66c301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,178 I 7580 9323] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: cd28e3d1cc775c147edc66c301000000, ip address: 10.133.95.40, port: 9133, worker_id: 11e7ddc105c7371c85e63bc4219c6d572937298721f0379d81c158f1, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:07,361 E 7580 7580] actor_manager.cc:93: Actor handle already exists cd28e3d1cc775c147edc66c301000000 | |
| [2021-04-29 17:04:12,648 I 7580 7580] core_worker.cc:2064: Finished executing task 67da3df726604953ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-0ce3511012c14a0bd887341718860aae4a378f20b4735efa732d5399_18051.log <== | |
| [2021-04-29 17:04:10,044 I 18051 19972] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,156 E 18051 18051] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,179 I 18051 19972] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,597 E 18051 18051] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,702 I 18051 18051] direct_task_transport.cc:47: Creating actor via GCS actor id = : 7268113e6ba6c3d1c4e24a3101000000 | |
| [2021-04-29 17:04:11,715 I 18051 19972] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 7268113e6ba6c3d1c4e24a3101000000, ip address: 10.134.142.16, port: 9188, worker_id: d28e673ed7aa65ebca0f58d9c075cc503dc1cb476699a6caeda935dc, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,385 I 18051 19972] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 7268113e6ba6c3d1c4e24a3101000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:27,446 I 18051 19972] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 7268113e6ba6c3d1c4e24a3101000000, ip address: 10.134.142.16, port: 9188, worker_id: d28e673ed7aa65ebca0f58d9c075cc503dc1cb476699a6caeda935dc, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,874 E 18051 18051] actor_manager.cc:93: Actor handle already exists 7268113e6ba6c3d1c4e24a3101000000 | |
| [2021-04-29 17:04:29,855 I 18051 18051] core_worker.cc:2064: Finished executing task 8a1466cf3d2570d1ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-0cf80f64c5e352e7fa344b4acc79b1ac97de3a33f403ec524a2a0aed_12111.log <== | |
| [2021-04-29 17:04:03,492 I 12111 12698] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,522 E 12111 12111] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:03,525 I 12111 12698] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,877 E 12111 12111] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,943 I 12111 12111] direct_task_transport.cc:47: Creating actor via GCS actor id = : 6a270f7e61918d9407f5104101000000 | |
| [2021-04-29 17:04:05,968 I 12111 12698] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 6a270f7e61918d9407f5104101000000, ip address: 10.134.60.23, port: 9107, worker_id: 9c2f78be8502f60738b4235859bab6202bab38f8d24162a2a4b4a9c2, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,236 I 12111 12698] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 6a270f7e61918d9407f5104101000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,395 I 12111 12698] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 6a270f7e61918d9407f5104101000000, ip address: 10.134.60.23, port: 9107, worker_id: 9c2f78be8502f60738b4235859bab6202bab38f8d24162a2a4b4a9c2, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,225 E 12111 12111] actor_manager.cc:93: Actor handle already exists 6a270f7e61918d9407f5104101000000 | |
| [2021-04-29 17:04:29,482 I 12111 12111] core_worker.cc:2064: Finished executing task d95d57740b1afe45ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-0d2f345f162b8cf8b45152e3e930b38ebcbadea1c2430453ab47c956_9678.log <== | |
| [2021-04-29 17:04:01,142 I 9678 10284] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,177 E 9678 9678] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,190 I 9678 10284] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,774 E 9678 9678] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:01,848 I 9678 9678] direct_task_transport.cc:47: Creating actor via GCS actor id = : ca30d93b34fb500229e6bcbe01000000 | |
| [2021-04-29 17:04:01,862 I 9678 10284] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: ca30d93b34fb500229e6bcbe01000000, ip address: 10.134.112.222, port: 9119, worker_id: fc9d1078c71d078b33b4da986c5e3c4fcbdbde1e1fca3d2e4d578740, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,845 I 9678 10284] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: ca30d93b34fb500229e6bcbe01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,389 I 9678 10284] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: ca30d93b34fb500229e6bcbe01000000, ip address: 10.134.112.222, port: 9119, worker_id: fc9d1078c71d078b33b4da986c5e3c4fcbdbde1e1fca3d2e4d578740, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,023 E 9678 9678] actor_manager.cc:93: Actor handle already exists ca30d93b34fb500229e6bcbe01000000 | |
| [2021-04-29 17:04:29,439 I 9678 9678] core_worker.cc:2064: Finished executing task 8cb702c47ae1b146ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-0d6bfc60e0d20cb364062ef723df5520df8e461d8c2fb962d54187d6_17456.log <== | |
| [2021-04-29 17:04:08,967 I 17456 18872] service_based_accessor.cc:579: Received notification for node id = ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, IsAlive = 1 | |
| [2021-04-29 17:04:08,967 I 17456 18872] service_based_accessor.cc:579: Received notification for node id = d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, IsAlive = 1 | |
| [2021-04-29 17:04:08,967 I 17456 18872] service_based_accessor.cc:579: Received notification for node id = b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, IsAlive = 1 | |
| [2021-04-29 17:04:08,967 I 17456 18872] service_based_accessor.cc:579: Received notification for node id = 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, IsAlive = 1 | |
| [2021-04-29 17:04:08,967 I 17456 18872] service_based_accessor.cc:579: Received notification for node id = 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, IsAlive = 1 | |
| [2021-04-29 17:04:10,127 I 17456 18872] core_worker.cc:717: Exit signal received, this process will exit after all outstanding tasks have finished, exit_type=INTENDED_EXIT | |
| [2021-04-29 17:04:10,127 I 17456 17456] raylet_client.cc:129: RayletClient::Disconnect, exit_type=INTENDED_EXIT, has creation_task_exception_pb_bytes=0 | |
| [2021-04-29 17:04:10,128 I 17456 17456] core_worker.cc:308: Removed worker 0d6bfc60e0d20cb364062ef723df5520df8e461d8c2fb962d54187d6 | |
| [2021-04-29 17:04:10,130 I 17456 17456] core_worker.cc:181: Destructing CoreWorkerProcess. pid: 17456 | |
| [2021-04-29 17:04:10,130 I 17456 17456] io_service_pool.cc:47: IOServicePool is stopped. | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-0db52ae61fc2971f9a5abb8710837142a2c95add643727ceffb514cd_9795.log <== | |
| [2021-04-29 17:04:02,170 I 9795 11461] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,197 E 9795 9795] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:02,204 I 9795 11461] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,328 E 9795 9795] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,407 I 9795 9795] direct_task_transport.cc:47: Creating actor via GCS actor id = : b7a79d009393085205709aec01000000 | |
| [2021-04-29 17:04:05,431 I 9795 11461] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: b7a79d009393085205709aec01000000, ip address: 10.133.95.40, port: 9107, worker_id: 9d040519202d287ac7b04b9b32f0cd434950dec8bb4878d6c0c5ebbd, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,534 I 9795 11461] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: b7a79d009393085205709aec01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,393 I 9795 11461] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: b7a79d009393085205709aec01000000, ip address: 10.133.95.40, port: 9107, worker_id: 9d040519202d287ac7b04b9b32f0cd434950dec8bb4878d6c0c5ebbd, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,726 E 9795 9795] actor_manager.cc:93: Actor handle already exists b7a79d009393085205709aec01000000 | |
| [2021-04-29 17:04:29,545 I 9795 9795] core_worker.cc:2064: Finished executing task a2d8c264c8fe745cffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-0e7600fa8dec0883cbb34cc2d1121c8aa3945641837044a5ad25249d_5393.log <== | |
| [2021-04-29 17:03:47,007 E 5393 5393] actor_manager.cc:93: Actor handle already exists ff125b7bd2b8fb7149bc306301000000 | |
| [2021-04-29 17:03:47,109 I 5393 5393] core_worker.cc:2064: Finished executing task ffffffffffffffffa444262b3aa594f55aa2744501000000, status=OK | |
| [2021-04-29 17:03:47,109 I 5393 5393] direct_actor_transport.cc:497: Actor creation task finished, task_id: ffffffffffffffffa444262b3aa594f55aa2744501000000, actor_id: a444262b3aa594f55aa2744501000000 | |
| [2021-04-29 17:03:47,111 I 5393 7132] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: a444262b3aa594f55aa2744501000000, ip address: 10.133.100.138, port: 9248, worker_id: 0e7600fa8dec0883cbb34cc2d1121c8aa3945641837044a5ad25249d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:47,112 I 5393 5393] core_worker.cc:2064: Finished executing task f063de1cf5192626a444262b3aa594f55aa2744501000000, status=OK | |
| [2021-04-29 17:03:47,155 I 5393 5393] core_worker.cc:2064: Finished executing task c3f743e98b333d57a444262b3aa594f55aa2744501000000, status=OK | |
| [2021-04-29 17:03:47,367 I 5393 5393] core_worker.cc:2064: Finished executing task f5a9948b8e06c314a444262b3aa594f55aa2744501000000, status=OK | |
| [2021-04-29 17:03:47,694 I 5393 5393] core_worker.cc:2064: Finished executing task 5ca5102445a6df17a444262b3aa594f55aa2744501000000, status=OK | |
| [2021-04-29 17:03:47,720 I 5393 5393] core_worker.cc:2064: Finished executing task 53826e6586f6f1c7a444262b3aa594f55aa2744501000000, status=OK | |
| [2021-04-29 17:03:51,178 I 5393 5393] core_worker.cc:2064: Finished executing task 13720f634c21a408a444262b3aa594f55aa2744501000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-0eafae254ec91663c8b7be864b42321c296c20ee979394b87c692365_9699.log <== | |
| [2021-04-29 17:04:01,218 I 9699 10536] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,232 E 9699 9699] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,241 I 9699 10536] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,904 E 9699 9699] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:01,956 I 9699 9699] direct_task_transport.cc:47: Creating actor via GCS actor id = : c08c310575422e798343f61c01000000 | |
| [2021-04-29 17:04:01,975 I 9699 10536] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: c08c310575422e798343f61c01000000, ip address: 10.134.60.23, port: 9116, worker_id: ab21a6e3c968f33ae3bd3c35e3e83ee2c0a40a954e4f2d401466629c, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,517 I 9699 10536] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: c08c310575422e798343f61c01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,390 I 9699 10536] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: c08c310575422e798343f61c01000000, ip address: 10.134.60.23, port: 9116, worker_id: ab21a6e3c968f33ae3bd3c35e3e83ee2c0a40a954e4f2d401466629c, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,035 E 9699 9699] actor_manager.cc:93: Actor handle already exists c08c310575422e798343f61c01000000 | |
| [2021-04-29 17:04:29,440 I 9699 9699] core_worker.cc:2064: Finished executing task de0b5ee8246daca0ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-0eb3ec5d7a0f92c24ee3bd56df567882438f10980ee7b679c1b40554_7429.log <== | |
| [2021-04-29 17:03:59,673 I 7429 7958] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,702 E 7429 7429] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,715 I 7429 7958] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:00,330 E 7429 7429] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:00,406 I 7429 7429] direct_task_transport.cc:47: Creating actor via GCS actor id = : 1f0ba69c706df7a4da22251501000000 | |
| [2021-04-29 17:04:00,414 I 7429 7958] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 1f0ba69c706df7a4da22251501000000, ip address: 10.134.61.130, port: 13217, worker_id: NIL_ID, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,732 I 7429 7958] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 1f0ba69c706df7a4da22251501000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,373 I 7429 7958] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1f0ba69c706df7a4da22251501000000, ip address: 10.134.61.130, port: 9112, worker_id: 1583a49e06e3180e23a95287b9fe1ccd1bfc51fceaa6308dc5424285, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,993 E 7429 7429] actor_manager.cc:93: Actor handle already exists 1f0ba69c706df7a4da22251501000000 | |
| [2021-04-29 17:04:29,434 I 7429 7429] core_worker.cc:2064: Finished executing task 907f551b7aeadf58ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-0eb7064847b8976463c3d5aea4327bd9b3e3be17f011f3796fe1607e_9664.log <== | |
| [2021-04-29 17:04:00,945 I 9664 10020] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:00,980 E 9664 9664] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,008 I 9664 10020] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,764 E 9664 9664] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:01,827 I 9664 9664] direct_task_transport.cc:47: Creating actor via GCS actor id = : 6aeba7a6691c5b81574d481a01000000 | |
| [2021-04-29 17:04:01,843 I 9664 10020] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 6aeba7a6691c5b81574d481a01000000, ip address: 10.134.61.130, port: 9111, worker_id: 6f8938b5319053a5fd6874e951ba73c896e3f0b8d4720214b7625d54, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,273 I 9664 10020] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 6aeba7a6691c5b81574d481a01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,389 I 9664 10020] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 6aeba7a6691c5b81574d481a01000000, ip address: 10.134.61.130, port: 9111, worker_id: 6f8938b5319053a5fd6874e951ba73c896e3f0b8d4720214b7625d54, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,003 E 9664 9664] actor_manager.cc:93: Actor handle already exists 6aeba7a6691c5b81574d481a01000000 | |
| [2021-04-29 17:04:29,435 I 9664 9664] core_worker.cc:2064: Finished executing task 22b13f712a003de3ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-0fec1aacb60cad078da76242a078d1ba86d3c46d6ad589b8997f3557_5277.log <== | |
| [2021-04-29 17:03:56,636 E 5277 5277] actor_manager.cc:93: Actor handle already exists ada1f9b78d6b0b3080f7874901000000 | |
| [2021-04-29 17:03:57,962 I 5277 5277] core_worker.cc:2064: Finished executing task 86ba55d80fe2d933ffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:03:57,979 E 5277 5277] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:57,979 E 5277 5277] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,028 I 5277 5277] direct_task_transport.cc:47: Creating actor via GCS actor id = : 7a889d1854055336630f70eb01000000 | |
| [2021-04-29 17:03:58,035 I 5277 6399] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 7a889d1854055336630f70eb01000000, ip address: 10.134.61.25, port: 9139, worker_id: 1bd02a58ebe1c8412b833eea34aa003015633922335d99aa0f04642b, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,071 I 5277 6399] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 7a889d1854055336630f70eb01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,734 I 5277 6399] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 7a889d1854055336630f70eb01000000, ip address: 10.134.61.25, port: 9139, worker_id: 1bd02a58ebe1c8412b833eea34aa003015633922335d99aa0f04642b, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:02,600 E 5277 5277] actor_manager.cc:93: Actor handle already exists 7a889d1854055336630f70eb01000000 | |
| [2021-04-29 17:04:10,508 I 5277 5277] core_worker.cc:2064: Finished executing task 296ac6274c7a4a80ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-10360758c77330d69eb14c3790509332a18f211c9c331676eaa0aa1a_15894.log <== | |
| [2021-04-29 17:04:07,912 I 15894 16768] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,081 E 15894 15894] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,134 I 15894 16768] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,849 E 15894 15894] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:10,942 I 15894 15894] direct_task_transport.cc:47: Creating actor via GCS actor id = : aeb6471aec1692489d29edb401000000 | |
| [2021-04-29 17:04:10,958 I 15894 16768] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: aeb6471aec1692489d29edb401000000, ip address: 10.133.95.40, port: 9202, worker_id: 7c63300160f1a122053097b21231befa9f9a1cadebbc7181bad6d4ee, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,368 I 15894 16768] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: aeb6471aec1692489d29edb401000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:23,955 I 15894 16768] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: aeb6471aec1692489d29edb401000000, ip address: 10.133.95.40, port: 9202, worker_id: 7c63300160f1a122053097b21231befa9f9a1cadebbc7181bad6d4ee, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:26,945 E 15894 15894] actor_manager.cc:93: Actor handle already exists aeb6471aec1692489d29edb401000000 | |
| [2021-04-29 17:04:29,780 I 15894 15894] core_worker.cc:2064: Finished executing task 353b32c083dadfdeffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-105fead3f8f96983da1372b77922f573aef0e71b11d2c2658d2f1839_7444.log <== | |
| [2021-04-29 17:03:58,085 I 7444 7993] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,112 E 7444 7444] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,115 I 7444 7993] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,702 E 7444 7444] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:58,756 I 7444 7444] direct_task_transport.cc:47: Creating actor via GCS actor id = : 40fb22b7f4dfce9192683e5f01000000 | |
| [2021-04-29 17:03:58,772 I 7444 7993] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 40fb22b7f4dfce9192683e5f01000000, ip address: 10.134.70.227, port: 9128, worker_id: 3dd50260f91389bc87634c371a5c24b422f5d255d34fa713d009bcb4, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,220 I 7444 7993] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 40fb22b7f4dfce9192683e5f01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,548 I 7444 7993] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 40fb22b7f4dfce9192683e5f01000000, ip address: 10.134.70.227, port: 9128, worker_id: 3dd50260f91389bc87634c371a5c24b422f5d255d34fa713d009bcb4, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:06,047 E 7444 7444] actor_manager.cc:93: Actor handle already exists 40fb22b7f4dfce9192683e5f01000000 | |
| [2021-04-29 17:04:11,923 I 7444 7444] core_worker.cc:2064: Finished executing task 95a741fc2f5df583ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-11264c4eeac6f1ba968aab86d890fb3d46cbe42d5bbda09375612323_16875.log <== | |
| [2021-04-29 17:04:09,252 I 16875 18687] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,434 E 16875 16875] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,461 I 16875 18687] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,487 E 16875 16875] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,586 I 16875 16875] direct_task_transport.cc:47: Creating actor via GCS actor id = : e6749707cce4a8150c2c1cfb01000000 | |
| [2021-04-29 17:04:11,625 I 16875 18687] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: e6749707cce4a8150c2c1cfb01000000, ip address: 10.133.95.40, port: 9188, worker_id: a822addd7bc88d6b0f93e43a5e0144de82525b1c9f0b3813bc7880fc, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,382 I 16875 18687] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: e6749707cce4a8150c2c1cfb01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:25,442 I 16875 18687] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: e6749707cce4a8150c2c1cfb01000000, ip address: 10.133.95.40, port: 9188, worker_id: a822addd7bc88d6b0f93e43a5e0144de82525b1c9f0b3813bc7880fc, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,411 E 16875 16875] actor_manager.cc:93: Actor handle already exists e6749707cce4a8150c2c1cfb01000000 | |
| [2021-04-29 17:04:29,842 I 16875 16875] core_worker.cc:2064: Finished executing task 337014eceabe82b3ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-11587a562249a27d1a00832f2f28d9d68df61412f71b03bc15d3b005_9888.log <== | |
| [2021-04-29 17:04:02,108 I 9888 11489] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,118 E 9888 9888] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:02,127 I 9888 11489] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,080 E 9888 9888] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,145 I 9888 9888] direct_task_transport.cc:47: Creating actor via GCS actor id = : 89f52366240dbd7cead7f22e01000000 | |
| [2021-04-29 17:04:05,150 I 9888 11489] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 89f52366240dbd7cead7f22e01000000, ip address: 10.134.142.16, port: 9110, worker_id: d277431de16db3756d0604831564b283c20feb9bcccedfcba9bf4095, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,217 I 9888 11489] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 89f52366240dbd7cead7f22e01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,393 I 9888 11489] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 89f52366240dbd7cead7f22e01000000, ip address: 10.134.142.16, port: 9110, worker_id: d277431de16db3756d0604831564b283c20feb9bcccedfcba9bf4095, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,177 E 9888 9888] actor_manager.cc:93: Actor handle already exists 89f52366240dbd7cead7f22e01000000 | |
| [2021-04-29 17:04:29,474 I 9888 9888] core_worker.cc:2064: Finished executing task a3fe2c4c49a68057ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-11d98774d5bdf09b2297011ac7a0fda58cc855ca2acd012601ab5107_5360.log <== | |
| [2021-04-29 17:03:46,187 E 5360 5360] actor_manager.cc:93: Actor handle already exists ff125b7bd2b8fb7149bc306301000000 | |
| [2021-04-29 17:03:46,288 I 5360 5360] core_worker.cc:2064: Finished executing task ffffffffffffffffcb61c47e003c8043bf367b4501000000, status=OK | |
| [2021-04-29 17:03:46,288 I 5360 5360] direct_actor_transport.cc:497: Actor creation task finished, task_id: ffffffffffffffffcb61c47e003c8043bf367b4501000000, actor_id: cb61c47e003c8043bf367b4501000000 | |
| [2021-04-29 17:03:46,290 I 5360 6851] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: cb61c47e003c8043bf367b4501000000, ip address: 10.133.100.138, port: 9233, worker_id: 11d98774d5bdf09b2297011ac7a0fda58cc855ca2acd012601ab5107, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:46,291 I 5360 5360] core_worker.cc:2064: Finished executing task 2de124b59ad79e8acb61c47e003c8043bf367b4501000000, status=OK | |
| [2021-04-29 17:03:46,294 I 5360 5360] core_worker.cc:2064: Finished executing task f497476391650ad1cb61c47e003c8043bf367b4501000000, status=OK | |
| [2021-04-29 17:03:46,311 I 5360 5360] core_worker.cc:2064: Finished executing task e5ea165974027a0ecb61c47e003c8043bf367b4501000000, status=OK | |
| [2021-04-29 17:03:46,595 I 5360 5360] core_worker.cc:2064: Finished executing task d67bc7af81220fe7cb61c47e003c8043bf367b4501000000, status=OK | |
| [2021-04-29 17:03:46,607 I 5360 5360] core_worker.cc:2064: Finished executing task 9a7b763f5466b11dcb61c47e003c8043bf367b4501000000, status=OK | |
| [2021-04-29 17:03:51,510 I 5360 5360] core_worker.cc:2064: Finished executing task 5b32c31cbc7d7079cb61c47e003c8043bf367b4501000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-12c765c87f6208c2d9c056a0df14a06628080040894711c5c0f28abd_9929.log <== | |
| [2021-04-29 17:04:02,004 I 9929 11698] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,038 E 9929 9929] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:02,047 I 9929 11698] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,814 E 9929 9929] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:04,927 I 9929 9929] direct_task_transport.cc:47: Creating actor via GCS actor id = : 25c08703bedf80d79c41b8d001000000 | |
| [2021-04-29 17:04:04,937 I 9929 11698] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 25c08703bedf80d79c41b8d001000000, ip address: 10.134.142.44, port: 12853, worker_id: NIL_ID, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,200 I 9929 11698] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 25c08703bedf80d79c41b8d001000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,395 I 9929 11698] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 25c08703bedf80d79c41b8d001000000, ip address: 10.134.142.44, port: 9142, worker_id: c0e7543376e006259b8097a5959b51d8bccf8ebd27753a9259946b3e, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,231 E 9929 9929] actor_manager.cc:93: Actor handle already exists 25c08703bedf80d79c41b8d001000000 | |
| [2021-04-29 17:04:29,479 I 9929 9929] core_worker.cc:2064: Finished executing task 8ea5af8bac205d97ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-12ef4b1aacc544f2f54a3bc558183b82972d7abbef8a898a69eb4738_7496.log <== | |
| [2021-04-29 17:03:59,419 I 7496 8493] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,440 E 7496 7496] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,445 I 7496 8493] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,755 E 7496 7496] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,821 I 7496 7496] direct_task_transport.cc:47: Creating actor via GCS actor id = : 2de4f064aadcca06280f814d01000000 | |
| [2021-04-29 17:03:59,833 I 7496 8493] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 2de4f064aadcca06280f814d01000000, ip address: 10.134.61.25, port: 9126, worker_id: 6ccf5ec9d3c7cd9e22997a29a7174a398741a5c9b23eb735044934d9, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,726 I 7496 8493] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 2de4f064aadcca06280f814d01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,273 I 7496 8493] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 2de4f064aadcca06280f814d01000000, ip address: 10.134.61.25, port: 9126, worker_id: 6ccf5ec9d3c7cd9e22997a29a7174a398741a5c9b23eb735044934d9, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,872 E 7496 7496] actor_manager.cc:93: Actor handle already exists 2de4f064aadcca06280f814d01000000 | |
| [2021-04-29 17:04:29,397 I 7496 7496] core_worker.cc:2064: Finished executing task 0f90e9db3a32a87cffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-13010aa4e8523afb87a987e2e6be1a357e0e47dcca8951dfd931b021_12102.log <== | |
| [2021-04-29 17:04:03,425 I 12102 12579] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,452 E 12102 12102] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:03,463 I 12102 12579] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,872 E 12102 12102] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,947 I 12102 12102] direct_task_transport.cc:47: Creating actor via GCS actor id = : dd305cd04b8663a1089b43cf01000000 | |
| [2021-04-29 17:04:05,968 I 12102 12579] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: dd305cd04b8663a1089b43cf01000000, ip address: 10.134.61.25, port: 9105, worker_id: 50109cb1818ff9e880dc1e4f500aa31e96519e5a8fad7a3f09e9e9a8, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,169 I 12102 12579] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: dd305cd04b8663a1089b43cf01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,395 I 12102 12579] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: dd305cd04b8663a1089b43cf01000000, ip address: 10.134.61.25, port: 9105, worker_id: 50109cb1818ff9e880dc1e4f500aa31e96519e5a8fad7a3f09e9e9a8, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,260 E 12102 12102] actor_manager.cc:93: Actor handle already exists dd305cd04b8663a1089b43cf01000000 | |
| [2021-04-29 17:04:29,482 I 12102 12102] core_worker.cc:2064: Finished executing task 084d01bc066bafe4ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-1328e59c95259059faab3b0c985ef68f5aefb97b6653d9975434ae95_13003.log <== | |
| [2021-04-29 17:04:05,319 I 13003 14810] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,319 E 13003 13003] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,321 I 13003 14810] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:07,705 E 13003 13003] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:07,910 I 13003 13003] direct_task_transport.cc:47: Creating actor via GCS actor id = : 650f31be08545a076d45afe901000000 | |
| [2021-04-29 17:04:08,002 I 13003 14810] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 650f31be08545a076d45afe901000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:17,540 I 13003 14810] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 650f31be08545a076d45afe901000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,397 I 13003 14810] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 650f31be08545a076d45afe901000000, ip address: 10.134.61.130, port: 9239, worker_id: c5006088bc6efec772410fcc9ce9c5d4345282fa7366ceb240b9f8c4, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,337 E 13003 13003] actor_manager.cc:93: Actor handle already exists 650f31be08545a076d45afe901000000 | |
| [2021-04-29 17:04:29,506 I 13003 13003] core_worker.cc:2064: Finished executing task a92c69bdf93ec9d4ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-13a0d0802a7bc44939c7ff273cb1e0b3e3527107c948b293bdb7d5f3_5213.log <== | |
| [2021-04-29 17:03:56,754 I 5213 5836] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:56,761 E 5213 5213] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:56,782 I 5213 5836] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:57,102 E 5213 5213] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:57,159 I 5213 5213] direct_task_transport.cc:47: Creating actor via GCS actor id = : eca0e70342a5e0493d95455201000000 | |
| [2021-04-29 17:03:57,170 I 5213 5836] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: eca0e70342a5e0493d95455201000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:57,482 I 5213 5836] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: eca0e70342a5e0493d95455201000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,729 I 5213 5836] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: eca0e70342a5e0493d95455201000000, ip address: 10.134.142.16, port: 9142, worker_id: 01b1efc8e4d28a34f0a5ccf9b78d68fbb8b10577d3e8ee7032de73ed, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:02,507 E 5213 5213] actor_manager.cc:93: Actor handle already exists eca0e70342a5e0493d95455201000000 | |
| [2021-04-29 17:04:10,355 I 5213 5213] core_worker.cc:2064: Finished executing task ece9c7923b5a1e47ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-13cc80bedb6fd9a04caaa07dd7a4f8abff264ea5f20903e420c3b33a_5250.log <== | |
| [2021-04-29 17:04:00,709 E 5250 5250] actor_manager.cc:93: Actor handle already exists f802d3c1cb1109986d9f795901000000 | |
| [2021-04-29 17:04:05,833 I 5250 5250] core_worker.cc:2064: Finished executing task 292814f1643a1c4dffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:05,840 E 5250 5250] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,840 E 5250 5250] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,901 I 5250 5250] direct_task_transport.cc:47: Creating actor via GCS actor id = : ac43b19fd039c1767b1339a201000000 | |
| [2021-04-29 17:04:05,948 I 5250 6090] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: ac43b19fd039c1767b1339a201000000, ip address: 10.134.142.16, port: 9106, worker_id: a94095df939f444de2d77cb742ff143f817000679e1454fec1595e50, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,522 I 5250 6090] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: ac43b19fd039c1767b1339a201000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,394 I 5250 6090] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: ac43b19fd039c1767b1339a201000000, ip address: 10.134.142.16, port: 9106, worker_id: a94095df939f444de2d77cb742ff143f817000679e1454fec1595e50, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,758 E 5250 5250] actor_manager.cc:93: Actor handle already exists ac43b19fd039c1767b1339a201000000 | |
| [2021-04-29 17:04:29,550 I 5250 5250] core_worker.cc:2064: Finished executing task 3cbb39f1d1e51558ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-13e09c017affe8a1bc7ddb3b904131cbddedfab7b8af82a5d6d501b0_9728.log <== | |
| [2021-04-29 17:04:01,278 I 9728 10667] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,302 E 9728 9728] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,310 I 9728 10667] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,924 E 9728 9728] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:02,007 I 9728 9728] direct_task_transport.cc:47: Creating actor via GCS actor id = : 57f5028df372d4b719bff2dc01000000 | |
| [2021-04-29 17:04:02,014 I 9728 10667] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 57f5028df372d4b719bff2dc01000000, ip address: 10.134.61.25, port: 9116, worker_id: 753e921db6fa311a5180d67495acf8bc3c53e14fd85656a22a4aeb02, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,540 I 9728 10667] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 57f5028df372d4b719bff2dc01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,391 I 9728 10667] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 57f5028df372d4b719bff2dc01000000, ip address: 10.134.61.25, port: 9116, worker_id: 753e921db6fa311a5180d67495acf8bc3c53e14fd85656a22a4aeb02, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,127 E 9728 9728] actor_manager.cc:93: Actor handle already exists 57f5028df372d4b719bff2dc01000000 | |
| [2021-04-29 17:04:29,470 I 9728 9728] core_worker.cc:2064: Finished executing task f89bc071b5750659ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-13f7f94c12c6b0d60483427d45bd95239bd1888b3527526da5e50c4a_10646.log <== | |
| [2021-04-29 17:04:01,826 I 10646 12000] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,861 E 10646 10646] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,872 I 10646 12000] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,146 E 10646 10646] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:04,248 I 10646 10646] direct_task_transport.cc:47: Creating actor via GCS actor id = : f346bb3f66926468fc1b85f901000000 | |
| [2021-04-29 17:04:04,268 I 10646 12000] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: f346bb3f66926468fc1b85f901000000, ip address: 10.134.60.23, port: 29553, worker_id: NIL_ID, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,750 I 10646 12000] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: f346bb3f66926468fc1b85f901000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,392 I 10646 12000] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: f346bb3f66926468fc1b85f901000000, ip address: 10.134.60.23, port: 9112, worker_id: 86c4e5d46934f087e56a7b6ec39c71c8a2d91c2cfb4d0d6a42ece662, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,154 E 10646 10646] actor_manager.cc:93: Actor handle already exists f346bb3f66926468fc1b85f901000000 | |
| [2021-04-29 17:04:29,467 I 10646 10646] core_worker.cc:2064: Finished executing task a6a42793817100b1ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-140bc00ed296975c117cf80f8220086072ce869bb3ce486d0a44aa9d_14019.log <== | |
| [2021-04-29 17:04:05,543 I 14019 15464] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,572 E 14019 14019] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,581 I 14019 15464] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:07,823 E 14019 14019] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:08,069 I 14019 14019] direct_task_transport.cc:47: Creating actor via GCS actor id = : 099ff9d6467e896e0903dab401000000 | |
| [2021-04-29 17:04:08,145 I 14019 15464] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 099ff9d6467e896e0903dab401000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:18,237 I 14019 15464] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 099ff9d6467e896e0903dab401000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,398 I 14019 15464] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 099ff9d6467e896e0903dab401000000, ip address: 10.134.61.130, port: 9236, worker_id: 4e0c70189257ace74971f00310929335a06ee003fc64555be6ca85e8, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,460 E 14019 14019] actor_manager.cc:93: Actor handle already exists 099ff9d6467e896e0903dab401000000 | |
| [2021-04-29 17:04:29,531 I 14019 14019] core_worker.cc:2064: Finished executing task 1596695187d794b8ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-147a6bbb29ad45eb418a81dc732c1d55c5a321f488cad5a128a9eb1b_16054.log <== | |
| [2021-04-29 17:04:08,744 I 16054 17973] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,802 E 16054 16054] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,829 I 16054 17973] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,198 E 16054 16054] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,301 I 16054 16054] direct_task_transport.cc:47: Creating actor via GCS actor id = : 8044862b4d02321612088c5601000000 | |
| [2021-04-29 17:04:11,327 I 16054 17973] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 8044862b4d02321612088c5601000000, ip address: 10.134.142.16, port: 9204, worker_id: 74ec32abaf6c2817cdc554db72aeda854f77453bbf94ed0d2b578fa3, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,378 I 16054 17973] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 8044862b4d02321612088c5601000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:25,198 I 16054 17973] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8044862b4d02321612088c5601000000, ip address: 10.134.142.16, port: 9204, worker_id: 74ec32abaf6c2817cdc554db72aeda854f77453bbf94ed0d2b578fa3, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,190 E 16054 16054] actor_manager.cc:93: Actor handle already exists 8044862b4d02321612088c5601000000 | |
| [2021-04-29 17:04:29,823 I 16054 16054] core_worker.cc:2064: Finished executing task 01843acbdb861fc3ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-147ac6268b57fd7a4d25e5e34ddaa6a770f0b44066408b0fc3a4359d_15978.log <== | |
| [2021-04-29 17:04:08,478 I 15978 17614] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,583 E 15978 15978] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,624 I 15978 17614] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,123 E 15978 15978] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,222 I 15978 15978] direct_task_transport.cc:47: Creating actor via GCS actor id = : 196f5e5f8d4f20b185fa435601000000 | |
| [2021-04-29 17:04:11,272 I 15978 17614] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 196f5e5f8d4f20b185fa435601000000, ip address: 10.134.61.25, port: 9199, worker_id: cffb7d5f028ffb8d3f3607e0e6c3ee33e97860fafaa9b085a79fb0d6, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,376 I 15978 17614] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 196f5e5f8d4f20b185fa435601000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,801 I 15978 17614] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 196f5e5f8d4f20b185fa435601000000, ip address: 10.134.61.25, port: 9199, worker_id: cffb7d5f028ffb8d3f3607e0e6c3ee33e97860fafaa9b085a79fb0d6, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:27,591 E 15978 15978] actor_manager.cc:93: Actor handle already exists 196f5e5f8d4f20b185fa435601000000 | |
| [2021-04-29 17:04:29,812 I 15978 15978] core_worker.cc:2064: Finished executing task d71825f0ecd9f170ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-1493cb7ce9083bad2417e484357c9f9536e44d718ab62fe932ae0da2_13321.log <== | |
| [2021-04-29 17:04:06,103 I 13321 15040] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,135 E 13321 13321] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:06,148 I 13321 15040] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,825 E 13321 13321] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:08,982 I 13321 13321] direct_task_transport.cc:47: Creating actor via GCS actor id = : 3cdcd2a019fd90f1199ed83c01000000 | |
| [2021-04-29 17:04:08,998 I 13321 15040] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 3cdcd2a019fd90f1199ed83c01000000, ip address: 10.134.142.44, port: 9219, worker_id: 14b18d245aac8bb1b3cf1992ebf75b1eb222cd100b4434b9eaf4107f, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:20,991 I 13321 15040] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 3cdcd2a019fd90f1199ed83c01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:23,971 I 13321 15040] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 3cdcd2a019fd90f1199ed83c01000000, ip address: 10.134.142.44, port: 9219, worker_id: 14b18d245aac8bb1b3cf1992ebf75b1eb222cd100b4434b9eaf4107f, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:26,948 E 13321 13321] actor_manager.cc:93: Actor handle already exists 3cdcd2a019fd90f1199ed83c01000000 | |
| [2021-04-29 17:04:29,771 I 13321 13321] core_worker.cc:2064: Finished executing task 4e368bfdc6e93d08ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-14deb7b5a474baa4ac225e23eb068a83ba336a67179625e1d280382a_9831.log <== | |
| [2021-04-29 17:04:01,569 I 9831 11510] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,589 E 9831 9831] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,595 I 9831 11510] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,104 E 9831 9831] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:02,154 I 9831 9831] direct_task_transport.cc:47: Creating actor via GCS actor id = : 83181ec01a957282445ef13901000000 | |
| [2021-04-29 17:04:02,169 I 9831 11510] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 83181ec01a957282445ef13901000000, ip address: 10.133.95.40, port: 9111, worker_id: feb8cb542764854510a02741fac9a4b7ae194aff117e389da25380ca, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,545 I 9831 11510] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 83181ec01a957282445ef13901000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,391 I 9831 11510] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 83181ec01a957282445ef13901000000, ip address: 10.133.95.40, port: 9111, worker_id: feb8cb542764854510a02741fac9a4b7ae194aff117e389da25380ca, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,145 E 9831 9831] actor_manager.cc:93: Actor handle already exists 83181ec01a957282445ef13901000000 | |
| [2021-04-29 17:04:29,469 I 9831 9831] core_worker.cc:2064: Finished executing task a7d16b9b714ff06bffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-14f5986dbea09ac81b62b2aa5fccfc83cc36eebf7380cbd2eeb93d01_5318.log <== | |
| [2021-04-29 17:03:56,488 E 5318 5318] actor_manager.cc:93: Actor handle already exists 5e830e66f2ab93db3a0fd82e01000000 | |
| [2021-04-29 17:03:57,282 I 5318 5318] core_worker.cc:2064: Finished executing task 32096b404933612bffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:03:57,374 E 5318 5318] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:57,374 E 5318 5318] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:57,472 I 5318 5318] direct_task_transport.cc:47: Creating actor via GCS actor id = : 42b12b2d44f85856033f0ae701000000 | |
| [2021-04-29 17:03:57,488 I 5318 6569] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 42b12b2d44f85856033f0ae701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:57,606 I 5318 6569] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 42b12b2d44f85856033f0ae701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,732 I 5318 6569] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 42b12b2d44f85856033f0ae701000000, ip address: 10.134.142.16, port: 9140, worker_id: 38a608e535e6b8b228bdea057f266ab4c230cc167cd042c5a5aa319c, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:02,571 E 5318 5318] actor_manager.cc:93: Actor handle already exists 42b12b2d44f85856033f0ae701000000 | |
| [2021-04-29 17:04:10,452 I 5318 5318] core_worker.cc:2064: Finished executing task 609c98f0cccaaa7fffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-154d0184970cba3165d916ae054806846fcfa6b8cdfa71b4ed703163_12848.log <== | |
| [2021-04-29 17:04:04,961 I 12848 13927] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,014 E 12848 12848] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,025 I 12848 13927] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,877 E 12848 12848] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:07,042 I 12848 12848] direct_task_transport.cc:47: Creating actor via GCS actor id = : 5f3142a8185355ef9402c38301000000 | |
| [2021-04-29 17:04:07,112 I 12848 13927] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 5f3142a8185355ef9402c38301000000, ip address: 10.134.142.16, port: 22629, worker_id: NIL_ID, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:14,951 I 12848 13927] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 5f3142a8185355ef9402c38301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,398 I 12848 13927] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 5f3142a8185355ef9402c38301000000, ip address: 10.134.142.16, port: 9154, worker_id: 08cccb81613a80d2423f3a9fed9c9805a68a68296a8bf2879b67cb5a, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,403 E 12848 12848] actor_manager.cc:93: Actor handle already exists 5f3142a8185355ef9402c38301000000 | |
| [2021-04-29 17:04:29,517 I 12848 12848] core_worker.cc:2064: Finished executing task 2f8e15f59c69ed50ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-157d02e61814600f6e8321c16198f87238ae28a337efa46b507fde65_12926.log <== | |
| [2021-04-29 17:04:05,000 I 12926 14707] service_based_accessor.cc:579: Received notification for node id = ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, IsAlive = 1 | |
| [2021-04-29 17:04:05,000 I 12926 14707] service_based_accessor.cc:579: Received notification for node id = d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, IsAlive = 1 | |
| [2021-04-29 17:04:05,000 I 12926 14707] service_based_accessor.cc:579: Received notification for node id = b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, IsAlive = 1 | |
| [2021-04-29 17:04:05,000 I 12926 14707] service_based_accessor.cc:579: Received notification for node id = 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, IsAlive = 1 | |
| [2021-04-29 17:04:05,000 I 12926 14707] service_based_accessor.cc:579: Received notification for node id = 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, IsAlive = 1 | |
| [2021-04-29 17:04:05,981 I 12926 14707] core_worker.cc:717: Exit signal received, this process will exit after all outstanding tasks have finished, exit_type=INTENDED_EXIT | |
| [2021-04-29 17:04:05,982 I 12926 12926] raylet_client.cc:129: RayletClient::Disconnect, exit_type=INTENDED_EXIT, has creation_task_exception_pb_bytes=0 | |
| [2021-04-29 17:04:05,982 I 12926 12926] core_worker.cc:308: Removed worker 157d02e61814600f6e8321c16198f87238ae28a337efa46b507fde65 | |
| [2021-04-29 17:04:05,986 I 12926 12926] core_worker.cc:181: Destructing CoreWorkerProcess. pid: 12926 | |
| [2021-04-29 17:04:05,986 I 12926 12926] io_service_pool.cc:47: IOServicePool is stopped. | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-1627e127ac325b5d9b98ad911fc656b4a96cd52a2c86cc12e3da2353_12088.log <== | |
| [2021-04-29 17:04:03,372 I 12088 12499] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,401 E 12088 12088] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:03,409 I 12088 12499] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,866 E 12088 12088] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,964 I 12088 12088] direct_task_transport.cc:47: Creating actor via GCS actor id = : 525b43e890d29e7361642e9001000000 | |
| [2021-04-29 17:04:05,966 I 12088 12499] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 525b43e890d29e7361642e9001000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,087 I 12088 12499] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 525b43e890d29e7361642e9001000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,394 I 12088 12499] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 525b43e890d29e7361642e9001000000, ip address: 10.134.142.16, port: 9103, worker_id: 8bc22a16111aef3d846474294dcc85920d69e99e1eff44acc8b628c9, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,200 E 12088 12088] actor_manager.cc:93: Actor handle already exists 525b43e890d29e7361642e9001000000 | |
| [2021-04-29 17:04:29,478 I 12088 12088] core_worker.cc:2064: Finished executing task a292f1cd53c96ba5ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-16aecfd3e581c17acf458222a814afd56058f4954f59fa3b3dca9c95_4058.log <== | |
| [2021-04-29 17:03:44,975 I 4058 4058] core_worker.cc:2064: Finished executing task 1cb7084d4da835a568cf802dd9de85da271a0fa001000000, status=OK | |
| [2021-04-29 17:04:35,754 I 4058 4058] core_worker.cc:2064: Finished executing task dbf3be25c7d331ad68cf802dd9de85da271a0fa001000000, status=OK | |
| [2021-04-29 17:04:49,019 I 4058 4058] core_worker.cc:2064: Finished executing task db45d7c26df6901c68cf802dd9de85da271a0fa001000000, status=OK | |
| [2021-04-29 17:04:49,160 I 4058 4058] core_worker.cc:2064: Finished executing task f07d0bfaf6ac913668cf802dd9de85da271a0fa001000000, status=OK | |
| [2021-04-29 17:04:51,384 I 4058 4058] core_worker.cc:2064: Finished executing task b90d02785410eee468cf802dd9de85da271a0fa001000000, status=OK | |
| [2021-04-29 17:04:51,678 I 4058 4058] core_worker.cc:2064: Finished executing task 08c640d2d70f60b068cf802dd9de85da271a0fa001000000, status=OK | |
| [2021-04-29 17:04:53,983 I 4058 4058] core_worker.cc:2064: Finished executing task 5dbaf03744f419e268cf802dd9de85da271a0fa001000000, status=OK | |
| [2021-04-29 17:04:55,269 I 4058 4058] core_worker.cc:2064: Finished executing task d85368d883ce5c4468cf802dd9de85da271a0fa001000000, status=OK | |
| [2021-04-29 17:04:55,410 I 4058 4058] core_worker.cc:2064: Finished executing task da50e04884d50a9e68cf802dd9de85da271a0fa001000000, status=OK | |
| [2021-04-29 17:04:57,233 I 4058 4058] core_worker.cc:2064: Finished executing task f3bd3a6e083f036168cf802dd9de85da271a0fa001000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-16d940a39756636166a40c34137b71588fa0ae99b69b9e9c8857114c_4123.log <== | |
| [2021-04-29 18:18:00,119 I 4123 4123] core_worker.cc:2064: Finished executing task ae4383642e9cf078ff125b7bd2b8fb7149bc306301000000, status=OK | |
| [2021-04-29 18:18:05,559 I 4123 4123] core_worker.cc:2064: Finished executing task 1572396c1f556539ff125b7bd2b8fb7149bc306301000000, status=OK | |
| [2021-04-29 18:18:10,841 I 4123 4123] core_worker.cc:2064: Finished executing task e1aa2a6e75df7ac4ff125b7bd2b8fb7149bc306301000000, status=OK | |
| [2021-04-29 18:18:16,472 I 4123 4123] core_worker.cc:2064: Finished executing task 5c674af8f0a1256eff125b7bd2b8fb7149bc306301000000, status=OK | |
| [2021-04-29 18:18:21,625 I 4123 4123] core_worker.cc:2064: Finished executing task 05d4e98c71204bbfff125b7bd2b8fb7149bc306301000000, status=OK | |
| [2021-04-29 18:18:27,284 I 4123 4123] core_worker.cc:2064: Finished executing task 080c7bf2479f4a4fff125b7bd2b8fb7149bc306301000000, status=OK | |
| [2021-04-29 18:18:32,674 I 4123 4123] core_worker.cc:2064: Finished executing task 0d049d7e90b6610aff125b7bd2b8fb7149bc306301000000, status=OK | |
| [2021-04-29 18:18:38,406 I 4123 4123] core_worker.cc:2064: Finished executing task 71f886471aaca7a6ff125b7bd2b8fb7149bc306301000000, status=OK | |
| [2021-04-29 18:18:43,973 I 4123 4123] core_worker.cc:2064: Finished executing task 05f446319d79dffbff125b7bd2b8fb7149bc306301000000, status=OK | |
| [2021-04-29 18:18:49,523 I 4123 4123] core_worker.cc:2064: Finished executing task a83a9931b98157f5ff125b7bd2b8fb7149bc306301000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-17db6238d65fc7dddec2009508a320cee55a013fbf5030a2f231ecb0_15946.log <== | |
| [2021-04-29 17:04:08,063 I 15946 17243] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,378 E 15946 15946] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,412 I 15946 17243] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,999 E 15946 15946] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,100 I 15946 15946] direct_task_transport.cc:47: Creating actor via GCS actor id = : 19500617140bf9b04ac3183c01000000 | |
| [2021-04-29 17:04:11,137 I 15946 17243] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 19500617140bf9b04ac3183c01000000, ip address: 10.134.112.222, port: 9203, worker_id: f696c984cd3d64664ae05367be61ed479620e954763019668ff6576b, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,372 I 15946 17243] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 19500617140bf9b04ac3183c01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,250 I 15946 17243] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 19500617140bf9b04ac3183c01000000, ip address: 10.134.112.222, port: 9203, worker_id: f696c984cd3d64664ae05367be61ed479620e954763019668ff6576b, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:27,076 E 15946 15946] actor_manager.cc:93: Actor handle already exists 19500617140bf9b04ac3183c01000000 | |
| [2021-04-29 17:04:29,779 I 15946 15946] core_worker.cc:2064: Finished executing task 85896d48f505f60cffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-1861a7b0c659e5e7493f3014f78c62edbc0e473f9edba82561480e70_12921.log <== | |
| [2021-04-29 17:04:04,775 I 12921 14427] service_based_accessor.cc:579: Received notification for node id = ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, IsAlive = 1 | |
| [2021-04-29 17:04:04,775 I 12921 14427] service_based_accessor.cc:579: Received notification for node id = d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, IsAlive = 1 | |
| [2021-04-29 17:04:04,775 I 12921 14427] service_based_accessor.cc:579: Received notification for node id = b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, IsAlive = 1 | |
| [2021-04-29 17:04:04,775 I 12921 14427] service_based_accessor.cc:579: Received notification for node id = 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, IsAlive = 1 | |
| [2021-04-29 17:04:04,775 I 12921 14427] service_based_accessor.cc:579: Received notification for node id = 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, IsAlive = 1 | |
| [2021-04-29 17:04:05,982 I 12921 14427] core_worker.cc:717: Exit signal received, this process will exit after all outstanding tasks have finished, exit_type=INTENDED_EXIT | |
| [2021-04-29 17:04:05,982 I 12921 12921] raylet_client.cc:129: RayletClient::Disconnect, exit_type=INTENDED_EXIT, has creation_task_exception_pb_bytes=0 | |
| [2021-04-29 17:04:05,982 I 12921 12921] core_worker.cc:308: Removed worker 1861a7b0c659e5e7493f3014f78c62edbc0e473f9edba82561480e70 | |
| [2021-04-29 17:04:05,985 I 12921 12921] core_worker.cc:181: Destructing CoreWorkerProcess. pid: 12921 | |
| [2021-04-29 17:04:05,985 I 12921 12921] io_service_pool.cc:47: IOServicePool is stopped. | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-195035de3fc159131c6128be66350e1387f75be54f6ebe4624097fde_9769.log <== | |
| [2021-04-29 17:04:02,358 I 9769 11166] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,378 E 9769 9769] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:02,386 I 9769 11166] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,416 E 9769 9769] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,477 I 9769 9769] direct_task_transport.cc:47: Creating actor via GCS actor id = : 5191a46517ad181df5bb04b001000000 | |
| [2021-04-29 17:04:05,494 I 9769 11166] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 5191a46517ad181df5bb04b001000000, ip address: 10.134.142.16, port: 9109, worker_id: 30c996dfec187af8c0a68ba32a06f645404a561d59df6551b14e46ee, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,174 I 9769 11166] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 5191a46517ad181df5bb04b001000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,394 I 9769 11166] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 5191a46517ad181df5bb04b001000000, ip address: 10.134.142.16, port: 9109, worker_id: 30c996dfec187af8c0a68ba32a06f645404a561d59df6551b14e46ee, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,247 E 9769 9769] actor_manager.cc:93: Actor handle already exists 5191a46517ad181df5bb04b001000000 | |
| [2021-04-29 17:04:29,488 I 9769 9769] core_worker.cc:2064: Finished executing task e3a05eb6c77f7107ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-1992a557e9e25b37f190ee17ff65b7476cbea4eeb7ed359c4d45fe11_18605.log <== | |
| [2021-04-29 17:04:10,244 I 18605 20077] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,351 E 18605 18605] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,373 I 18605 20077] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,628 E 18605 18605] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,742 I 18605 18605] direct_task_transport.cc:47: Creating actor via GCS actor id = : 82b59d902486f73eed8c5bde01000000 | |
| [2021-04-29 17:04:11,780 I 18605 20077] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 82b59d902486f73eed8c5bde01000000, ip address: 10.134.61.130, port: 9204, worker_id: 66c82898a4db0f204bd8ccda59dafd25d39f3693a463e2116f05236f, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,388 I 18605 20077] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 82b59d902486f73eed8c5bde01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:28,491 I 18605 20077] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 82b59d902486f73eed8c5bde01000000, ip address: 10.134.61.130, port: 9204, worker_id: 66c82898a4db0f204bd8ccda59dafd25d39f3693a463e2116f05236f, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:29,411 E 18605 18605] actor_manager.cc:93: Actor handle already exists 82b59d902486f73eed8c5bde01000000 | |
| [2021-04-29 17:04:29,873 I 18605 18605] core_worker.cc:2064: Finished executing task 728de8ca6ce077d9ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-19ddd03000ab1f74d54f6589c8e127ce07d5b203814ef9f918ef9407_17798.log <== | |
| [2021-04-29 17:04:09,621 I 17798 19290] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,767 E 17798 17798] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,790 I 17798 19290] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,575 E 17798 17798] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,704 I 17798 17798] direct_task_transport.cc:47: Creating actor via GCS actor id = : 37d92d87c276876c06ccd2ed01000000 | |
| [2021-04-29 17:04:11,755 I 17798 19290] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 37d92d87c276876c06ccd2ed01000000, ip address: 10.134.61.130, port: 9201, worker_id: a71ef005e22bdeb61ebe6a38e4803740885933bf2de8745b6bf0ed95, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,386 I 17798 19290] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 37d92d87c276876c06ccd2ed01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:28,201 I 17798 19290] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 37d92d87c276876c06ccd2ed01000000, ip address: 10.134.61.130, port: 9201, worker_id: a71ef005e22bdeb61ebe6a38e4803740885933bf2de8745b6bf0ed95, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:29,013 E 17798 17798] actor_manager.cc:93: Actor handle already exists 37d92d87c276876c06ccd2ed01000000 | |
| [2021-04-29 17:04:29,867 I 17798 17798] core_worker.cc:2064: Finished executing task 5d3d96455661ee08ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-1a3d6c8933b2662e3e8955b3c3becebdc2861ec5df5ae04d9c6172ea_3993.log <== | |
| [2021-04-29 17:03:44,983 I 3993 3993] core_worker.cc:2064: Finished executing task ed70558e8dfedd5636e07b250e732fb8fecaaff501000000, status=OK | |
| [2021-04-29 17:04:45,971 I 3993 3993] core_worker.cc:2064: Finished executing task 8f5fe351b9108d9f36e07b250e732fb8fecaaff501000000, status=OK | |
| [2021-04-29 17:04:49,924 I 3993 3993] core_worker.cc:2064: Finished executing task 50f6bd3c2dc689b336e07b250e732fb8fecaaff501000000, status=OK | |
| [2021-04-29 17:04:49,956 I 3993 3993] core_worker.cc:2064: Finished executing task 54a26dadaf38f63236e07b250e732fb8fecaaff501000000, status=OK | |
| [2021-04-29 17:04:51,311 I 3993 3993] core_worker.cc:2064: Finished executing task b8847a6cc5d0cf9036e07b250e732fb8fecaaff501000000, status=OK | |
| [2021-04-29 17:04:51,736 I 3993 3993] core_worker.cc:2064: Finished executing task cf5814c2439f591136e07b250e732fb8fecaaff501000000, status=OK | |
| [2021-04-29 17:04:53,444 I 3993 3993] core_worker.cc:2064: Finished executing task 23150d45a683c77836e07b250e732fb8fecaaff501000000, status=OK | |
| [2021-04-29 17:04:55,181 I 3993 3993] core_worker.cc:2064: Finished executing task 0d3828e396404ba936e07b250e732fb8fecaaff501000000, status=OK | |
| [2021-04-29 17:04:55,867 I 3993 3993] core_worker.cc:2064: Finished executing task c73d8332b889771536e07b250e732fb8fecaaff501000000, status=OK | |
| [2021-04-29 17:04:57,473 I 3993 3993] core_worker.cc:2064: Finished executing task 8dd928b80366a48036e07b250e732fb8fecaaff501000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-1b2ca7939179abb668299735c44c791af2987e68b22a1176634c37ca_9806.log <== | |
| [2021-04-29 17:04:02,196 I 9806 11414] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,222 E 9806 9806] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:02,230 I 9806 11414] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,345 E 9806 9806] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,423 I 9806 9806] direct_task_transport.cc:47: Creating actor via GCS actor id = : c6d65d1127d31d1fcc3b02fe01000000 | |
| [2021-04-29 17:04:05,431 I 9806 11414] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: c6d65d1127d31d1fcc3b02fe01000000, ip address: 10.134.61.25, port: 9107, worker_id: 30ea9e02e57d73dd069644e1313bca4b8ab158d6efd08ebe0c89b6cc, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,707 I 9806 11414] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: c6d65d1127d31d1fcc3b02fe01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,395 I 9806 11414] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: c6d65d1127d31d1fcc3b02fe01000000, ip address: 10.134.61.25, port: 9107, worker_id: 30ea9e02e57d73dd069644e1313bca4b8ab158d6efd08ebe0c89b6cc, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,229 E 9806 9806] actor_manager.cc:93: Actor handle already exists c6d65d1127d31d1fcc3b02fe01000000 | |
| [2021-04-29 17:04:29,485 I 9806 9806] core_worker.cc:2064: Finished executing task 26bdadc291655b54ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-1ba988381490bcbc851425d5789b202507c61f108b5761dde5861452_9877.log <== | |
| [2021-04-29 17:04:02,154 I 9877 11462] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,188 E 9877 9877] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:02,195 I 9877 11462] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,326 E 9877 9877] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,370 I 9877 9877] direct_task_transport.cc:47: Creating actor via GCS actor id = : 8fa82cba55942a16056a355001000000 | |
| [2021-04-29 17:04:05,386 I 9877 11462] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 8fa82cba55942a16056a355001000000, ip address: 10.134.60.23, port: 9110, worker_id: 885f8c39573b673e3d5d8bee1ca8b5e267ecc77032c517bb307682b0, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,458 I 9877 11462] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 8fa82cba55942a16056a355001000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,393 I 9877 11462] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8fa82cba55942a16056a355001000000, ip address: 10.134.60.23, port: 9110, worker_id: 885f8c39573b673e3d5d8bee1ca8b5e267ecc77032c517bb307682b0, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,179 E 9877 9877] actor_manager.cc:93: Actor handle already exists 8fa82cba55942a16056a355001000000 | |
| [2021-04-29 17:04:29,480 I 9877 9877] core_worker.cc:2064: Finished executing task 931f0c8b2b7bb6c9ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-1bb6f7970ec83338593d4c425e68922ad3add9d412e5aebccc2c0fd3_5355.log <== | |
| [2021-04-29 17:03:46,424 I 5355 5355] core_worker.cc:2064: Finished executing task ca34645a4de5324fddc5f3edeb1f1e5ab18c056501000000, status=OK | |
| [2021-04-29 17:04:48,791 I 5355 5355] core_worker.cc:2064: Finished executing task 4c1feaa244043946ddc5f3edeb1f1e5ab18c056501000000, status=OK | |
| [2021-04-29 17:04:49,939 I 5355 5355] core_worker.cc:2064: Finished executing task 5dfce9af7ebca6d7ddc5f3edeb1f1e5ab18c056501000000, status=OK | |
| [2021-04-29 17:04:50,263 I 5355 5355] core_worker.cc:2064: Finished executing task f288102aba72b964ddc5f3edeb1f1e5ab18c056501000000, status=OK | |
| [2021-04-29 17:04:52,263 I 5355 5355] core_worker.cc:2064: Finished executing task 5801760b80360c42ddc5f3edeb1f1e5ab18c056501000000, status=OK | |
| [2021-04-29 17:04:53,610 I 5355 5355] core_worker.cc:2064: Finished executing task ec93536d25d9abcaddc5f3edeb1f1e5ab18c056501000000, status=OK | |
| [2021-04-29 17:04:55,090 I 5355 5355] core_worker.cc:2064: Finished executing task a45bb995ecfafe17ddc5f3edeb1f1e5ab18c056501000000, status=OK | |
| [2021-04-29 17:04:55,593 I 5355 5355] core_worker.cc:2064: Finished executing task 9ac98b3919322ee7ddc5f3edeb1f1e5ab18c056501000000, status=OK | |
| [2021-04-29 17:04:56,172 I 5355 5355] core_worker.cc:2064: Finished executing task 23caef5498105e32ddc5f3edeb1f1e5ab18c056501000000, status=OK | |
| [2021-04-29 17:04:58,259 I 5355 5355] core_worker.cc:2064: Finished executing task 822b2d64f9093b3eddc5f3edeb1f1e5ab18c056501000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-1c046dc5cc4aede2853b1bfa8df13cad58c846d6bb1d05156ec48a9d_10108.log <== | |
| [2021-04-29 17:04:01,719 I 10108 11859] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,721 E 10108 10108] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,722 I 10108 11859] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,299 E 10108 10108] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:02,366 I 10108 10108] direct_task_transport.cc:47: Creating actor via GCS actor id = : 1ec721fe9b95ff869682f7f501000000 | |
| [2021-04-29 17:04:02,377 I 10108 11859] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 1ec721fe9b95ff869682f7f501000000, ip address: 10.134.142.44, port: 9103, worker_id: 24c3a8863675ca16bc30536e4da8502fdf99d1997e64a251afea89f0, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,547 I 10108 11859] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 1ec721fe9b95ff869682f7f501000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,392 I 10108 11859] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ec721fe9b95ff869682f7f501000000, ip address: 10.134.142.44, port: 9103, worker_id: 24c3a8863675ca16bc30536e4da8502fdf99d1997e64a251afea89f0, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,720 E 10108 10108] actor_manager.cc:93: Actor handle already exists 1ec721fe9b95ff869682f7f501000000 | |
| [2021-04-29 17:04:29,542 I 10108 10108] core_worker.cc:2064: Finished executing task d87372cc01cf3864ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-1c1fa8f5af6b356958a59ad02c6d27da6658c4444ab7cd8031c4c230_13198.log <== | |
| [2021-04-29 17:04:05,344 I 13198 14856] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,345 E 13198 13198] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,373 I 13198 14856] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:07,793 E 13198 13198] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:07,968 I 13198 13198] direct_task_transport.cc:47: Creating actor via GCS actor id = : 2eda8db04aa80d2621da7eb201000000 | |
| [2021-04-29 17:04:08,064 I 13198 14856] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 2eda8db04aa80d2621da7eb201000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:17,747 I 13198 14856] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 2eda8db04aa80d2621da7eb201000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,400 I 13198 14856] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 2eda8db04aa80d2621da7eb201000000, ip address: 10.134.142.16, port: 9164, worker_id: 91d0010c707bb389290e2548c4a2b33139dc6b4b6757d50e15fb77f1, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,491 E 13198 13198] actor_manager.cc:93: Actor handle already exists 2eda8db04aa80d2621da7eb201000000 | |
| [2021-04-29 17:04:29,540 I 13198 13198] core_worker.cc:2064: Finished executing task 1fc8b39ff8367945ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-1c225ea19ec5fa24f70ddd7eac44687a04c35adbbae4a1b2098605d8_15882.log <== | |
| [2021-04-29 17:04:07,851 I 15882 16617] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,021 E 15882 15882] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,061 I 15882 16617] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,758 E 15882 15882] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:10,828 I 15882 15882] direct_task_transport.cc:47: Creating actor via GCS actor id = : f78c2dc52ef241e59163432b01000000 | |
| [2021-04-29 17:04:10,855 I 15882 16617] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: f78c2dc52ef241e59163432b01000000, ip address: 10.134.61.130, port: 9221, worker_id: 270f6813ee0f2a4b3247d17a5848149b8b04dd3ea7e36cc60fea2111, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,368 I 15882 16617] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: f78c2dc52ef241e59163432b01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,885 I 15882 16617] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: f78c2dc52ef241e59163432b01000000, ip address: 10.134.61.130, port: 9221, worker_id: 270f6813ee0f2a4b3247d17a5848149b8b04dd3ea7e36cc60fea2111, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:27,598 E 15882 15882] actor_manager.cc:93: Actor handle already exists f78c2dc52ef241e59163432b01000000 | |
| [2021-04-29 17:04:29,818 I 15882 15882] core_worker.cc:2064: Finished executing task 96a378bff7a371efffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-1c24b29a8665f709061cbc7152f6f1da380d99f64ce9f75c532e2d65_17891.log <== | |
| [2021-04-29 17:04:09,379 I 17891 19452] service_based_accessor.cc:579: Received notification for node id = ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, IsAlive = 1 | |
| [2021-04-29 17:04:09,379 I 17891 19452] service_based_accessor.cc:579: Received notification for node id = d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, IsAlive = 1 | |
| [2021-04-29 17:04:09,379 I 17891 19452] service_based_accessor.cc:579: Received notification for node id = b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, IsAlive = 1 | |
| [2021-04-29 17:04:09,379 I 17891 19452] service_based_accessor.cc:579: Received notification for node id = 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, IsAlive = 1 | |
| [2021-04-29 17:04:09,379 I 17891 19452] service_based_accessor.cc:579: Received notification for node id = 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, IsAlive = 1 | |
| [2021-04-29 17:04:10,532 I 17891 19452] core_worker.cc:717: Exit signal received, this process will exit after all outstanding tasks have finished, exit_type=INTENDED_EXIT | |
| [2021-04-29 17:04:10,532 I 17891 17891] raylet_client.cc:129: RayletClient::Disconnect, exit_type=INTENDED_EXIT, has creation_task_exception_pb_bytes=0 | |
| [2021-04-29 17:04:10,533 I 17891 17891] core_worker.cc:308: Removed worker 1c24b29a8665f709061cbc7152f6f1da380d99f64ce9f75c532e2d65 | |
| [2021-04-29 17:04:10,535 I 17891 17891] core_worker.cc:181: Destructing CoreWorkerProcess. pid: 17891 | |
| [2021-04-29 17:04:10,535 I 17891 17891] io_service_pool.cc:47: IOServicePool is stopped. | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-1c78acbfec670241208d0775c5294b9d001194b5486d5cbffa5b3e3f_17896.log <== | |
| [2021-04-29 17:04:09,790 I 17896 19464] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,870 E 17896 17896] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,892 I 17896 19464] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,579 E 17896 17896] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,704 I 17896 17896] direct_task_transport.cc:47: Creating actor via GCS actor id = : f6b930537a9ef64fede538b701000000 | |
| [2021-04-29 17:04:11,756 I 17896 19464] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: f6b930537a9ef64fede538b701000000, ip address: 10.134.112.222, port: 9179, worker_id: f37016dbc19816aa1731da763e50c58ea2de402bd52e4612b8d06022, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,386 I 17896 19464] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: f6b930537a9ef64fede538b701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:27,531 I 17896 19464] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: f6b930537a9ef64fede538b701000000, ip address: 10.134.112.222, port: 9179, worker_id: f37016dbc19816aa1731da763e50c58ea2de402bd52e4612b8d06022, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,891 E 17896 17896] actor_manager.cc:93: Actor handle already exists f6b930537a9ef64fede538b701000000 | |
| [2021-04-29 17:04:29,855 I 17896 17896] core_worker.cc:2064: Finished executing task 352e2fdc39b44a9dffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-1d1f82a9b55f1e2adf278626a85beae0437cb2eab001543ad26378a1_5297.log <== | |
| [2021-04-29 17:03:46,200 E 5297 5297] actor_manager.cc:93: Actor handle already exists ff125b7bd2b8fb7149bc306301000000 | |
| [2021-04-29 17:03:46,301 I 5297 5297] core_worker.cc:2064: Finished executing task ffffffffffffffff66d7f4c12a7a4712590d8be201000000, status=OK | |
| [2021-04-29 17:03:46,301 I 5297 5297] direct_actor_transport.cc:497: Actor creation task finished, task_id: ffffffffffffffff66d7f4c12a7a4712590d8be201000000, actor_id: 66d7f4c12a7a4712590d8be201000000 | |
| [2021-04-29 17:03:46,303 I 5297 6427] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 66d7f4c12a7a4712590d8be201000000, ip address: 10.133.100.138, port: 9205, worker_id: 1d1f82a9b55f1e2adf278626a85beae0437cb2eab001543ad26378a1, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:46,304 I 5297 5297] core_worker.cc:2064: Finished executing task 1a424e93ca0d6b4a66d7f4c12a7a4712590d8be201000000, status=OK | |
| [2021-04-29 17:03:46,307 I 5297 5297] core_worker.cc:2064: Finished executing task 0951e2d98fb39b5a66d7f4c12a7a4712590d8be201000000, status=OK | |
| [2021-04-29 17:03:46,324 I 5297 5297] core_worker.cc:2064: Finished executing task 2a1c1ec0a6b6137466d7f4c12a7a4712590d8be201000000, status=OK | |
| [2021-04-29 17:03:46,596 I 5297 5297] core_worker.cc:2064: Finished executing task bb9647252cc6e95366d7f4c12a7a4712590d8be201000000, status=OK | |
| [2021-04-29 17:03:46,608 I 5297 5297] core_worker.cc:2064: Finished executing task cae7743e29ceabee66d7f4c12a7a4712590d8be201000000, status=OK | |
| [2021-04-29 17:03:51,613 I 5297 5297] core_worker.cc:2064: Finished executing task 759b0186be5eb1bf66d7f4c12a7a4712590d8be201000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-1d272df9899b24fc27957b9c16c0585cccc9889f2bbf8cd16fd77061_9674.log <== | |
| [2021-04-29 17:04:01,097 I 9674 10195] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,137 E 9674 9674] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,157 I 9674 10195] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,772 E 9674 9674] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:01,844 I 9674 9674] direct_task_transport.cc:47: Creating actor via GCS actor id = : 2d29b539f3b964b4ea604fe401000000 | |
| [2021-04-29 17:04:01,862 I 9674 10195] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 2d29b539f3b964b4ea604fe401000000, ip address: 10.134.61.25, port: 9118, worker_id: 183c2aecbf3da7b6628405e98a2ced514590a8f7913b288666d65d22, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,758 I 9674 10195] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 2d29b539f3b964b4ea604fe401000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,390 I 9674 10195] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 2d29b539f3b964b4ea604fe401000000, ip address: 10.134.61.25, port: 9118, worker_id: 183c2aecbf3da7b6628405e98a2ced514590a8f7913b288666d65d22, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,019 E 9674 9674] actor_manager.cc:93: Actor handle already exists 2d29b539f3b964b4ea604fe401000000 | |
| [2021-04-29 17:04:29,437 I 9674 9674] core_worker.cc:2064: Finished executing task 21d1c42d60cd641effffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-1e0ee5222527c7bd91c4395b3b4c50ad891e4039501578a02cc3b620_7524.log <== | |
| [2021-04-29 17:03:59,334 I 7524 8544] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,350 E 7524 7524] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,356 I 7524 8544] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,733 E 7524 7524] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,813 I 7524 8544] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 4c7b120315929cf009d0655301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,815 I 7524 7524] direct_task_transport.cc:47: Creating actor via GCS actor id = : 4c7b120315929cf009d0655301000000 | |
| [2021-04-29 17:04:01,725 I 7524 8544] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 4c7b120315929cf009d0655301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,004 I 7524 8544] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 4c7b120315929cf009d0655301000000, ip address: 10.134.112.222, port: 9129, worker_id: c100dc8104c164ca7f5e6e074ed33143c149d39378fafb91466e3341, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,587 E 7524 7524] actor_manager.cc:93: Actor handle already exists 4c7b120315929cf009d0655301000000 | |
| [2021-04-29 17:04:29,071 I 7524 7524] core_worker.cc:2064: Finished executing task 80c4ba4dac9f70eeffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-1f07febef1c96d4b098931d89f89be5a2c58b7f5884bcf209db38eb2_12108.log <== | |
| [2021-04-29 17:04:03,452 I 12108 12627] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,481 E 12108 12108] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:03,491 I 12108 12627] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,874 E 12108 12108] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,947 I 12108 12108] direct_task_transport.cc:47: Creating actor via GCS actor id = : a5b8fc65cfdac9448df1d01d01000000 | |
| [2021-04-29 17:04:05,968 I 12108 12627] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: a5b8fc65cfdac9448df1d01d01000000, ip address: 10.134.60.23, port: 9105, worker_id: 7b735b3fd6f98e7ab5ee476224886ddb908c7cdf1fec1654416c56d0, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,304 I 12108 12627] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: a5b8fc65cfdac9448df1d01d01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,395 I 12108 12627] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: a5b8fc65cfdac9448df1d01d01000000, ip address: 10.134.60.23, port: 9105, worker_id: 7b735b3fd6f98e7ab5ee476224886ddb908c7cdf1fec1654416c56d0, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,262 E 12108 12108] actor_manager.cc:93: Actor handle already exists a5b8fc65cfdac9448df1d01d01000000 | |
| [2021-04-29 17:04:29,492 I 12108 12108] core_worker.cc:2064: Finished executing task ef1c2c7e448d258effffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-1f166c1eed79f94b36da8adb1148b2c6861b296ac5de09bf489c2962_9725.log <== | |
| [2021-04-29 17:04:01,243 I 9725 10600] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,255 E 9725 9725] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,264 I 9725 10600] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,904 E 9725 9725] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:01,956 I 9725 9725] direct_task_transport.cc:47: Creating actor via GCS actor id = : 991452d2c10a4a91395de85801000000 | |
| [2021-04-29 17:04:01,975 I 9725 10600] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 991452d2c10a4a91395de85801000000, ip address: 10.134.61.25, port: 9117, worker_id: bc04f49402510fdc72d7d4b429a5ca9767f9d02c3e0069fc81bfb6d1, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,353 I 9725 10600] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 991452d2c10a4a91395de85801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,391 I 9725 10600] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 991452d2c10a4a91395de85801000000, ip address: 10.134.61.25, port: 9117, worker_id: bc04f49402510fdc72d7d4b429a5ca9767f9d02c3e0069fc81bfb6d1, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,090 E 9725 9725] actor_manager.cc:93: Actor handle already exists 991452d2c10a4a91395de85801000000 | |
| [2021-04-29 17:04:29,455 I 9725 9725] core_worker.cc:2064: Finished executing task 6462a8ca93c73a86ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-1f56c2d8b046c77fafcf099956cc90007178fd439723da26aa7e8509_12735.log <== | |
| [2021-04-29 17:04:04,520 I 12735 13236] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,584 E 12735 12735] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:04,616 I 12735 13236] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,509 E 12735 12735] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:06,627 I 12735 12735] direct_task_transport.cc:47: Creating actor via GCS actor id = : 1d1287254590d5ed7ae2c49d01000000 | |
| [2021-04-29 17:04:06,676 I 12735 13236] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 1d1287254590d5ed7ae2c49d01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:13,296 I 12735 13236] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 1d1287254590d5ed7ae2c49d01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,395 I 12735 13236] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1d1287254590d5ed7ae2c49d01000000, ip address: 10.134.112.222, port: 9101, worker_id: f23592dd1b9d9b3afffad9e1242cb5858e56e34e16637a14d360fedc, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,255 E 12735 12735] actor_manager.cc:93: Actor handle already exists 1d1287254590d5ed7ae2c49d01000000 | |
| [2021-04-29 17:04:29,487 I 12735 12735] core_worker.cc:2064: Finished executing task 82b38426e32c48eaffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-205984af2517bb7ceef7a313bb192c1d170f0bc3a05b167b2accd0c8_7537.log <== | |
| [2021-04-29 17:03:59,074 I 7537 8873] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,099 E 7537 7537] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,104 I 7537 8873] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,635 E 7537 7537] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,685 I 7537 7537] direct_task_transport.cc:47: Creating actor via GCS actor id = : 4ade7c03e09c4465ffe6038701000000 | |
| [2021-04-29 17:03:59,701 I 7537 8873] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 4ade7c03e09c4465ffe6038701000000, ip address: 10.134.61.130, port: 9119, worker_id: 734f15a0247050e3cafbde348756ee518bdf6311ac3795af33c6ed73, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,721 I 7537 8873] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 4ade7c03e09c4465ffe6038701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:19,263 I 7537 8873] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 4ade7c03e09c4465ffe6038701000000, ip address: 10.134.61.130, port: 9119, worker_id: 734f15a0247050e3cafbde348756ee518bdf6311ac3795af33c6ed73, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,057 E 7537 7537] actor_manager.cc:93: Actor handle already exists 4ade7c03e09c4465ffe6038701000000 | |
| [2021-04-29 17:04:28,912 I 7537 7537] core_worker.cc:2064: Finished executing task cc21f3fc8bf01331ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-20e03facb7b49bf188449eb9c267c000eb1834341082e2e11cb621cd_5258.log <== | |
| [2021-04-29 17:04:01,876 E 5258 5258] actor_manager.cc:93: Actor handle already exists f2f703ba93f95b5203e516f001000000 | |
| [2021-04-29 17:04:09,106 I 5258 5258] core_worker.cc:2064: Finished executing task a182654db2a9cf17ffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:09,127 E 5258 5258] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,127 E 5258 5258] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,216 I 5258 5258] direct_task_transport.cc:47: Creating actor via GCS actor id = : 34e83cfb267193b6ef99452e01000000 | |
| [2021-04-29 17:04:09,251 I 5258 6065] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 34e83cfb267193b6ef99452e01000000, ip address: 10.134.142.44, port: 9215, worker_id: 7295b8fb1e0a24bcc26ce9c11c3d3a56a930f50c221554b00604536b, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,209 I 5258 6065] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 34e83cfb267193b6ef99452e01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,108 I 5258 6065] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 34e83cfb267193b6ef99452e01000000, ip address: 10.134.142.44, port: 9215, worker_id: 7295b8fb1e0a24bcc26ce9c11c3d3a56a930f50c221554b00604536b, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:26,990 E 5258 5258] actor_manager.cc:93: Actor handle already exists 34e83cfb267193b6ef99452e01000000 | |
| [2021-04-29 17:04:29,775 I 5258 5258] core_worker.cc:2064: Finished executing task b28ee076a5e1b3efffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-21003954f6dc9716b21b5852c1d2fe3ba8c83f19f4555c9ee047b85a_7413.log <== | |
| [2021-04-29 17:03:58,036 I 7413 7626] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,065 E 7413 7413] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,070 I 7413 7626] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,677 E 7413 7413] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:58,742 I 7413 7413] direct_task_transport.cc:47: Creating actor via GCS actor id = : 5382bf7d1df05de39fca8e6501000000 | |
| [2021-04-29 17:03:58,754 I 7413 7626] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 5382bf7d1df05de39fca8e6501000000, ip address: 10.134.70.227, port: 9129, worker_id: 3cb0f7cd4956983fb7c61114053a7bf498dca5ff1d1ecd3f0af88fe3, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,110 I 7413 7626] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 5382bf7d1df05de39fca8e6501000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,548 I 7413 7626] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 5382bf7d1df05de39fca8e6501000000, ip address: 10.134.70.227, port: 9129, worker_id: 3cb0f7cd4956983fb7c61114053a7bf498dca5ff1d1ecd3f0af88fe3, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:06,078 E 7413 7413] actor_manager.cc:93: Actor handle already exists 5382bf7d1df05de39fca8e6501000000 | |
| [2021-04-29 17:04:11,942 I 7413 7413] core_worker.cc:2064: Finished executing task 83dbb8fe88e98d6bffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-224d47981c03704a332f55dbd2b0cb6b2a61f9ad905d9fd9a388a4ac_13894.log <== | |
| [2021-04-29 17:04:05,582 I 13894 15472] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,597 E 13894 13894] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,608 I 13894 15472] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:07,867 E 13894 13894] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:08,052 I 13894 13894] direct_task_transport.cc:47: Creating actor via GCS actor id = : 8253bf814c999d13b2d2d60b01000000 | |
| [2021-04-29 17:04:08,148 I 13894 15472] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 8253bf814c999d13b2d2d60b01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:17,815 I 13894 15472] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 8253bf814c999d13b2d2d60b01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,400 I 13894 15472] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8253bf814c999d13b2d2d60b01000000, ip address: 10.134.112.222, port: 9163, worker_id: 9f11cd718a9bd1f81c4b8cbe2249ffd20857c3dfe4ca51e13094f4b2, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,475 E 13894 13894] actor_manager.cc:93: Actor handle already exists 8253bf814c999d13b2d2d60b01000000 | |
| [2021-04-29 17:04:29,540 I 13894 13894] core_worker.cc:2064: Finished executing task 766fac2bd4e30c52ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-225021f0da0a88f05c2bf394bda15216f9e85f2b043c0c7a5abee8f8_9698.log <== | |
| [2021-04-29 17:04:01,143 I 9698 10376] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,161 E 9698 9698] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,191 I 9698 10376] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,773 E 9698 9698] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:01,869 I 9698 9698] direct_task_transport.cc:47: Creating actor via GCS actor id = : 72c387fa35abd8f54e7de4b701000000 | |
| [2021-04-29 17:04:01,873 I 9698 10376] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 72c387fa35abd8f54e7de4b701000000, ip address: 10.134.61.25, port: 9119, worker_id: 469dfd2e7742bf7ac21bb2840da0bbe41ea25438a863641ba60ed46f, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,219 I 9698 10376] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 72c387fa35abd8f54e7de4b701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,389 I 9698 10376] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 72c387fa35abd8f54e7de4b701000000, ip address: 10.134.61.25, port: 9119, worker_id: 469dfd2e7742bf7ac21bb2840da0bbe41ea25438a863641ba60ed46f, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,012 E 9698 9698] actor_manager.cc:93: Actor handle already exists 72c387fa35abd8f54e7de4b701000000 | |
| [2021-04-29 17:04:29,442 I 9698 9698] core_worker.cc:2064: Finished executing task b1b00e50401a764effffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-225db18bbb6fac7051724896026616d19216e2b089da468d8314b097_5256.log <== | |
| [2021-04-29 17:03:56,664 I 5256 5938] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:56,681 E 5256 5256] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:56,685 I 5256 5938] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:57,061 E 5256 5256] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:57,107 I 5256 5256] direct_task_transport.cc:47: Creating actor via GCS actor id = : ff8a459a8ca0d5a300ecc40001000000 | |
| [2021-04-29 17:03:57,114 I 5256 5938] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: ff8a459a8ca0d5a300ecc40001000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:57,240 I 5256 5938] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: ff8a459a8ca0d5a300ecc40001000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,729 I 5256 5938] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: ff8a459a8ca0d5a300ecc40001000000, ip address: 10.134.70.227, port: 9132, worker_id: aa8eb20ec8a8c08b54ed7b7c666073efbd02ae5c55592581698450f6, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:02,424 E 5256 5256] actor_manager.cc:93: Actor handle already exists ff8a459a8ca0d5a300ecc40001000000 | |
| [2021-04-29 17:04:10,233 I 5256 5256] core_worker.cc:2064: Finished executing task dd98cdae75bbc39affffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-233fba4160e4f0473a9e58403444ce1060980b4e9afaca00ca805601_13994.log <== | |
| [2021-04-29 17:04:05,495 I 13994 15489] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,544 E 13994 13994] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,555 I 13994 15489] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:07,818 E 13994 13994] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:08,089 I 13994 13994] direct_task_transport.cc:47: Creating actor via GCS actor id = : 0e0d80b8ac3fd11683a627e901000000 | |
| [2021-04-29 17:04:08,134 I 13994 15489] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 0e0d80b8ac3fd11683a627e901000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:18,564 I 13994 15489] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 0e0d80b8ac3fd11683a627e901000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,400 I 13994 15489] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 0e0d80b8ac3fd11683a627e901000000, ip address: 10.134.60.23, port: 9161, worker_id: 208f1748de1681ac136d50bbb997ab45328fc0bc600ea69d3e5c9c77, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,535 E 13994 13994] actor_manager.cc:93: Actor handle already exists 0e0d80b8ac3fd11683a627e901000000 | |
| [2021-04-29 17:04:29,536 I 13994 13994] core_worker.cc:2064: Finished executing task a00d5a6f2112b0d9ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-23777adfc171ca963c1aedabd2ac22f4a600228af6e255f8380db8cb_4001.log <== | |
| [2021-04-29 17:03:44,988 I 4001 4001] core_worker.cc:2064: Finished executing task ae7da552c361adc67fec1c909a465fbc984d7d6801000000, status=OK | |
| [2021-04-29 17:04:48,327 I 4001 4001] core_worker.cc:2064: Finished executing task 06f849e02ab392007fec1c909a465fbc984d7d6801000000, status=OK | |
| [2021-04-29 17:04:49,397 I 4001 4001] core_worker.cc:2064: Finished executing task 4204b1058608bd427fec1c909a465fbc984d7d6801000000, status=OK | |
| [2021-04-29 17:04:50,735 I 4001 4001] core_worker.cc:2064: Finished executing task 4876831fe2e8202d7fec1c909a465fbc984d7d6801000000, status=OK | |
| [2021-04-29 17:04:51,697 I 4001 4001] core_worker.cc:2064: Finished executing task ddf4481eb20f6f9b7fec1c909a465fbc984d7d6801000000, status=OK | |
| [2021-04-29 17:04:53,207 I 4001 4001] core_worker.cc:2064: Finished executing task 0f694a361ca671ba7fec1c909a465fbc984d7d6801000000, status=OK | |
| [2021-04-29 17:04:53,895 I 4001 4001] core_worker.cc:2064: Finished executing task f7124fb44aa7cc6c7fec1c909a465fbc984d7d6801000000, status=OK | |
| [2021-04-29 17:04:55,293 I 4001 4001] core_worker.cc:2064: Finished executing task e985111e24e959277fec1c909a465fbc984d7d6801000000, status=OK | |
| [2021-04-29 17:04:57,062 I 4001 4001] core_worker.cc:2064: Finished executing task a1987f3c0a06f8cf7fec1c909a465fbc984d7d6801000000, status=OK | |
| [2021-04-29 17:04:57,507 I 4001 4001] core_worker.cc:2064: Finished executing task 15dd4c7f24a7d2b97fec1c909a465fbc984d7d6801000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-23957d9bfaae6d9504536c0cc82eb49d83f158f8ba526e89b1ca3e0b_17058.log <== | |
| [2021-04-29 17:04:08,922 I 17058 18784] service_based_accessor.cc:579: Received notification for node id = ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, IsAlive = 1 | |
| [2021-04-29 17:04:08,922 I 17058 18784] service_based_accessor.cc:579: Received notification for node id = d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, IsAlive = 1 | |
| [2021-04-29 17:04:08,922 I 17058 18784] service_based_accessor.cc:579: Received notification for node id = b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, IsAlive = 1 | |
| [2021-04-29 17:04:08,922 I 17058 18784] service_based_accessor.cc:579: Received notification for node id = 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, IsAlive = 1 | |
| [2021-04-29 17:04:08,922 I 17058 18784] service_based_accessor.cc:579: Received notification for node id = 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, IsAlive = 1 | |
| [2021-04-29 17:04:09,926 I 17058 18784] core_worker.cc:717: Exit signal received, this process will exit after all outstanding tasks have finished, exit_type=INTENDED_EXIT | |
| [2021-04-29 17:04:09,926 I 17058 17058] raylet_client.cc:129: RayletClient::Disconnect, exit_type=INTENDED_EXIT, has creation_task_exception_pb_bytes=0 | |
| [2021-04-29 17:04:09,926 I 17058 17058] core_worker.cc:308: Removed worker 23957d9bfaae6d9504536c0cc82eb49d83f158f8ba526e89b1ca3e0b | |
| [2021-04-29 17:04:09,929 I 17058 17058] core_worker.cc:181: Destructing CoreWorkerProcess. pid: 17058 | |
| [2021-04-29 17:04:09,929 I 17058 17058] io_service_pool.cc:47: IOServicePool is stopped. | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-241041b9313baaf6df2225fad0fe306a4f639e181bd174211f095a4b_15938.log <== | |
| [2021-04-29 17:04:08,204 I 15938 17322] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,420 E 15938 15938] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,444 I 15938 17322] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,029 E 15938 15938] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,149 I 15938 15938] direct_task_transport.cc:47: Creating actor via GCS actor id = : d83df9080707269af4c5966f01000000 | |
| [2021-04-29 17:04:11,170 I 15938 17322] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: d83df9080707269af4c5966f01000000, ip address: 10.134.70.227, port: 9206, worker_id: ea0e383fd6b6972372c85f475a00186c5708ed34c8f89610fa5bde88, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,374 I 15938 17322] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: d83df9080707269af4c5966f01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:25,375 I 15938 17322] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: d83df9080707269af4c5966f01000000, ip address: 10.134.70.227, port: 9206, worker_id: ea0e383fd6b6972372c85f475a00186c5708ed34c8f89610fa5bde88, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,388 E 15938 15938] actor_manager.cc:93: Actor handle already exists d83df9080707269af4c5966f01000000 | |
| [2021-04-29 17:04:29,831 I 15938 15938] core_worker.cc:2064: Finished executing task 024f8591ca044ddbffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-245c0643f2505e7eded3c598144c5dd6697f73593b799bf716289049_14084.log <== | |
| [2021-04-29 17:04:05,665 I 14084 15303] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,692 E 14084 14084] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,702 I 14084 15303] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,095 E 14084 14084] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:08,357 I 14084 14084] direct_task_transport.cc:47: Creating actor via GCS actor id = : 081dbd098e9624e630b953f801000000 | |
| [2021-04-29 17:04:08,413 I 14084 15303] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 081dbd098e9624e630b953f801000000, ip address: 10.134.61.130, port: 9237, worker_id: 1b1279db2bd17ee127cf58753f7edd3580ec8752b6b33d37ebf5064a, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:18,877 I 14084 15303] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 081dbd098e9624e630b953f801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,399 I 14084 15303] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 081dbd098e9624e630b953f801000000, ip address: 10.134.61.130, port: 9237, worker_id: 1b1279db2bd17ee127cf58753f7edd3580ec8752b6b33d37ebf5064a, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,497 E 14084 14084] actor_manager.cc:93: Actor handle already exists 081dbd098e9624e630b953f801000000 | |
| [2021-04-29 17:04:29,532 I 14084 14084] core_worker.cc:2064: Finished executing task b57e94da2a8cf21affffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-2530bbdae5eb25c397744a34f4b94f39049e9b67ed8cf5dc6af55c48_4100.log <== | |
| [2021-04-29 17:03:58,475 E 4100 4100] actor_manager.cc:93: Actor handle already exists 2146c598330db034f884759201000000 | |
| [2021-04-29 17:03:59,839 I 4100 4100] core_worker.cc:2064: Finished executing task cf36fa19539364a9ffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:03:59,846 E 4100 4100] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,846 E 4100 4100] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,906 I 4100 4100] direct_task_transport.cc:47: Creating actor via GCS actor id = : 21ee460db2eea1bc200c90a601000000 | |
| [2021-04-29 17:03:59,910 I 4100 4978] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 21ee460db2eea1bc200c90a601000000, ip address: 10.134.61.25, port: 9125, worker_id: 6e9f12a0a36dfd3a21aad22614b67a8ce77b18ab242c1a08eb521918, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,728 I 4100 4978] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 21ee460db2eea1bc200c90a601000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,367 I 4100 4978] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 21ee460db2eea1bc200c90a601000000, ip address: 10.134.61.25, port: 9125, worker_id: 6e9f12a0a36dfd3a21aad22614b67a8ce77b18ab242c1a08eb521918, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,939 E 4100 4100] actor_manager.cc:93: Actor handle already exists 21ee460db2eea1bc200c90a601000000 | |
| [2021-04-29 17:04:29,417 I 4100 4100] core_worker.cc:2064: Finished executing task 20edb60dd83a818affffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-25e8c770088865fa84489421965fda1aa395e9d6d5556dfe7903160a_15931.log <== | |
| [2021-04-29 17:04:08,150 I 15931 17146] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,307 E 15931 15931] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,414 I 15931 17146] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,001 E 15931 15931] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,093 I 15931 15931] direct_task_transport.cc:47: Creating actor via GCS actor id = : 9742adb29894d383ad4129ca01000000 | |
| [2021-04-29 17:04:11,138 I 15931 17146] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 9742adb29894d383ad4129ca01000000, ip address: 10.134.142.44, port: 9216, worker_id: b8fdb8ea0bbfb57d704f6cbbbbd9417a52322bffbce9dfef5ec379b0, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,372 I 15931 17146] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 9742adb29894d383ad4129ca01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:25,338 I 15931 17146] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 9742adb29894d383ad4129ca01000000, ip address: 10.134.142.44, port: 9216, worker_id: b8fdb8ea0bbfb57d704f6cbbbbd9417a52322bffbce9dfef5ec379b0, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,367 E 15931 15931] actor_manager.cc:93: Actor handle already exists 9742adb29894d383ad4129ca01000000 | |
| [2021-04-29 17:04:29,829 I 15931 15931] core_worker.cc:2064: Finished executing task 0a7d7dc1dcd4f7dfffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-25fc1cd4161adb34ea683d84515db7aa30590ced50ced903980737fa_13394.log <== | |
| [2021-04-29 17:04:06,074 I 13394 15052] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,104 E 13394 13394] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:06,117 I 13394 15052] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,803 E 13394 13394] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:08,961 I 13394 13394] direct_task_transport.cc:47: Creating actor via GCS actor id = : effe924d38d198fa59c7f7d801000000 | |
| [2021-04-29 17:04:08,966 I 13394 15052] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: effe924d38d198fa59c7f7d801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:20,383 I 13394 15052] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: effe924d38d198fa59c7f7d801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,401 I 13394 15052] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: effe924d38d198fa59c7f7d801000000, ip address: 10.134.142.16, port: 9183, worker_id: e126545ad7b6443c849c9eb2336698e117372e9466293589367ccba1, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,556 E 13394 13394] actor_manager.cc:93: Actor handle already exists effe924d38d198fa59c7f7d801000000 | |
| [2021-04-29 17:04:29,538 I 13394 13394] core_worker.cc:2064: Finished executing task ef8f67df968cb1bbffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-26c43710d5666711838599019a27d32adf752ba0d04a4755bb13282d_16007.log <== | |
| [2021-04-29 17:04:08,446 I 16007 17610] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,541 E 16007 16007] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,578 I 16007 17610] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,122 E 16007 16007] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,232 I 16007 16007] direct_task_transport.cc:47: Creating actor via GCS actor id = : 663fd1cfa6f4c25e1869e4a801000000 | |
| [2021-04-29 17:04:11,272 I 16007 17610] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 663fd1cfa6f4c25e1869e4a801000000, ip address: 10.134.61.130, port: 9206, worker_id: b3deccf77a11a561e09a0b82b5ada6493298a8a837517f432d7e654b, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,377 I 16007 17610] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 663fd1cfa6f4c25e1869e4a801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:26,967 I 16007 17610] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 663fd1cfa6f4c25e1869e4a801000000, ip address: 10.134.61.130, port: 9206, worker_id: b3deccf77a11a561e09a0b82b5ada6493298a8a837517f432d7e654b, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,728 E 16007 16007] actor_manager.cc:93: Actor handle already exists 663fd1cfa6f4c25e1869e4a801000000 | |
| [2021-04-29 17:04:29,851 I 16007 16007] core_worker.cc:2064: Finished executing task 075fd7b1c1129664ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-26d1f7ac3601aa3b3f59baa670a498143ac68608927a2918d68a8ac7_16029.log <== | |
| [2021-04-29 17:04:08,478 I 16029 17707] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,582 E 16029 16029] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,624 I 16029 17707] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,123 E 16029 16029] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,224 I 16029 16029] direct_task_transport.cc:47: Creating actor via GCS actor id = : 52e25057ca04e06c2c0ca71701000000 | |
| [2021-04-29 17:04:11,272 I 16029 17707] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 52e25057ca04e06c2c0ca71701000000, ip address: 10.133.95.40, port: 9200, worker_id: 965ec97a4f58e99f818223fe0707ccb184300babf919bc296667e9e4, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,377 I 16029 17707] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 52e25057ca04e06c2c0ca71701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,771 I 16029 17707] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 52e25057ca04e06c2c0ca71701000000, ip address: 10.133.95.40, port: 9200, worker_id: 965ec97a4f58e99f818223fe0707ccb184300babf919bc296667e9e4, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:27,587 E 16029 16029] actor_manager.cc:93: Actor handle already exists 52e25057ca04e06c2c0ca71701000000 | |
| [2021-04-29 17:04:29,808 I 16029 16029] core_worker.cc:2064: Finished executing task 80cea237e3eb2ea4ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-2a2c4a6686c659bb3d93b7d6c49dfe8f206d5d03a5b862cd9e5bf335_13307.log <== | |
| [2021-04-29 17:04:06,025 I 13307 15101] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,061 E 13307 13307] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:06,073 I 13307 15101] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,793 E 13307 13307] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:08,961 I 13307 13307] direct_task_transport.cc:47: Creating actor via GCS actor id = : d79f94980d11a948619d32fd01000000 | |
| [2021-04-29 17:04:08,967 I 13307 15101] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: d79f94980d11a948619d32fd01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:20,271 I 13307 15101] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: d79f94980d11a948619d32fd01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,401 I 13307 15101] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: d79f94980d11a948619d32fd01000000, ip address: 10.133.95.40, port: 9184, worker_id: 515dc2b0bcedf5fad0670473bbbf903f0404ff48afb3497a1efad37b, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,546 E 13307 13307] actor_manager.cc:93: Actor handle already exists d79f94980d11a948619d32fd01000000 | |
| [2021-04-29 17:04:29,538 I 13307 13307] core_worker.cc:2064: Finished executing task 1220409cb349364cffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-2a9029e1339d86523b9ae85877b2f84cc349df3872ea64d77830e959_16023.log <== | |
| [2021-04-29 17:04:08,665 I 16023 17860] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,719 E 16023 16023] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,742 I 16023 17860] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,162 E 16023 16023] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,253 I 16023 16023] direct_task_transport.cc:47: Creating actor via GCS actor id = : b7aab6539df4ee880f773ab701000000 | |
| [2021-04-29 17:04:11,303 I 16023 17860] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: b7aab6539df4ee880f773ab701000000, ip address: 10.134.61.25, port: 9198, worker_id: fc1475f7c5034876fe7b4f4374b6c85f09c7156996dab8721674e8c9, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,377 I 16023 17860] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: b7aab6539df4ee880f773ab701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,974 I 16023 17860] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: b7aab6539df4ee880f773ab701000000, ip address: 10.134.61.25, port: 9198, worker_id: fc1475f7c5034876fe7b4f4374b6c85f09c7156996dab8721674e8c9, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:27,696 E 16023 16023] actor_manager.cc:93: Actor handle already exists b7aab6539df4ee880f773ab701000000 | |
| [2021-04-29 17:04:29,812 I 16023 16023] core_worker.cc:2064: Finished executing task a67e8eb1abac37e2ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-2ac726a8ba7992dc34d3c3ff9ece0fe73dde03eac5ebdf1f04e2fae1_4062.log <== | |
| [2021-04-29 17:03:44,975 I 4062 4062] core_worker.cc:2064: Finished executing task 7ac9d2d7158c582953f25d6651fcd03511c3cf2e01000000, status=OK | |
| [2021-04-29 17:04:42,677 I 4062 4062] core_worker.cc:2064: Finished executing task 5253f085438b450e53f25d6651fcd03511c3cf2e01000000, status=OK | |
| [2021-04-29 17:04:48,791 I 4062 4062] core_worker.cc:2064: Finished executing task 41a76175f939dd8053f25d6651fcd03511c3cf2e01000000, status=OK | |
| [2021-04-29 17:04:50,290 I 4062 4062] core_worker.cc:2064: Finished executing task 14dcaf5516a676a653f25d6651fcd03511c3cf2e01000000, status=OK | |
| [2021-04-29 17:04:51,248 I 4062 4062] core_worker.cc:2064: Finished executing task f41290ff5661d6c953f25d6651fcd03511c3cf2e01000000, status=OK | |
| [2021-04-29 17:04:52,007 I 4062 4062] core_worker.cc:2064: Finished executing task 795189cc442add4853f25d6651fcd03511c3cf2e01000000, status=OK | |
| [2021-04-29 17:04:53,202 I 4062 4062] core_worker.cc:2064: Finished executing task d46a2559df9158a853f25d6651fcd03511c3cf2e01000000, status=OK | |
| [2021-04-29 17:04:55,206 I 4062 4062] core_worker.cc:2064: Finished executing task fdfeaee2d04a04e453f25d6651fcd03511c3cf2e01000000, status=OK | |
| [2021-04-29 17:04:55,378 I 4062 4062] core_worker.cc:2064: Finished executing task 9f78944b97a9e3f453f25d6651fcd03511c3cf2e01000000, status=OK | |
| [2021-04-29 17:04:57,559 I 4062 4062] core_worker.cc:2064: Finished executing task 5437839b1e69ab1053f25d6651fcd03511c3cf2e01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-2ad96c7e9ed193043831e29ac33836592d33ba22b73634c87341a4a0_4034.log <== | |
| [2021-04-29 17:03:44,980 I 4034 4034] core_worker.cc:2064: Finished executing task 1b172af1c54fb75592922ec1c09d8d377d4fc7c101000000, status=OK | |
| [2021-04-29 17:04:38,787 I 4034 4034] core_worker.cc:2064: Finished executing task 49494552e3a53bd192922ec1c09d8d377d4fc7c101000000, status=OK | |
| [2021-04-29 17:04:49,067 I 4034 4034] core_worker.cc:2064: Finished executing task 5f9f03106a40390c92922ec1c09d8d377d4fc7c101000000, status=OK | |
| [2021-04-29 17:04:50,043 I 4034 4034] core_worker.cc:2064: Finished executing task 47186136acce606292922ec1c09d8d377d4fc7c101000000, status=OK | |
| [2021-04-29 17:04:51,497 I 4034 4034] core_worker.cc:2064: Finished executing task 8731217f2542d1fe92922ec1c09d8d377d4fc7c101000000, status=OK | |
| [2021-04-29 17:04:52,130 I 4034 4034] core_worker.cc:2064: Finished executing task 9865cd0ae5ccf9d892922ec1c09d8d377d4fc7c101000000, status=OK | |
| [2021-04-29 17:04:53,903 I 4034 4034] core_worker.cc:2064: Finished executing task 13f97bc5f78fb71992922ec1c09d8d377d4fc7c101000000, status=OK | |
| [2021-04-29 17:04:54,250 I 4034 4034] core_worker.cc:2064: Finished executing task de92f7c7d97b6b6792922ec1c09d8d377d4fc7c101000000, status=OK | |
| [2021-04-29 17:04:56,555 I 4034 4034] core_worker.cc:2064: Finished executing task 2bad5de6c9ce6c1692922ec1c09d8d377d4fc7c101000000, status=OK | |
| [2021-04-29 17:04:57,406 I 4034 4034] core_worker.cc:2064: Finished executing task b992c307b30e2b4992922ec1c09d8d377d4fc7c101000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-2adf98d08b4cc2c4c67051cffd172bbc6afaa1c06cb6a79e457480e3_9784.log <== | |
| [2021-04-29 17:04:01,513 I 9784 11300] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,534 E 9784 9784] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,540 I 9784 11300] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,103 E 9784 9784] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:02,155 I 9784 9784] direct_task_transport.cc:47: Creating actor via GCS actor id = : 0a74cf64f63fa9d3089753f501000000 | |
| [2021-04-29 17:04:02,169 I 9784 11300] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 0a74cf64f63fa9d3089753f501000000, ip address: 10.134.142.44, port: 9104, worker_id: 6931c625fcf137230332c5d35646b11cef77ed5294c9600d6094f866, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,545 I 9784 11300] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 0a74cf64f63fa9d3089753f501000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,391 I 9784 11300] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 0a74cf64f63fa9d3089753f501000000, ip address: 10.134.142.44, port: 9104, worker_id: 6931c625fcf137230332c5d35646b11cef77ed5294c9600d6094f866, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,767 E 9784 9784] actor_manager.cc:93: Actor handle already exists 0a74cf64f63fa9d3089753f501000000 | |
| [2021-04-29 17:04:29,543 I 9784 9784] core_worker.cc:2064: Finished executing task 0275c47ccf26ab9fffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-2c23c5593dd14f03f6aba7e7459bd7a9f61e98237f95402375201d4c_12785.log <== | |
| [2021-04-29 17:04:04,717 I 12785 13499] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,764 E 12785 12785] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:04,787 I 12785 13499] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,546 E 12785 12785] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:06,750 I 12785 12785] direct_task_transport.cc:47: Creating actor via GCS actor id = : de5d11759e1706c29e59e18501000000 | |
| [2021-04-29 17:04:06,782 I 12785 13499] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: de5d11759e1706c29e59e18501000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:13,662 I 12785 13499] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: de5d11759e1706c29e59e18501000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,396 I 12785 13499] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: de5d11759e1706c29e59e18501000000, ip address: 10.134.142.44, port: 9234, worker_id: 681119314a37c78ac454c9a66280833dc18b3fbace4fc1a5b91d4f29, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,324 E 12785 12785] actor_manager.cc:93: Actor handle already exists de5d11759e1706c29e59e18501000000 | |
| [2021-04-29 17:04:29,503 I 12785 12785] core_worker.cc:2064: Finished executing task c01706977032402affffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-2c629652f7d5ae05b5f80a58cf5dd84cb63db0cdd1eb4a56d9a332ac_17845.log <== | |
| [2021-04-29 17:04:09,819 I 17845 19522] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,895 E 17845 17845] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,916 I 17845 19522] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,580 E 17845 17845] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,704 I 17845 17845] direct_task_transport.cc:47: Creating actor via GCS actor id = : 0b23a86eb93a863d3374b85f01000000 | |
| [2021-04-29 17:04:11,756 I 17845 19522] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 0b23a86eb93a863d3374b85f01000000, ip address: 10.134.142.44, port: 9199, worker_id: d5f941dbd15b11075730c23c763990be4e8c35055d8555215c1c2da6, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,386 I 17845 19522] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 0b23a86eb93a863d3374b85f01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:28,246 I 17845 19522] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 0b23a86eb93a863d3374b85f01000000, ip address: 10.134.142.44, port: 9199, worker_id: d5f941dbd15b11075730c23c763990be4e8c35055d8555215c1c2da6, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:29,034 E 17845 17845] actor_manager.cc:93: Actor handle already exists 0b23a86eb93a863d3374b85f01000000 | |
| [2021-04-29 17:04:29,862 I 17845 17845] core_worker.cc:2064: Finished executing task e6cd9fe6971d50afffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-2d191dfcf6bfbd9c22683d194c290c1ee81d08cd6b26da6316da55c5_15891.log <== | |
| [2021-04-29 17:04:07,911 I 15891 16766] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,205 E 15891 15891] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,249 I 15891 16766] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,964 E 15891 15891] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,059 I 15891 15891] direct_task_transport.cc:47: Creating actor via GCS actor id = : a7f32c92a2311528cbc64ec201000000 | |
| [2021-04-29 17:04:11,102 I 15891 16766] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: a7f32c92a2311528cbc64ec201000000, ip address: 10.134.61.130, port: 9218, worker_id: e6916b8a3f1c9e450de906ab3a178eabd8ab2a2dda664685e55f205b, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,370 I 15891 16766] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: a7f32c92a2311528cbc64ec201000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:25,147 I 15891 16766] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: a7f32c92a2311528cbc64ec201000000, ip address: 10.134.61.130, port: 9218, worker_id: e6916b8a3f1c9e450de906ab3a178eabd8ab2a2dda664685e55f205b, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,144 E 15891 15891] actor_manager.cc:93: Actor handle already exists a7f32c92a2311528cbc64ec201000000 | |
| [2021-04-29 17:04:29,820 I 15891 15891] core_worker.cc:2064: Finished executing task 03c6d8e8cf927343ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-2dbdf23a32d6bbc0b6bfdfbecea66eb913f15025fab0499cf0a9ed7c_5347.log <== | |
| [2021-04-29 17:03:46,415 I 5347 5347] core_worker.cc:2064: Finished executing task ef6f7088a4df657eca2c0210873fd66f8487aa0701000000, status=OK | |
| [2021-04-29 17:04:36,892 I 5347 5347] core_worker.cc:2064: Finished executing task 7044289b4f295a88ca2c0210873fd66f8487aa0701000000, status=OK | |
| [2021-04-29 17:04:48,755 I 5347 5347] core_worker.cc:2064: Finished executing task 43db2d3c01851ff6ca2c0210873fd66f8487aa0701000000, status=OK | |
| [2021-04-29 17:04:50,107 I 5347 5347] core_worker.cc:2064: Finished executing task 57c283d616eb864fca2c0210873fd66f8487aa0701000000, status=OK | |
| [2021-04-29 17:04:50,966 I 5347 5347] core_worker.cc:2064: Finished executing task 1e1ac2dedf9c252aca2c0210873fd66f8487aa0701000000, status=OK | |
| [2021-04-29 17:04:52,692 I 5347 5347] core_worker.cc:2064: Finished executing task b8daefb48ef02b52ca2c0210873fd66f8487aa0701000000, status=OK | |
| [2021-04-29 17:04:52,851 I 5347 5347] core_worker.cc:2064: Finished executing task 3dec9a4793bd9ed2ca2c0210873fd66f8487aa0701000000, status=OK | |
| [2021-04-29 17:04:53,095 I 5347 5347] core_worker.cc:2064: Finished executing task 6fc49d73223a8aeeca2c0210873fd66f8487aa0701000000, status=OK | |
| [2021-04-29 17:04:56,489 I 5347 5347] core_worker.cc:2064: Finished executing task 33e5d8c3b1dee1f9ca2c0210873fd66f8487aa0701000000, status=OK | |
| [2021-04-29 17:04:57,808 I 5347 5347] core_worker.cc:2064: Finished executing task e442fdb599b266fcca2c0210873fd66f8487aa0701000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-2e472c93f2368766f0a8e80a3aecc6c5352cb2fbfbb07a79d91bf758_8072.log <== | |
| [2021-04-29 17:03:58,625 I 8072 9527] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,643 E 8072 8072] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,654 I 8072 9527] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,146 E 8072 8072] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,192 I 8072 8072] direct_task_transport.cc:47: Creating actor via GCS actor id = : eb9aec818d6bfcf125536f1801000000 | |
| [2021-04-29 17:03:59,195 I 8072 9527] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: eb9aec818d6bfcf125536f1801000000, ip address: 10.133.95.40, port: 9132, worker_id: 7f3e4b3585fef3bb837e0fd3e8781c9dfca118c97f20846a94e4eaa1, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,100 I 8072 9527] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: eb9aec818d6bfcf125536f1801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,299 I 8072 9527] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: eb9aec818d6bfcf125536f1801000000, ip address: 10.133.95.40, port: 9132, worker_id: 7f3e4b3585fef3bb837e0fd3e8781c9dfca118c97f20846a94e4eaa1, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:07,705 E 8072 8072] actor_manager.cc:93: Actor handle already exists eb9aec818d6bfcf125536f1801000000 | |
| [2021-04-29 17:04:12,939 I 8072 8072] core_worker.cc:2064: Finished executing task 4f3608240105afabffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-2ec7da3ede68f79f66260ddec1f9fdce0d9bcd3041c6299d3e64e4bd_12851.log <== | |
| [2021-04-29 17:04:04,938 I 12851 14021] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,985 E 12851 12851] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:04,999 I 12851 14021] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,873 E 12851 12851] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:07,078 I 12851 12851] direct_task_transport.cc:47: Creating actor via GCS actor id = : fc52bccffece478d8ea6f60a01000000 | |
| [2021-04-29 17:04:07,104 I 12851 14021] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: fc52bccffece478d8ea6f60a01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:15,669 I 12851 14021] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: fc52bccffece478d8ea6f60a01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,399 I 12851 14021] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: fc52bccffece478d8ea6f60a01000000, ip address: 10.134.60.23, port: 9154, worker_id: 471492bc43145b5e59401c790c5b54b2e221eb778861486c8e954ee2, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,419 E 12851 12851] actor_manager.cc:93: Actor handle already exists fc52bccffece478d8ea6f60a01000000 | |
| [2021-04-29 17:04:29,521 I 12851 12851] core_worker.cc:2064: Finished executing task 2b24bf72dfb49b71ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-2ee0c7594e159c10305df734200e391f004ac23c0c348c14006dec50_12622.log <== | |
| [2021-04-29 17:04:04,520 I 12622 13248] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,583 E 12622 12622] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:04,616 I 12622 13248] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,509 E 12622 12622] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:06,601 I 12622 12622] direct_task_transport.cc:47: Creating actor via GCS actor id = : 11ed5d1dcdb6dd94b471ad4801000000 | |
| [2021-04-29 17:04:06,683 I 12622 13248] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 11ed5d1dcdb6dd94b471ad4801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:12,310 I 12622 13248] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 11ed5d1dcdb6dd94b471ad4801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,395 I 12622 13248] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 11ed5d1dcdb6dd94b471ad4801000000, ip address: 10.134.112.222, port: 9103, worker_id: 4b278fe0c0ac90601cb35462cbbf5b0220a09ddff76d81358339ee53, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,733 E 12622 12622] actor_manager.cc:93: Actor handle already exists 11ed5d1dcdb6dd94b471ad4801000000 | |
| [2021-04-29 17:04:29,540 I 12622 12622] core_worker.cc:2064: Finished executing task d2550e9fb4cacad5ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-2ee3919c56dcc85e772880f54778be46bba883f10865ad4c8205069f_16031.log <== | |
| [2021-04-29 17:04:08,694 I 16031 17919] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,774 E 16031 16031] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,798 I 16031 17919] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,192 E 16031 16031] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,302 I 16031 16031] direct_task_transport.cc:47: Creating actor via GCS actor id = : 47c93b8ed407c05a50ba4cd201000000 | |
| [2021-04-29 17:04:11,327 I 16031 17919] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 47c93b8ed407c05a50ba4cd201000000, ip address: 10.134.61.25, port: 9200, worker_id: 42623ffe5011de970bda647bc58ade2a51235dd662c81a92d59887d6, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,379 I 16031 17919] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 47c93b8ed407c05a50ba4cd201000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:25,181 I 16031 17919] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 47c93b8ed407c05a50ba4cd201000000, ip address: 10.134.61.25, port: 9200, worker_id: 42623ffe5011de970bda647bc58ade2a51235dd662c81a92d59887d6, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,188 E 16031 16031] actor_manager.cc:93: Actor handle already exists 47c93b8ed407c05a50ba4cd201000000 | |
| [2021-04-29 17:04:29,821 I 16031 16031] core_worker.cc:2064: Finished executing task 4a9e7935e1400817ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d_3992.log <== | |
| [2021-04-29 17:17:09,843 I 3992 3992] core_worker.cc:2064: Finished executing task 8afa7bc8bc4336098458f0d3cd94cfc58ef7456001000000, status=OK | |
| [2021-04-29 17:17:09,904 I 3992 3992] core_worker.cc:2064: Finished executing task eb8bbdbd712b4a5c8458f0d3cd94cfc58ef7456001000000, status=OK | |
| [2021-04-29 17:52:18,463 I 3992 4102] actor_manager.cc:127: received notification on actor, state: DEAD, actor_id: d61ced7ddc924cda5c353b5f01000000, ip address: 10.134.61.25, port: 9140, worker_id: 37307554699c9ff814689169f3e10379e9f581ef19357b0fe38ef162, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:52:18,463 I 3992 4102] actor_manager.cc:141: Creation task formatted exception: , actor_id: d61ced7ddc924cda5c353b5f01000000 | |
| [2021-04-29 17:52:18,464 I 3992 4102] direct_actor_transport.cc:204: Failing pending tasks for actor d61ced7ddc924cda5c353b5f01000000 | |
| [2021-04-29 17:52:18,464 I 3992 4102] direct_actor_transport.cc:222: Failing tasks waiting for death info, size=0, actor_id=d61ced7ddc924cda5c353b5f01000000 | |
| [2021-04-29 17:59:54,362 I 3992 4102] actor_manager.cc:127: received notification on actor, state: DEAD, actor_id: d4b4c9c748b84dbdb465d53e01000000, ip address: 10.134.61.130, port: 9207, worker_id: 97b501e0f48725b6d7de74fcb5d126348ea06172564173a86ab8f044, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:59:54,362 I 3992 4102] actor_manager.cc:141: Creation task formatted exception: , actor_id: d4b4c9c748b84dbdb465d53e01000000 | |
| [2021-04-29 17:59:54,362 I 3992 4102] direct_actor_transport.cc:204: Failing pending tasks for actor d4b4c9c748b84dbdb465d53e01000000 | |
| [2021-04-29 17:59:54,362 I 3992 4102] direct_actor_transport.cc:222: Failing tasks waiting for death info, size=0, actor_id=d4b4c9c748b84dbdb465d53e01000000 | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-2f3045767662f6497cdd0a582ccbf3dda405e1d3f621ae80f59b68c6_16478.log <== | |
| [2021-04-29 17:04:09,178 I 16478 18487] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,307 E 16478 16478] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,335 I 16478 18487] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,419 E 16478 16478] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,491 I 16478 16478] direct_task_transport.cc:47: Creating actor via GCS actor id = : 92a6de8fa954a9e9d7ffb7a701000000 | |
| [2021-04-29 17:04:11,529 I 16478 18487] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 92a6de8fa954a9e9d7ffb7a701000000, ip address: 10.134.142.44, port: 9200, worker_id: b8d9fb50c1bf98e5c84faf05fccc97df4ba1ef86b8a02ff94f1c56c4, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,380 I 16478 18487] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 92a6de8fa954a9e9d7ffb7a701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:27,597 I 16478 18487] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 92a6de8fa954a9e9d7ffb7a701000000, ip address: 10.134.142.44, port: 9200, worker_id: b8d9fb50c1bf98e5c84faf05fccc97df4ba1ef86b8a02ff94f1c56c4, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,912 E 16478 16478] actor_manager.cc:93: Actor handle already exists 92a6de8fa954a9e9d7ffb7a701000000 | |
| [2021-04-29 17:04:29,853 I 16478 16478] core_worker.cc:2064: Finished executing task 4ff7bfaad8ff47e5ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-2fa0f43e98bf64f8c3aabea797680925a0184615fe3f4790cbfa09d7_7925.log <== | |
| [2021-04-29 17:03:58,574 I 7925 9550] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,575 E 7925 7925] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,576 I 7925 9550] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,958 E 7925 7925] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,026 I 7925 7925] direct_task_transport.cc:47: Creating actor via GCS actor id = : e0cea56f1836e1040b73accf01000000 | |
| [2021-04-29 17:03:59,030 I 7925 9550] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: e0cea56f1836e1040b73accf01000000, ip address: 10.134.142.16, port: 9135, worker_id: f96cf3c7fbe94b7df80948be5cca66846b2f4b73fdaaded8774db3f9, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:00,775 I 7925 9550] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: e0cea56f1836e1040b73accf01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,213 I 7925 9550] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: e0cea56f1836e1040b73accf01000000, ip address: 10.134.142.16, port: 9135, worker_id: f96cf3c7fbe94b7df80948be5cca66846b2f4b73fdaaded8774db3f9, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:07,524 E 7925 7925] actor_manager.cc:93: Actor handle already exists e0cea56f1836e1040b73accf01000000 | |
| [2021-04-29 17:04:12,741 I 7925 7925] core_worker.cc:2064: Finished executing task 65473442eccc53d2ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-2ff5b4262a4846c9ce9fe9c9e2a5ad138c9a0f69981157bc376dec1d_7440.log <== | |
| [2021-04-29 17:03:59,746 I 7440 7939] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,777 E 7440 7440] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,812 I 7440 7939] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:00,425 E 7440 7440] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:00,498 I 7440 7440] direct_task_transport.cc:47: Creating actor via GCS actor id = : 5c63b56e91d29b5a9280d24301000000 | |
| [2021-04-29 17:04:00,511 I 7440 7939] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 5c63b56e91d29b5a9280d24301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,733 I 7440 7939] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 5c63b56e91d29b5a9280d24301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,376 I 7440 7939] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 5c63b56e91d29b5a9280d24301000000, ip address: 10.134.142.44, port: 9113, worker_id: 0ac789f33b51627e71ccc23f2c473dd2888edb69edf2a5089f7bd733, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,997 E 7440 7440] actor_manager.cc:93: Actor handle already exists 5c63b56e91d29b5a9280d24301000000 | |
| [2021-04-29 17:04:29,434 I 7440 7440] core_worker.cc:2064: Finished executing task ecb5a3111dc21cd3ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-303f4744f799ebb6bbc72c91556f373f5debc539f28e1057a5e7c436_12831.log <== | |
| [2021-04-29 17:04:04,848 I 12831 13775] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,921 E 12831 12831] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:04,937 I 12831 13775] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,806 E 12831 12831] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:06,983 I 12831 12831] direct_task_transport.cc:47: Creating actor via GCS actor id = : ded3e5fac0d1f04b16d2195601000000 | |
| [2021-04-29 17:04:07,053 I 12831 13775] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: ded3e5fac0d1f04b16d2195601000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:14,794 I 12831 13775] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: ded3e5fac0d1f04b16d2195601000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,398 I 12831 13775] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: ded3e5fac0d1f04b16d2195601000000, ip address: 10.133.95.40, port: 9154, worker_id: 77a7e65555147fedb30fada7fa8ecc48a8169a21c00bcf3f3c51817d, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,407 E 12831 12831] actor_manager.cc:93: Actor handle already exists ded3e5fac0d1f04b16d2195601000000 | |
| [2021-04-29 17:04:29,518 I 12831 12831] core_worker.cc:2064: Finished executing task d6ec5c82cbfcf327ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-315808b227276f4e460b58001b56e6521c04cca003559e106b586019_8108.log <== | |
| [2021-04-29 17:03:58,594 I 8108 9548] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,595 E 8108 8108] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,596 I 8108 9548] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,045 E 8108 8108] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,111 I 8108 8108] direct_task_transport.cc:47: Creating actor via GCS actor id = : 9838cd4e1dcba1cb0035f68d01000000 | |
| [2021-04-29 17:03:59,113 I 8108 9548] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 9838cd4e1dcba1cb0035f68d01000000, ip address: 10.134.142.44, port: 9122, worker_id: eff610c80fdff46d5a9a7c75d8c73e28c2022bb61418c469e1db4176, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:00,901 I 8108 9548] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 9838cd4e1dcba1cb0035f68d01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,218 I 8108 9548] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 9838cd4e1dcba1cb0035f68d01000000, ip address: 10.134.142.44, port: 9122, worker_id: eff610c80fdff46d5a9a7c75d8c73e28c2022bb61418c469e1db4176, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:07,545 E 8108 8108] actor_manager.cc:93: Actor handle already exists 9838cd4e1dcba1cb0035f68d01000000 | |
| [2021-04-29 17:04:12,742 I 8108 8108] core_worker.cc:2064: Finished executing task a3598f4e7e2fe126ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-32af5080389749ad049df344c72ab81ac450c387335bc9f21b444f9a_7412.log <== | |
| [2021-04-29 17:03:58,036 I 7412 7722] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,065 E 7412 7412] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,070 I 7412 7722] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,677 E 7412 7412] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:58,744 I 7412 7412] direct_task_transport.cc:47: Creating actor via GCS actor id = : b446e588fed46cf77713083001000000 | |
| [2021-04-29 17:03:58,754 I 7412 7722] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: b446e588fed46cf77713083001000000, ip address: 10.133.95.40, port: 9138, worker_id: ffdfef260df403e366fd896ce83d3fab4ac524b39c713defe9467f60, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,070 I 7412 7722] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: b446e588fed46cf77713083001000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,548 I 7412 7722] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: b446e588fed46cf77713083001000000, ip address: 10.133.95.40, port: 9138, worker_id: ffdfef260df403e366fd896ce83d3fab4ac524b39c713defe9467f60, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:06,093 E 7412 7412] actor_manager.cc:93: Actor handle already exists b446e588fed46cf77713083001000000 | |
| [2021-04-29 17:04:11,942 I 7412 7412] core_worker.cc:2064: Finished executing task e22b8850f93c4dd2ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-33c39cdce1eb984a76a2a41be0809ceb612e88f734e9b24ce56ceebd_4039.log <== | |
| [2021-04-29 17:04:02,325 E 4039 4039] actor_manager.cc:93: Actor handle already exists 43982aacb0852772ea04774b01000000 | |
| [2021-04-29 17:04:10,038 I 4039 4039] core_worker.cc:2064: Finished executing task 1bbf8a81f73cc404ffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:10,050 E 4039 4039] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,051 E 4039 4039] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,133 I 4039 4039] direct_task_transport.cc:47: Creating actor via GCS actor id = : 296eebc0046fc7d0e2845de701000000 | |
| [2021-04-29 17:04:10,180 I 4039 4714] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 296eebc0046fc7d0e2845de701000000, ip address: 10.134.61.25, port: 9207, worker_id: 6725dd41eeb4f46efc91e0a688bc06cef61995bf7cfc1457dcf96ae5, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,362 I 4039 4714] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 296eebc0046fc7d0e2845de701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,696 I 4039 4714] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 296eebc0046fc7d0e2845de701000000, ip address: 10.134.61.25, port: 9207, worker_id: 6725dd41eeb4f46efc91e0a688bc06cef61995bf7cfc1457dcf96ae5, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:24,460 E 4039 4039] actor_manager.cc:93: Actor handle already exists 296eebc0046fc7d0e2845de701000000 | |
| [2021-04-29 17:04:29,593 I 4039 4039] core_worker.cc:2064: Finished executing task bd6e9307e88373b4ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-34627ea50f8bd0dc7a35fc9dacd1598086a03668f99854b70f756db8_17970.log <== | |
| [2021-04-29 17:04:09,843 I 17970 19605] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,919 E 17970 17970] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,943 I 17970 19605] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,582 E 17970 17970] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,716 I 17970 17970] direct_task_transport.cc:47: Creating actor via GCS actor id = : 3afc4fd101839148861c66d101000000 | |
| [2021-04-29 17:04:11,755 I 17970 19605] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 3afc4fd101839148861c66d101000000, ip address: 10.134.60.23, port: 9184, worker_id: d7a72af59603b420b529b0626ae3bee9c577d57892c8da9f44d5ac6e, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,387 I 17970 19605] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 3afc4fd101839148861c66d101000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:27,126 I 17970 19605] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 3afc4fd101839148861c66d101000000, ip address: 10.134.60.23, port: 9184, worker_id: d7a72af59603b420b529b0626ae3bee9c577d57892c8da9f44d5ac6e, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,806 E 17970 17970] actor_manager.cc:93: Actor handle already exists 3afc4fd101839148861c66d101000000 | |
| [2021-04-29 17:04:29,847 I 17970 17970] core_worker.cc:2064: Finished executing task 77e9b4515468d550ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-347451b69df8210b870c72cc838dc0270b636e61e641c0fcc427d354_5365.log <== | |
| [2021-04-29 17:04:02,525 E 5365 5365] actor_manager.cc:93: Actor handle already exists 6cb2e2176013c788a41c2b2a01000000 | |
| [2021-04-29 17:04:10,370 I 5365 5365] core_worker.cc:2064: Finished executing task 8d7b37f7d270b54dffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:10,378 E 5365 5365] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,378 E 5365 5365] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,473 I 5365 5365] direct_task_transport.cc:47: Creating actor via GCS actor id = : f1f8cb7655e1da27fcc8215f01000000 | |
| [2021-04-29 17:04:10,517 I 5365 7009] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: f1f8cb7655e1da27fcc8215f01000000, ip address: 10.134.142.16, port: 9208, worker_id: 757fcc7917d062222e0b5ac496a11d267c3a58fa0f5a065c5bcaf0fa, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,366 I 5365 7009] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: f1f8cb7655e1da27fcc8215f01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:22,845 I 5365 7009] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: f1f8cb7655e1da27fcc8215f01000000, ip address: 10.134.142.16, port: 9208, worker_id: 757fcc7917d062222e0b5ac496a11d267c3a58fa0f5a065c5bcaf0fa, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:25,489 E 5365 5365] actor_manager.cc:93: Actor handle already exists f1f8cb7655e1da27fcc8215f01000000 | |
| [2021-04-29 17:04:29,744 I 5365 5365] core_worker.cc:2064: Finished executing task 35c830e867f97fcfffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-35249c583f1f47f0ccc36651fa0620af5035bef0d390566416819969_12099.log <== | |
| [2021-04-29 17:04:03,438 I 12099 12598] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,467 E 12099 12099] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:03,476 I 12099 12598] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,873 E 12099 12099] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,959 I 12099 12099] direct_task_transport.cc:47: Creating actor via GCS actor id = : 76088e6c9ab19c12fdeb984401000000 | |
| [2021-04-29 17:04:05,968 I 12099 12598] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 76088e6c9ab19c12fdeb984401000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,742 I 12099 12598] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 76088e6c9ab19c12fdeb984401000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,395 I 12099 12598] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 76088e6c9ab19c12fdeb984401000000, ip address: 10.134.112.222, port: 9105, worker_id: 6aae7ac2d51f3bc5d3f049d4a3bf0e68116a5ebef0c5a5f4f4c0579e, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,244 E 12099 12099] actor_manager.cc:93: Actor handle already exists 76088e6c9ab19c12fdeb984401000000 | |
| [2021-04-29 17:04:29,487 I 12099 12099] core_worker.cc:2064: Finished executing task 887e2551b2f3c790ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-366024b147a66542ae6e558ee7594811b3ad8bce4c767fee5c4bd6ab_12961.log <== | |
| [2021-04-29 17:04:06,249 I 12961 14784] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,305 E 12961 12961] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:06,319 I 12961 14784] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,940 E 12961 12961] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:09,040 I 12961 12961] direct_task_transport.cc:47: Creating actor via GCS actor id = : 3c56ae9f48ea8e4a4123855c01000000 | |
| [2021-04-29 17:04:09,084 I 12961 14784] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 3c56ae9f48ea8e4a4123855c01000000, ip address: 10.134.60.23, port: 9183, worker_id: 804d65237fae0ccc4051f118451dc425d5a989264fbfe5c8335973fd, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,102 I 12961 14784] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 3c56ae9f48ea8e4a4123855c01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,401 I 12961 14784] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 3c56ae9f48ea8e4a4123855c01000000, ip address: 10.134.60.23, port: 9183, worker_id: 804d65237fae0ccc4051f118451dc425d5a989264fbfe5c8335973fd, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,508 E 12961 12961] actor_manager.cc:93: Actor handle already exists 3c56ae9f48ea8e4a4123855c01000000 | |
| [2021-04-29 17:04:29,530 I 12961 12961] core_worker.cc:2064: Finished executing task 647c248c489904fcffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-37339b4d6d06410e8fc08ef1b7c33df978ada4fa5ff95aa39af6f4e4_3995.log <== | |
| [2021-04-29 17:03:44,990 I 3995 3995] core_worker.cc:2064: Finished executing task 16f01148fb1b8184faf20559f68add19a61976bc01000000, status=OK | |
| [2021-04-29 17:04:48,354 I 3995 3995] core_worker.cc:2064: Finished executing task 08700f4e16eb5469faf20559f68add19a61976bc01000000, status=OK | |
| [2021-04-29 17:04:49,768 I 3995 3995] core_worker.cc:2064: Finished executing task b876075bad80610cfaf20559f68add19a61976bc01000000, status=OK | |
| [2021-04-29 17:04:50,367 I 3995 3995] core_worker.cc:2064: Finished executing task c85edb57d68505e9faf20559f68add19a61976bc01000000, status=OK | |
| [2021-04-29 17:04:52,129 I 3995 3995] core_worker.cc:2064: Finished executing task c8ede7d42bc04a8dfaf20559f68add19a61976bc01000000, status=OK | |
| [2021-04-29 17:04:53,004 I 3995 3995] core_worker.cc:2064: Finished executing task da7dc32705fe7f37faf20559f68add19a61976bc01000000, status=OK | |
| [2021-04-29 17:04:54,467 I 3995 3995] core_worker.cc:2064: Finished executing task 10fa808246a89a66faf20559f68add19a61976bc01000000, status=OK | |
| [2021-04-29 17:04:55,152 I 3995 3995] core_worker.cc:2064: Finished executing task 519d472ff6c4c6a9faf20559f68add19a61976bc01000000, status=OK | |
| [2021-04-29 17:04:56,355 I 3995 3995] core_worker.cc:2064: Finished executing task 7f938f90246063bcfaf20559f68add19a61976bc01000000, status=OK | |
| [2021-04-29 17:04:57,794 I 3995 3995] core_worker.cc:2064: Finished executing task 4fdfd2047df0ff78faf20559f68add19a61976bc01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-383089fcce549fda143208fbc3ade6e7e37f54c574446ec1d540877f_18057.log <== | |
| [2021-04-29 17:04:09,703 I 18057 19846] service_based_accessor.cc:579: Received notification for node id = ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, IsAlive = 1 | |
| [2021-04-29 17:04:09,703 I 18057 19846] service_based_accessor.cc:579: Received notification for node id = d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, IsAlive = 1 | |
| [2021-04-29 17:04:09,703 I 18057 19846] service_based_accessor.cc:579: Received notification for node id = b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, IsAlive = 1 | |
| [2021-04-29 17:04:09,703 I 18057 19846] service_based_accessor.cc:579: Received notification for node id = 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, IsAlive = 1 | |
| [2021-04-29 17:04:09,703 I 18057 19846] service_based_accessor.cc:579: Received notification for node id = 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, IsAlive = 1 | |
| [2021-04-29 17:04:10,733 I 18057 19846] core_worker.cc:717: Exit signal received, this process will exit after all outstanding tasks have finished, exit_type=INTENDED_EXIT | |
| [2021-04-29 17:04:10,733 I 18057 18057] raylet_client.cc:129: RayletClient::Disconnect, exit_type=INTENDED_EXIT, has creation_task_exception_pb_bytes=0 | |
| [2021-04-29 17:04:10,734 I 18057 18057] core_worker.cc:308: Removed worker 383089fcce549fda143208fbc3ade6e7e37f54c574446ec1d540877f | |
| [2021-04-29 17:04:10,738 I 18057 18057] core_worker.cc:181: Destructing CoreWorkerProcess. pid: 18057 | |
| [2021-04-29 17:04:10,738 I 18057 18057] io_service_pool.cc:47: IOServicePool is stopped. | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-3857fc6d7e92a9c562668231c3a5b6dcc97abc901ee6f3558fcdfdde_13327.log <== | |
| [2021-04-29 17:04:05,365 I 13327 15110] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,421 E 13327 13327] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,430 I 13327 15110] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:07,809 E 13327 13327] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:07,947 I 13327 13327] direct_task_transport.cc:47: Creating actor via GCS actor id = : 06ba16ba80bdd90a940d2a4f01000000 | |
| [2021-04-29 17:04:07,993 I 13327 15110] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 06ba16ba80bdd90a940d2a4f01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:17,678 I 13327 15110] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 06ba16ba80bdd90a940d2a4f01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,400 I 13327 15110] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 06ba16ba80bdd90a940d2a4f01000000, ip address: 10.134.60.23, port: 9178, worker_id: 2ed7f550379c5697499886fc3c1e715e474df85ae1ad00ba9a8235d8, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,464 E 13327 13327] actor_manager.cc:93: Actor handle already exists 06ba16ba80bdd90a940d2a4f01000000 | |
| [2021-04-29 17:04:29,529 I 13327 13327] core_worker.cc:2064: Finished executing task df638a084e0d2336ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-3858b433d53b7b660a1dfe992219fd121ce17432a61c8f4ba74a85f9_15944.log <== | |
| [2021-04-29 17:04:08,147 I 15944 17241] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,320 E 15944 15944] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,414 I 15944 17241] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,001 E 15944 15944] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,093 I 15944 15944] direct_task_transport.cc:47: Creating actor via GCS actor id = : 1bbe38f17a30915d50d589c001000000 | |
| [2021-04-29 17:04:11,138 I 15944 17241] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 1bbe38f17a30915d50d589c001000000, ip address: 10.134.142.16, port: 9199, worker_id: d4a0f6fcdc9e6438947273d41a30ec24fdf3ee6e8a44bea142c4c4cc, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,372 I 15944 17241] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 1bbe38f17a30915d50d589c001000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,367 I 15944 17241] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1bbe38f17a30915d50d589c001000000, ip address: 10.134.142.16, port: 9199, worker_id: d4a0f6fcdc9e6438947273d41a30ec24fdf3ee6e8a44bea142c4c4cc, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:27,093 E 15944 15944] actor_manager.cc:93: Actor handle already exists 1bbe38f17a30915d50d589c001000000 | |
| [2021-04-29 17:04:29,791 I 15944 15944] core_worker.cc:2064: Finished executing task 08dcfe2348af132effffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-38996c5e1e5cf79ab7d012b6f32bda179f9bdc18c0b7c1469752403b_12862.log <== | |
| [2021-04-29 17:04:04,938 I 12862 13928] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,982 E 12862 12862] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:04,999 I 12862 13928] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,873 E 12862 12862] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:07,070 I 12862 12862] direct_task_transport.cc:47: Creating actor via GCS actor id = : 9158fab4719f717146c4644701000000 | |
| [2021-04-29 17:04:07,105 I 12862 13928] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 9158fab4719f717146c4644701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:15,458 I 12862 13928] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 9158fab4719f717146c4644701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,397 I 12862 13928] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 9158fab4719f717146c4644701000000, ip address: 10.134.70.227, port: 9234, worker_id: 95ba7bb06c5de4e45cf632eab0d67edffd3c540f6dc16c17bc104dbe, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,335 E 12862 12862] actor_manager.cc:93: Actor handle already exists 9158fab4719f717146c4644701000000 | |
| [2021-04-29 17:04:29,504 I 12862 12862] core_worker.cc:2064: Finished executing task 7069734bbbeaa44cffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-38a5abb99f0d6b4c0064a2e40d19507d52704970402724ad53d45472_7478.log <== | |
| [2021-04-29 17:03:59,426 I 7478 8492] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,447 E 7478 7478] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,452 I 7478 8492] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,755 E 7478 7478] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,820 I 7478 7478] direct_task_transport.cc:47: Creating actor via GCS actor id = : 248bdd55016e39fc8ea32e1401000000 | |
| [2021-04-29 17:03:59,833 I 7478 8492] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 248bdd55016e39fc8ea32e1401000000, ip address: 10.134.61.130, port: 9115, worker_id: 20285f85f888e212bb9ad113c8a41e803cd14da68061a9b623d9c372, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,726 I 7478 8492] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 248bdd55016e39fc8ea32e1401000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,323 I 7478 8492] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 248bdd55016e39fc8ea32e1401000000, ip address: 10.134.61.130, port: 9115, worker_id: 20285f85f888e212bb9ad113c8a41e803cd14da68061a9b623d9c372, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,878 E 7478 7478] actor_manager.cc:93: Actor handle already exists 248bdd55016e39fc8ea32e1401000000 | |
| [2021-04-29 17:04:29,401 I 7478 7478] core_worker.cc:2064: Finished executing task 1f2f28a5133076d2ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-396398af0ee8d1d4026383bb7ee6192f1cd97c96318749f7043ebf44_14269.log <== | |
| [2021-04-29 17:04:05,742 E 14269 14269] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,780 I 14269 15565] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,795 I 14269 15565] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,241 E 14269 14269] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:08,414 I 14269 14269] direct_task_transport.cc:47: Creating actor via GCS actor id = : 63d6015c5c5dfc2eef4242e401000000 | |
| [2021-04-29 17:04:08,445 I 14269 15565] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 63d6015c5c5dfc2eef4242e401000000, ip address: 10.134.70.227, port: 9220, worker_id: b588cc7f19cfe20f318a39de92047b589ef64166324c5e9395f47b26, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:19,125 I 14269 15565] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 63d6015c5c5dfc2eef4242e401000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:22,739 I 14269 15565] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 63d6015c5c5dfc2eef4242e401000000, ip address: 10.134.70.227, port: 9220, worker_id: b588cc7f19cfe20f318a39de92047b589ef64166324c5e9395f47b26, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:25,467 E 14269 14269] actor_manager.cc:93: Actor handle already exists 63d6015c5c5dfc2eef4242e401000000 | |
| [2021-04-29 17:04:29,750 I 14269 14269] core_worker.cc:2064: Finished executing task e13a6155abe0cf3effffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-3a920047928c5eadb3ae96982a81d473dda3f1681ba113402f81d7e3_17370.log <== | |
| [2021-04-29 17:04:09,422 I 17370 19012] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,593 E 17370 17370] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,620 I 17370 19012] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,530 E 17370 17370] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,617 I 17370 17370] direct_task_transport.cc:47: Creating actor via GCS actor id = : f954578d545f76c79b5c453101000000 | |
| [2021-04-29 17:04:11,668 I 17370 19012] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: f954578d545f76c79b5c453101000000, ip address: 10.134.61.130, port: 9211, worker_id: 03ac60ed1559d8c5784636e8b6e19230bfcd584e00466e0ed41a6511, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,383 I 17370 19012] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: f954578d545f76c79b5c453101000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:28,101 I 17370 19012] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: f954578d545f76c79b5c453101000000, ip address: 10.134.61.130, port: 9211, worker_id: 03ac60ed1559d8c5784636e8b6e19230bfcd584e00466e0ed41a6511, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,993 E 17370 17370] actor_manager.cc:93: Actor handle already exists f954578d545f76c79b5c453101000000 | |
| [2021-04-29 17:04:29,862 I 17370 17370] core_worker.cc:2064: Finished executing task fae00d9dfb71113bffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-3aa28e70b3c46285ea9f2c35368cdcc7dabb5e4323ebc5e19f02be16_17515.log <== | |
| [2021-04-29 17:04:09,421 I 17515 18982] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,594 E 17515 17515] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,620 I 17515 18982] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,532 E 17515 17515] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,620 I 17515 17515] direct_task_transport.cc:47: Creating actor via GCS actor id = : 0c22e2c3fc0934709a17350601000000 | |
| [2021-04-29 17:04:11,667 I 17515 18982] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 0c22e2c3fc0934709a17350601000000, ip address: 10.133.95.40, port: 9186, worker_id: c02de6654e328d86ca00ae3d76c4d9c101ae34e76c3c1303c4eed164, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,384 I 17515 18982] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 0c22e2c3fc0934709a17350601000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:25,777 I 17515 18982] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 0c22e2c3fc0934709a17350601000000, ip address: 10.133.95.40, port: 9186, worker_id: c02de6654e328d86ca00ae3d76c4d9c101ae34e76c3c1303c4eed164, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,514 E 17515 17515] actor_manager.cc:93: Actor handle already exists 0c22e2c3fc0934709a17350601000000 | |
| [2021-04-29 17:04:29,843 I 17515 17515] core_worker.cc:2064: Finished executing task d3c6b7ddcaed7142ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-3b50ce9249d4393d82997a4ba38e570d581fe4c93017677e5ffc42c7_17130.log <== | |
| [2021-04-29 17:04:08,966 I 17130 18887] service_based_accessor.cc:579: Received notification for node id = ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, IsAlive = 1 | |
| [2021-04-29 17:04:08,966 I 17130 18887] service_based_accessor.cc:579: Received notification for node id = d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, IsAlive = 1 | |
| [2021-04-29 17:04:08,966 I 17130 18887] service_based_accessor.cc:579: Received notification for node id = b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, IsAlive = 1 | |
| [2021-04-29 17:04:08,966 I 17130 18887] service_based_accessor.cc:579: Received notification for node id = 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, IsAlive = 1 | |
| [2021-04-29 17:04:08,966 I 17130 18887] service_based_accessor.cc:579: Received notification for node id = 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, IsAlive = 1 | |
| [2021-04-29 17:04:10,129 I 17130 18887] core_worker.cc:717: Exit signal received, this process will exit after all outstanding tasks have finished, exit_type=INTENDED_EXIT | |
| [2021-04-29 17:04:10,129 I 17130 17130] raylet_client.cc:129: RayletClient::Disconnect, exit_type=INTENDED_EXIT, has creation_task_exception_pb_bytes=0 | |
| [2021-04-29 17:04:10,129 I 17130 17130] core_worker.cc:308: Removed worker 3b50ce9249d4393d82997a4ba38e570d581fe4c93017677e5ffc42c7 | |
| [2021-04-29 17:04:10,133 I 17130 17130] core_worker.cc:181: Destructing CoreWorkerProcess. pid: 17130 | |
| [2021-04-29 17:04:10,133 I 17130 17130] io_service_pool.cc:47: IOServicePool is stopped. | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-3bc8e6a4f5cf63a5e34c4216c333820848057d7781cf0560ec0267dc_17765.log <== | |
| [2021-04-29 17:04:09,177 I 17765 19198] service_based_accessor.cc:579: Received notification for node id = ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, IsAlive = 1 | |
| [2021-04-29 17:04:09,177 I 17765 19198] service_based_accessor.cc:579: Received notification for node id = d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, IsAlive = 1 | |
| [2021-04-29 17:04:09,177 I 17765 19198] service_based_accessor.cc:579: Received notification for node id = b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, IsAlive = 1 | |
| [2021-04-29 17:04:09,177 I 17765 19198] service_based_accessor.cc:579: Received notification for node id = 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, IsAlive = 1 | |
| [2021-04-29 17:04:09,177 I 17765 19198] service_based_accessor.cc:579: Received notification for node id = 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, IsAlive = 1 | |
| [2021-04-29 17:04:10,332 I 17765 19198] core_worker.cc:717: Exit signal received, this process will exit after all outstanding tasks have finished, exit_type=INTENDED_EXIT | |
| [2021-04-29 17:04:10,332 I 17765 17765] raylet_client.cc:129: RayletClient::Disconnect, exit_type=INTENDED_EXIT, has creation_task_exception_pb_bytes=0 | |
| [2021-04-29 17:04:10,332 I 17765 17765] core_worker.cc:308: Removed worker 3bc8e6a4f5cf63a5e34c4216c333820848057d7781cf0560ec0267dc | |
| [2021-04-29 17:04:10,335 I 17765 17765] core_worker.cc:181: Destructing CoreWorkerProcess. pid: 17765 | |
| [2021-04-29 17:04:10,335 I 17765 17765] io_service_pool.cc:47: IOServicePool is stopped. | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-3cd23c5a486546daef316e3b62f52903be8ee720960a538cd17b4ef6_4023.log <== | |
| [2021-04-29 17:03:44,982 I 4023 4023] core_worker.cc:2064: Finished executing task 442c6011791fb0f90ff93096536654a818f7d24e01000000, status=OK | |
| [2021-04-29 17:04:39,019 I 4023 4023] core_worker.cc:2064: Finished executing task 3f7567ea63e40f600ff93096536654a818f7d24e01000000, status=OK | |
| [2021-04-29 17:04:49,160 I 4023 4023] core_worker.cc:2064: Finished executing task 127da05faf14ddcf0ff93096536654a818f7d24e01000000, status=OK | |
| [2021-04-29 17:04:49,505 I 4023 4023] core_worker.cc:2064: Finished executing task ab659b19a36cafaf0ff93096536654a818f7d24e01000000, status=OK | |
| [2021-04-29 17:04:51,618 I 4023 4023] core_worker.cc:2064: Finished executing task 251f7c6fad04b2b50ff93096536654a818f7d24e01000000, status=OK | |
| [2021-04-29 17:04:52,614 I 4023 4023] core_worker.cc:2064: Finished executing task b827e97025aeeaa40ff93096536654a818f7d24e01000000, status=OK | |
| [2021-04-29 17:04:53,057 I 4023 4023] core_worker.cc:2064: Finished executing task 8349fcf1701d7c610ff93096536654a818f7d24e01000000, status=OK | |
| [2021-04-29 17:04:54,696 I 4023 4023] core_worker.cc:2064: Finished executing task dce51bf869eb2c310ff93096536654a818f7d24e01000000, status=OK | |
| [2021-04-29 17:04:55,578 I 4023 4023] core_worker.cc:2064: Finished executing task b6b437d7bc0e05b30ff93096536654a818f7d24e01000000, status=OK | |
| [2021-04-29 17:04:56,890 I 4023 4023] core_worker.cc:2064: Finished executing task cf9e0414e20b5a500ff93096536654a818f7d24e01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-3da6f91a24ffb83b5cfcee05a85c7a052ddfc316780a242df3475b5a_4036.log <== | |
| [2021-04-29 17:04:02,146 E 4036 4036] actor_manager.cc:93: Actor handle already exists 8f94dceff41d429f537839ee01000000 | |
| [2021-04-29 17:04:09,673 I 4036 4036] core_worker.cc:2064: Finished executing task abe04aeece53ecf6ffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:09,683 E 4036 4036] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,683 E 4036 4036] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,766 I 4036 4036] direct_task_transport.cc:47: Creating actor via GCS actor id = : 0436aedd01b0cf849e41093f01000000 | |
| [2021-04-29 17:04:09,790 I 4036 4466] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 0436aedd01b0cf849e41093f01000000, ip address: 10.134.70.227, port: 9213, worker_id: 30197a4a5349c7ed469177288598b128c4eb497d1dfe426bc3940a33, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,359 I 4036 4466] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 0436aedd01b0cf849e41093f01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,301 I 4036 4466] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 0436aedd01b0cf849e41093f01000000, ip address: 10.134.70.227, port: 9213, worker_id: 30197a4a5349c7ed469177288598b128c4eb497d1dfe426bc3940a33, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:27,083 E 4036 4036] actor_manager.cc:93: Actor handle already exists 0436aedd01b0cf849e41093f01000000 | |
| [2021-04-29 17:04:29,781 I 4036 4036] core_worker.cc:2064: Finished executing task db9839993ad38312ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-3dbe40c8afac89dcacb3027bb5da146b46a3c4b0c7974e37f5964aef_14254.log <== | |
| [2021-04-29 17:04:05,911 I 14254 15776] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,969 E 14254 14254] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,981 I 14254 15776] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,772 E 14254 14254] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:08,885 I 14254 14254] direct_task_transport.cc:47: Creating actor via GCS actor id = : bb966e8162e1e7c94dd055ed01000000 | |
| [2021-04-29 17:04:08,922 I 14254 15776] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: bb966e8162e1e7c94dd055ed01000000, ip address: 10.133.95.40, port: 9182, worker_id: 68c455b63ce46bc0fd8d613237048a2d28137774ec6eb440a63fc085, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:19,755 I 14254 15776] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: bb966e8162e1e7c94dd055ed01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,401 I 14254 15776] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: bb966e8162e1e7c94dd055ed01000000, ip address: 10.133.95.40, port: 9182, worker_id: 68c455b63ce46bc0fd8d613237048a2d28137774ec6eb440a63fc085, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,559 E 14254 14254] actor_manager.cc:93: Actor handle already exists bb966e8162e1e7c94dd055ed01000000 | |
| [2021-04-29 17:04:29,539 I 14254 14254] core_worker.cc:2064: Finished executing task e9269d435ff0e637ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-3dcaec3fad7a69fe54a99023f805655cbf3eaee6aa1879c36a31cbb6_17799.log <== | |
| [2021-04-29 17:04:09,621 I 17799 19334] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,792 E 17799 17799] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,819 I 17799 19334] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,576 E 17799 17799] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,721 I 17799 17799] direct_task_transport.cc:47: Creating actor via GCS actor id = : e5f01ecc0b0040e6555c285f01000000 | |
| [2021-04-29 17:04:11,754 I 17799 19334] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: e5f01ecc0b0040e6555c285f01000000, ip address: 10.134.60.23, port: 9163, worker_id: 8f88b1ef1ed2fd0ca42a96c60ba80419111071867e8c2232c41228b0, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,388 I 17799 19334] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: e5f01ecc0b0040e6555c285f01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:27,414 I 17799 19334] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: e5f01ecc0b0040e6555c285f01000000, ip address: 10.134.60.23, port: 9163, worker_id: 8f88b1ef1ed2fd0ca42a96c60ba80419111071867e8c2232c41228b0, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,857 E 17799 17799] actor_manager.cc:93: Actor handle already exists e5f01ecc0b0040e6555c285f01000000 | |
| [2021-04-29 17:04:29,858 I 17799 17799] core_worker.cc:2064: Finished executing task 22907372f28f6e90ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-3e8db03400845bc98f99458b1d00c10df5ef005a472bdaa17ed7baff_12091.log <== | |
| [2021-04-29 17:04:03,386 I 12091 12561] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,412 E 12091 12091] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:03,424 I 12091 12561] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,870 E 12091 12091] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,951 I 12091 12091] direct_task_transport.cc:47: Creating actor via GCS actor id = : 0500bd5c5d4d8d6006827ee601000000 | |
| [2021-04-29 17:04:05,967 I 12091 12561] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 0500bd5c5d4d8d6006827ee601000000, ip address: 10.133.95.40, port: 9104, worker_id: a350b27461bf465c29c49106596fe44447ffad9e45ce9a2a69ccc70e, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,437 I 12091 12561] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 0500bd5c5d4d8d6006827ee601000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,395 I 12091 12561] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 0500bd5c5d4d8d6006827ee601000000, ip address: 10.133.95.40, port: 9104, worker_id: a350b27461bf465c29c49106596fe44447ffad9e45ce9a2a69ccc70e, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,253 E 12091 12091] actor_manager.cc:93: Actor handle already exists 0500bd5c5d4d8d6006827ee601000000 | |
| [2021-04-29 17:04:29,493 I 12091 12091] core_worker.cc:2064: Finished executing task b1fa8f3b49486ca4ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-3ef98c110aa0947f10992a9dc882d9f32d2ca05481323a2ecea0eea6_17077.log <== | |
| [2021-04-29 17:04:09,294 I 17077 18709] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,434 E 17077 17077] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,460 I 17077 18709] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,487 E 17077 17077] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,586 I 17077 17077] direct_task_transport.cc:47: Creating actor via GCS actor id = : 84a172ba66e133300130220901000000 | |
| [2021-04-29 17:04:11,625 I 17077 18709] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 84a172ba66e133300130220901000000, ip address: 10.134.142.44, port: 9198, worker_id: b5295da21c05485b002454cc9832db61cd2f952ed338be272470907b, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,382 I 17077 18709] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 84a172ba66e133300130220901000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:27,720 I 17077 18709] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 84a172ba66e133300130220901000000, ip address: 10.134.142.44, port: 9198, worker_id: b5295da21c05485b002454cc9832db61cd2f952ed338be272470907b, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,959 E 17077 17077] actor_manager.cc:93: Actor handle already exists 84a172ba66e133300130220901000000 | |
| [2021-04-29 17:04:29,855 I 17077 17077] core_worker.cc:2064: Finished executing task 68aced00c6a3fa98ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-3f8044e8120500ad3ace00bb3f7b138dedd2e3bb38850dea24e2ac3e_4057.log <== | |
| [2021-04-29 17:03:44,976 I 4057 4057] core_worker.cc:2064: Finished executing task 355c65057edaf1f3bc2df8bd268052d5b6aab9a901000000, status=OK | |
| [2021-04-29 17:04:34,762 I 4057 4057] core_worker.cc:2064: Finished executing task 1152a332429db217bc2df8bd268052d5b6aab9a901000000, status=OK | |
| [2021-04-29 17:04:49,088 I 4057 4057] core_worker.cc:2064: Finished executing task b64fb65d41681491bc2df8bd268052d5b6aab9a901000000, status=OK | |
| [2021-04-29 17:04:50,197 I 4057 4057] core_worker.cc:2064: Finished executing task 65b8003503bb8061bc2df8bd268052d5b6aab9a901000000, status=OK | |
| [2021-04-29 17:04:50,625 I 4057 4057] core_worker.cc:2064: Finished executing task ce852e0b856441e5bc2df8bd268052d5b6aab9a901000000, status=OK | |
| [2021-04-29 17:04:51,882 I 4057 4057] core_worker.cc:2064: Finished executing task 2d833b07f6d6a227bc2df8bd268052d5b6aab9a901000000, status=OK | |
| [2021-04-29 17:04:54,430 I 4057 4057] core_worker.cc:2064: Finished executing task 99899eb19f93a517bc2df8bd268052d5b6aab9a901000000, status=OK | |
| [2021-04-29 17:04:56,027 I 4057 4057] core_worker.cc:2064: Finished executing task 47b64edb3878c1acbc2df8bd268052d5b6aab9a901000000, status=OK | |
| [2021-04-29 17:04:56,261 I 4057 4057] core_worker.cc:2064: Finished executing task 1e8086bd17dbc2ddbc2df8bd268052d5b6aab9a901000000, status=OK | |
| [2021-04-29 17:04:57,546 I 4057 4057] core_worker.cc:2064: Finished executing task bf6eb8b98678f4f2bc2df8bd268052d5b6aab9a901000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-3f894b8796757a31396124ec782a7d2fbd62ebffbc7109b088190b87_7520.log <== | |
| [2021-04-29 17:03:59,225 I 7520 8778] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,247 E 7520 7520] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,254 I 7520 8778] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,685 E 7520 7520] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,749 I 7520 7520] direct_task_transport.cc:47: Creating actor via GCS actor id = : 6934e155697d7f66c805ae7301000000 | |
| [2021-04-29 17:03:59,759 I 7520 8778] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 6934e155697d7f66c805ae7301000000, ip address: 10.134.70.227, port: 9117, worker_id: 7500492fd073042115367a53fd04606c82336b64c5b6bcfccef4f10a, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,724 I 7520 8778] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 6934e155697d7f66c805ae7301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:19,263 I 7520 8778] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 6934e155697d7f66c805ae7301000000, ip address: 10.134.70.227, port: 9117, worker_id: 7500492fd073042115367a53fd04606c82336b64c5b6bcfccef4f10a, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,040 E 7520 7520] actor_manager.cc:93: Actor handle already exists 6934e155697d7f66c805ae7301000000 | |
| [2021-04-29 17:04:28,908 I 7520 7520] core_worker.cc:2064: Finished executing task 8f8419487afdf03dffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-3fcf2b214904afe8819e8267fd007f9cde55d784946130f12031a93c_7433.log <== | |
| [2021-04-29 17:03:58,071 I 7433 7927] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,087 E 7433 7433] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,105 I 7433 7927] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,699 E 7433 7433] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:58,778 I 7433 7433] direct_task_transport.cc:47: Creating actor via GCS actor id = : 244d989d722b422879aa400801000000 | |
| [2021-04-29 17:03:58,787 I 7433 7927] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 244d989d722b422879aa400801000000, ip address: 10.134.142.16, port: 9138, worker_id: 5a503e57a2deb2ba36249c1f25bceb46fd5935c69474d3dae61fa4e1, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,381 I 7433 7927] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 244d989d722b422879aa400801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,548 I 7433 7927] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 244d989d722b422879aa400801000000, ip address: 10.134.142.16, port: 9138, worker_id: 5a503e57a2deb2ba36249c1f25bceb46fd5935c69474d3dae61fa4e1, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:05,987 E 7433 7433] actor_manager.cc:93: Actor handle already exists 244d989d722b422879aa400801000000 | |
| [2021-04-29 17:04:11,921 I 7433 7433] core_worker.cc:2064: Finished executing task 3d4e63da4ad8e4baffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-3ff9c8c85ac11465abd691573e96244cc647fa2ab3ce2f032ec680be_12348.log <== | |
| [2021-04-29 17:04:04,519 I 12348 13204] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,585 E 12348 12348] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:04,615 I 12348 13204] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,514 E 12348 12348] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:06,617 I 12348 12348] direct_task_transport.cc:47: Creating actor via GCS actor id = : fd8524ca5c1f1ae92ce8da1801000000 | |
| [2021-04-29 17:04:06,680 I 12348 13204] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: fd8524ca5c1f1ae92ce8da1801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:12,594 I 12348 13204] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: fd8524ca5c1f1ae92ce8da1801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,396 I 12348 13204] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: fd8524ca5c1f1ae92ce8da1801000000, ip address: 10.134.142.44, port: 9236, worker_id: 6c351070ca23f0aa9f630a0a6ffb2f6c1cdd8a57131c3279a1a3b78e, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,342 E 12348 12348] actor_manager.cc:93: Actor handle already exists fd8524ca5c1f1ae92ce8da1801000000 | |
| [2021-04-29 17:04:29,507 I 12348 12348] core_worker.cc:2064: Finished executing task 7a354da1f1f93532ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-41596038e90f30b01ac8e7c4dfd227de1ee176af325f9a2f14d65e82_9692.log <== | |
| [2021-04-29 17:04:01,231 I 9692 10537] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,256 E 9692 9692] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,264 I 9692 10537] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,906 E 9692 9692] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:01,957 I 9692 9692] direct_task_transport.cc:47: Creating actor via GCS actor id = : ec09b52f4d6cd95e12e0623101000000 | |
| [2021-04-29 17:04:01,975 I 9692 10537] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: ec09b52f4d6cd95e12e0623101000000, ip address: 10.134.70.227, port: 9107, worker_id: 52927b3fc14ff5a690a5afbdf33b207163cfefa7eb536c381c09d663, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,524 I 9692 10537] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: ec09b52f4d6cd95e12e0623101000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,390 I 9692 10537] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: ec09b52f4d6cd95e12e0623101000000, ip address: 10.134.70.227, port: 9107, worker_id: 52927b3fc14ff5a690a5afbdf33b207163cfefa7eb536c381c09d663, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,037 E 9692 9692] actor_manager.cc:93: Actor handle already exists ec09b52f4d6cd95e12e0623101000000 | |
| [2021-04-29 17:04:29,442 I 9692 9692] core_worker.cc:2064: Finished executing task b50d5087fe74c7faffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-41e010e9d72032dccacf232acb1ebc982775a7dcc40e7cfe2ecaafe5_16030.log <== | |
| [2021-04-29 17:04:08,666 I 16030 17853] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,745 E 16030 16030] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,771 I 16030 17853] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,185 E 16030 16030] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,302 I 16030 16030] direct_task_transport.cc:47: Creating actor via GCS actor id = : 3b8cda4cd232b10eab55c31901000000 | |
| [2021-04-29 17:04:11,327 I 16030 17853] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 3b8cda4cd232b10eab55c31901000000, ip address: 10.134.112.222, port: 9192, worker_id: 0c4408f4947edef794f8622d99d9bca6f4a5f56308bce28651e4762f, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,378 I 16030 17853] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 3b8cda4cd232b10eab55c31901000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:25,068 I 16030 17853] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 3b8cda4cd232b10eab55c31901000000, ip address: 10.134.112.222, port: 9192, worker_id: 0c4408f4947edef794f8622d99d9bca6f4a5f56308bce28651e4762f, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:27,998 E 16030 16030] actor_manager.cc:93: Actor handle already exists 3b8cda4cd232b10eab55c31901000000 | |
| [2021-04-29 17:04:29,816 I 16030 16030] core_worker.cc:2064: Finished executing task 32103517bde2e40cffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-42232618cd40d7d97e6aac5291f32248f66e61a2e3df17d29af44209_9755.log <== | |
| [2021-04-29 17:04:01,454 I 9755 11058] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,475 E 9755 9755] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,482 I 9755 11058] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,026 E 9755 9755] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:02,098 I 9755 9755] direct_task_transport.cc:47: Creating actor via GCS actor id = : a5a8b7912851c51a58d726c301000000 | |
| [2021-04-29 17:04:02,107 I 9755 11058] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: a5a8b7912851c51a58d726c301000000, ip address: 10.134.112.222, port: 9111, worker_id: e8b601fdd653262c54cec90ed6162751b8ac7a48c117caac56a3185b, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,544 I 9755 11058] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: a5a8b7912851c51a58d726c301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,391 I 9755 11058] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: a5a8b7912851c51a58d726c301000000, ip address: 10.134.112.222, port: 9111, worker_id: e8b601fdd653262c54cec90ed6162751b8ac7a48c117caac56a3185b, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,101 E 9755 9755] actor_manager.cc:93: Actor handle already exists a5a8b7912851c51a58d726c301000000 | |
| [2021-04-29 17:04:29,457 I 9755 9755] core_worker.cc:2064: Finished executing task b5ba1735bcd80fe4ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-42ddcc37c4e87b569c34f929724f7fe8614015eeb7600e85bf7bc4bf_14355.log <== | |
| [2021-04-29 17:04:05,949 I 14355 15757] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,987 E 14355 14355] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:06,007 I 14355 15757] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,778 E 14355 14355] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:08,871 I 14355 14355] direct_task_transport.cc:47: Creating actor via GCS actor id = : e67f527fd4312dc78f77676601000000 | |
| [2021-04-29 17:04:08,931 I 14355 15757] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: e67f527fd4312dc78f77676601000000, ip address: 10.134.112.222, port: 9181, worker_id: 3b0eae3ee9988a4eb8e54771a38f7e01bf1615d19a97f11be900ec0b, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:19,504 I 14355 15757] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: e67f527fd4312dc78f77676601000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,401 I 14355 15757] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: e67f527fd4312dc78f77676601000000, ip address: 10.134.112.222, port: 9181, worker_id: 3b0eae3ee9988a4eb8e54771a38f7e01bf1615d19a97f11be900ec0b, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,552 E 14355 14355] actor_manager.cc:93: Actor handle already exists e67f527fd4312dc78f77676601000000 | |
| [2021-04-29 17:04:29,536 I 14355 14355] core_worker.cc:2064: Finished executing task d486cfa03dc04c31ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-43127d17f7909b8edf60fd49ae5f0d80a25fd6875b784e221c73fde1_12081.log <== | |
| [2021-04-29 17:04:03,323 I 12081 12332] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,349 E 12081 12081] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:03,358 I 12081 12332] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,864 E 12081 12081] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,945 I 12081 12081] direct_task_transport.cc:47: Creating actor via GCS actor id = : 475f846b8f1bc563195cbd2801000000 | |
| [2021-04-29 17:04:05,968 I 12081 12332] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 475f846b8f1bc563195cbd2801000000, ip address: 10.134.112.222, port: 9106, worker_id: f9a9238d90f726ceb9d65a12ccbe610aeebf759c9c214ed61a1fd9ed, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,034 I 12081 12332] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 475f846b8f1bc563195cbd2801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,394 I 12081 12332] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 475f846b8f1bc563195cbd2801000000, ip address: 10.134.112.222, port: 9106, worker_id: f9a9238d90f726ceb9d65a12ccbe610aeebf759c9c214ed61a1fd9ed, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,209 E 12081 12081] actor_manager.cc:93: Actor handle already exists 475f846b8f1bc563195cbd2801000000 | |
| [2021-04-29 17:04:29,481 I 12081 12081] core_worker.cc:2064: Finished executing task 700a88c2e9653333ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-433c3c7c456e23db446cf653dc33977497b5ac69be87b7ff0c5cdfd2_12120.log <== | |
| [2021-04-29 17:04:03,477 I 12120 12657] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,510 E 12120 12120] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:03,519 I 12120 12657] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,876 E 12120 12120] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,944 I 12120 12120] direct_task_transport.cc:47: Creating actor via GCS actor id = : 046e2a0f0e831e2087c6256901000000 | |
| [2021-04-29 17:04:05,967 I 12120 12657] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 046e2a0f0e831e2087c6256901000000, ip address: 10.134.61.130, port: 13217, worker_id: NIL_ID, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,931 I 12120 12657] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 046e2a0f0e831e2087c6256901000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,396 I 12120 12657] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 046e2a0f0e831e2087c6256901000000, ip address: 10.134.61.130, port: 9148, worker_id: 4b4026c1f4d61032882a2e016c2d89a69d7d070aa6e567bbb84724af, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,300 E 12120 12120] actor_manager.cc:93: Actor handle already exists 046e2a0f0e831e2087c6256901000000 | |
| [2021-04-29 17:04:29,489 I 12120 12120] core_worker.cc:2064: Finished executing task 26039677217bb97effffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-435eedefd66a2f89c57ba303a1470955822807f5bd710f58b262e504_7507.log <== | |
| [2021-04-29 17:03:59,263 I 7507 8612] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,282 E 7507 7507] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,288 I 7507 8612] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,687 E 7507 7507] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,746 I 7507 7507] direct_task_transport.cc:47: Creating actor via GCS actor id = : d257ed14042e9c0c995a053b01000000 | |
| [2021-04-29 17:03:59,759 I 7507 8612] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: d257ed14042e9c0c995a053b01000000, ip address: 10.134.60.23, port: 9127, worker_id: 19a174bdb6605befcce7898eaf25f9e3fd6ae8e5dee76755087ab384, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,724 I 7507 8612] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: d257ed14042e9c0c995a053b01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:19,263 I 7507 8612] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: d257ed14042e9c0c995a053b01000000, ip address: 10.134.60.23, port: 9127, worker_id: 19a174bdb6605befcce7898eaf25f9e3fd6ae8e5dee76755087ab384, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,051 E 7507 7507] actor_manager.cc:93: Actor handle already exists d257ed14042e9c0c995a053b01000000 | |
| [2021-04-29 17:04:28,912 I 7507 7507] core_worker.cc:2064: Finished executing task 345cc52898ff9ff8ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-4383956ecf680c5790b14a9f6b1c40dad6b6c5e4ab3cddd9a573cff5_9714.log <== | |
| [2021-04-29 17:04:01,192 I 9714 10437] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,219 E 9714 9714] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,230 I 9714 10437] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,777 E 9714 9714] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:01,836 I 9714 9714] direct_task_transport.cc:47: Creating actor via GCS actor id = : 2604272260f1b176eff54d2801000000 | |
| [2021-04-29 17:04:01,842 I 9714 10437] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 2604272260f1b176eff54d2801000000, ip address: 10.134.112.222, port: 21101, worker_id: NIL_ID, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,363 I 9714 10437] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 2604272260f1b176eff54d2801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,390 I 9714 10437] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 2604272260f1b176eff54d2801000000, ip address: 10.134.112.222, port: 9118, worker_id: 41e64341f5a73f5ea580938d3652c3f4fa33150389d65d0444666c2e, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,735 E 9714 9714] actor_manager.cc:93: Actor handle already exists 2604272260f1b176eff54d2801000000 | |
| [2021-04-29 17:04:29,542 I 9714 9714] core_worker.cc:2064: Finished executing task 8a8361e333f44505ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-43b9b5b213006116a3baa4c1246abfbe22266af95f0fb8368b874a3d_5221.log <== | |
| [2021-04-29 17:03:56,693 I 5221 5931] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:56,712 E 5221 5221] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:56,716 I 5221 5931] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:57,097 E 5221 5221] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:57,153 I 5221 5221] direct_task_transport.cc:47: Creating actor via GCS actor id = : f6a7fb977eea7a7ad0db5ad401000000 | |
| [2021-04-29 17:03:57,157 I 5221 5931] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: f6a7fb977eea7a7ad0db5ad401000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:57,424 I 5221 5931] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: f6a7fb977eea7a7ad0db5ad401000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,729 I 5221 5931] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: f6a7fb977eea7a7ad0db5ad401000000, ip address: 10.134.61.25, port: 9143, worker_id: c31588d45f9e59fae362ca0f7b92e7f75e5ebb5b1e552c1fa38d0d1d, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:02,380 E 5221 5221] actor_manager.cc:93: Actor handle already exists f6a7fb977eea7a7ad0db5ad401000000 | |
| [2021-04-29 17:04:10,154 I 5221 5221] core_worker.cc:2064: Finished executing task 88714be0931edae7ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-442ec7e59c902e969d1432ea179f3e2985b70c3a7b278fdfee37f1d8_15969.log <== | |
| [2021-04-29 17:04:08,416 I 15969 17591] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,505 E 15969 15969] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,538 I 15969 17591] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,054 E 15969 15969] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,165 I 15969 15969] direct_task_transport.cc:47: Creating actor via GCS actor id = : d552afd85c03ac1843fb6f7c01000000 | |
| [2021-04-29 17:04:11,194 I 15969 17591] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: d552afd85c03ac1843fb6f7c01000000, ip address: 10.134.70.227, port: 9208, worker_id: f083e5a713f3eca21f62d2cace3c7585d3eda6a84d85f74a41581b29, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,376 I 15969 17591] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: d552afd85c03ac1843fb6f7c01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:25,727 I 15969 17591] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: d552afd85c03ac1843fb6f7c01000000, ip address: 10.134.70.227, port: 9208, worker_id: f083e5a713f3eca21f62d2cace3c7585d3eda6a84d85f74a41581b29, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,486 E 15969 15969] actor_manager.cc:93: Actor handle already exists d552afd85c03ac1843fb6f7c01000000 | |
| [2021-04-29 17:04:29,848 I 15969 15969] core_worker.cc:2064: Finished executing task 6bb0196bc514f89cffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-44d8f489f30d33f2b34e64d3cc0ddd3f160bcfaf958bc2931e65ac27_7453.log <== | |
| [2021-04-29 17:03:58,297 I 7453 8456] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,299 E 7453 7453] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,301 I 7453 8456] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,716 E 7453 7453] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:58,801 I 7453 7453] direct_task_transport.cc:47: Creating actor via GCS actor id = : 3692644e35e110bbd48eaa6601000000 | |
| [2021-04-29 17:03:58,803 I 7453 8456] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 3692644e35e110bbd48eaa6601000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,546 I 7453 8456] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 3692644e35e110bbd48eaa6601000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,902 I 7453 8456] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 3692644e35e110bbd48eaa6601000000, ip address: 10.134.112.222, port: 9135, worker_id: 8f61c9e7d01df9e988205c55dbefe64bb8e8181da3e6d5eaf4d22229, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:06,750 E 7453 7453] actor_manager.cc:93: Actor handle already exists 3692644e35e110bbd48eaa6601000000 | |
| [2021-04-29 17:04:12,383 I 7453 7453] core_worker.cc:2064: Finished executing task c55c0848eaad6353ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-44e6b809774f802b4255746bbdfd29bc3b797e5ce0d2de8eecd3bc21_12846.log <== | |
| [2021-04-29 17:04:04,877 I 12846 13776] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,941 E 12846 12846] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:04,959 I 12846 13776] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,813 E 12846 12846] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:06,980 I 12846 12846] direct_task_transport.cc:47: Creating actor via GCS actor id = : 1cf52e88cb744e148732b32d01000000 | |
| [2021-04-29 17:04:07,054 I 12846 13776] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 1cf52e88cb744e148732b32d01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:14,643 I 12846 13776] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 1cf52e88cb744e148732b32d01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,398 I 12846 13776] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1cf52e88cb744e148732b32d01000000, ip address: 10.134.60.23, port: 9153, worker_id: 581cc449a6f71a43f7e679e3ae3f9d781a73a58cc561261788d8fa61, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,394 E 12846 12846] actor_manager.cc:93: Actor handle already exists 1cf52e88cb744e148732b32d01000000 | |
| [2021-04-29 17:04:29,516 I 12846 12846] core_worker.cc:2064: Finished executing task dc62d70b3d5b5d07ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-44e8bc018ef786ba62589e8e00f2b66e6f2f2fb796b8960510d549c7_9810.log <== | |
| [2021-04-29 17:04:01,551 I 9810 11419] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,571 E 9810 9810] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,577 I 9810 11419] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,104 E 9810 9810] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:02,169 I 9810 9810] direct_task_transport.cc:47: Creating actor via GCS actor id = : 10c042788285c2ab39c9358401000000 | |
| [2021-04-29 17:04:02,186 I 9810 11419] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 10c042788285c2ab39c9358401000000, ip address: 10.134.60.23, port: 9115, worker_id: 7909c8e90f0861ab247a51534b426c315994f690045c7b26ea9f7c84, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,545 I 9810 11419] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 10c042788285c2ab39c9358401000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,391 I 9810 11419] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 10c042788285c2ab39c9358401000000, ip address: 10.134.60.23, port: 9115, worker_id: 7909c8e90f0861ab247a51534b426c315994f690045c7b26ea9f7c84, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,110 E 9810 9810] actor_manager.cc:93: Actor handle already exists 10c042788285c2ab39c9358401000000 | |
| [2021-04-29 17:04:29,466 I 9810 9810] core_worker.cc:2064: Finished executing task d7b5a74bfe49b8a7ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-4500349c533f661f8ea80550be59b688170bf289065f46224018f117_17362.log <== | |
| [2021-04-29 17:04:09,421 I 17362 19006] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,593 E 17362 17362] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,620 I 17362 19006] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,531 E 17362 17362] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,618 I 17362 17362] direct_task_transport.cc:47: Creating actor via GCS actor id = : 532adac42cecd78f663caa9f01000000 | |
| [2021-04-29 17:04:11,667 I 17362 19006] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 532adac42cecd78f663caa9f01000000, ip address: 10.134.112.222, port: 9188, worker_id: c3d5d1edc50346c6d5d48de79d2a1a5678dffe5772175dee9a8ecba7, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,383 I 17362 19006] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 532adac42cecd78f663caa9f01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:25,743 I 17362 19006] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 532adac42cecd78f663caa9f01000000, ip address: 10.134.112.222, port: 9188, worker_id: c3d5d1edc50346c6d5d48de79d2a1a5678dffe5772175dee9a8ecba7, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,491 E 17362 17362] actor_manager.cc:93: Actor handle already exists 532adac42cecd78f663caa9f01000000 | |
| [2021-04-29 17:04:29,846 I 17362 17362] core_worker.cc:2064: Finished executing task b70342e7eab9acc2ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-452c5dedb253763e6a1ca2eb1244abc3517799414ed22e8d7cff1544_7479.log <== | |
| [2021-04-29 17:03:59,378 I 7479 8518] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,392 E 7479 7479] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,398 I 7479 8518] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,737 E 7479 7479] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,815 I 7479 7479] direct_task_transport.cc:47: Creating actor via GCS actor id = : e8fe890ba99e18cf67c9952001000000 | |
| [2021-04-29 17:03:59,819 I 7479 8518] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: e8fe890ba99e18cf67c9952001000000, ip address: 10.134.60.23, port: 29553, worker_id: NIL_ID, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,725 I 7479 8518] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: e8fe890ba99e18cf67c9952001000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,167 I 7479 8518] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: e8fe890ba99e18cf67c9952001000000, ip address: 10.134.60.23, port: 9126, worker_id: c4cc1de89e9abde728b68fbbd682694a3684d3ca352a702fe3a4d28c, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,856 E 7479 7479] actor_manager.cc:93: Actor handle already exists e8fe890ba99e18cf67c9952001000000 | |
| [2021-04-29 17:04:29,384 I 7479 7479] core_worker.cc:2064: Finished executing task 5a31043ff0ef92edffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-45c80afaea885daafcac20fe2733a5cb0e9164b9ad57513d9159c3e3_5212.log <== | |
| [2021-04-29 17:04:02,516 E 5212 5212] actor_manager.cc:93: Actor handle already exists af6baf84aeb9a0ef7815f4c601000000 | |
| [2021-04-29 17:04:10,356 I 5212 5212] core_worker.cc:2064: Finished executing task f22fd774433658a8ffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:10,364 E 5212 5212] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,364 E 5212 5212] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,487 I 5212 5212] direct_task_transport.cc:47: Creating actor via GCS actor id = : a48ea0eb9099086e3bc4596f01000000 | |
| [2021-04-29 17:04:10,517 I 5212 5571] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: a48ea0eb9099086e3bc4596f01000000, ip address: 10.134.112.222, port: 9200, worker_id: e75c29dd1f63b347d533b05f6ed884bdd9e40b1ff4bd1b8bc512a0df, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,366 I 5212 5571] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: a48ea0eb9099086e3bc4596f01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:22,828 I 5212 5571] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: a48ea0eb9099086e3bc4596f01000000, ip address: 10.134.112.222, port: 9200, worker_id: e75c29dd1f63b347d533b05f6ed884bdd9e40b1ff4bd1b8bc512a0df, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:25,487 E 5212 5212] actor_manager.cc:93: Actor handle already exists a48ea0eb9099086e3bc4596f01000000 | |
| [2021-04-29 17:04:29,745 I 5212 5212] core_worker.cc:2064: Finished executing task 6aecc3a1877c298dffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-45df4b12131aae5688da501d7b88b1fcefcd95dd568cf654a6664320_15905.log <== | |
| [2021-04-29 17:04:08,004 I 15905 16878] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,202 E 15905 15905] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,249 I 15905 16878] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,963 E 15905 15905] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,059 I 15905 15905] direct_task_transport.cc:47: Creating actor via GCS actor id = : b78b83eb2ac3fd344b4a394201000000 | |
| [2021-04-29 17:04:11,103 I 15905 16878] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: b78b83eb2ac3fd344b4a394201000000, ip address: 10.134.60.23, port: 9190, worker_id: 408387cbb995f6ef62c90ec8f1fb20a19d601ff8172931cdf3514c1a, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,370 I 15905 16878] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: b78b83eb2ac3fd344b4a394201000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,041 I 15905 16878] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: b78b83eb2ac3fd344b4a394201000000, ip address: 10.134.60.23, port: 9190, worker_id: 408387cbb995f6ef62c90ec8f1fb20a19d601ff8172931cdf3514c1a, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:26,960 E 15905 15905] actor_manager.cc:93: Actor handle already exists b78b83eb2ac3fd344b4a394201000000 | |
| [2021-04-29 17:04:29,773 I 15905 15905] core_worker.cc:2064: Finished executing task a0b6159ff1150249ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-46185b76830fc05a65787c5b3302f89e81b7e628f54243cde9031a16_9662.log <== | |
| [2021-04-29 17:04:00,943 I 9662 9991] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:00,982 E 9662 9662] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,008 I 9662 9991] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,764 E 9662 9662] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:01,819 I 9662 9662] direct_task_transport.cc:47: Creating actor via GCS actor id = : c587fefd42cb82e11e4ba92f01000000 | |
| [2021-04-29 17:04:01,825 I 9662 9991] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: c587fefd42cb82e11e4ba92f01000000, ip address: 10.133.95.40, port: 9121, worker_id: 0e9075d3e4800556fd6aaee0f52890467438704beadb86ef6e80a3ef, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,967 I 9662 9991] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: c587fefd42cb82e11e4ba92f01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,390 I 9662 9991] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: c587fefd42cb82e11e4ba92f01000000, ip address: 10.133.95.40, port: 9121, worker_id: 0e9075d3e4800556fd6aaee0f52890467438704beadb86ef6e80a3ef, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,051 E 9662 9662] actor_manager.cc:93: Actor handle already exists c587fefd42cb82e11e4ba92f01000000 | |
| [2021-04-29 17:04:29,446 I 9662 9662] core_worker.cc:2064: Finished executing task fcc268f0d8156e6bffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-46afbf3e35245b01d5853b2bd924d5a70e24037494d4d645ce93c573_12905.log <== | |
| [2021-04-29 17:04:05,171 I 12905 14339] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,181 E 12905 12905] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,188 I 12905 14339] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:07,363 E 12905 12905] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:07,459 I 12905 12905] direct_task_transport.cc:47: Creating actor via GCS actor id = : 7156bb8e3a7c5f5363a3357c01000000 | |
| [2021-04-29 17:04:07,480 I 12905 14339] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 7156bb8e3a7c5f5363a3357c01000000, ip address: 10.134.61.130, port: 9240, worker_id: c2cb13d8ea15a0719ec3dfee4789e79535b8f1f8467e8e93668387ce, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:16,988 I 12905 14339] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 7156bb8e3a7c5f5363a3357c01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,397 I 12905 14339] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 7156bb8e3a7c5f5363a3357c01000000, ip address: 10.134.61.130, port: 9240, worker_id: c2cb13d8ea15a0719ec3dfee4789e79535b8f1f8467e8e93668387ce, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,348 E 12905 12905] actor_manager.cc:93: Actor handle already exists 7156bb8e3a7c5f5363a3357c01000000 | |
| [2021-04-29 17:04:29,505 I 12905 12905] core_worker.cc:2064: Finished executing task 4afa99d131a2ea73ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-46b353cced130ec47c7ca75582c9662e07b69dc15369be6eb6b84209_5219.log <== | |
| [2021-04-29 17:04:02,469 E 5219 5219] actor_manager.cc:93: Actor handle already exists fa6f6b7c8ca20b45b2bde90e01000000 | |
| [2021-04-29 17:04:10,294 I 5219 5219] core_worker.cc:2064: Finished executing task b5cc5396f222de8cffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:10,316 E 5219 5219] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,316 E 5219 5219] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,412 I 5219 5219] direct_task_transport.cc:47: Creating actor via GCS actor id = : d7d88f3c4bdb48d32b29772101000000 | |
| [2021-04-29 17:04:10,447 I 5219 5936] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: d7d88f3c4bdb48d32b29772101000000, ip address: 10.134.60.23, port: 9203, worker_id: 36c2a4d79cefe673a21300a7f5c2f23e54592733be59449e9fa2d464, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,366 I 5219 5936] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: d7d88f3c4bdb48d32b29772101000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:22,783 I 5219 5936] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: d7d88f3c4bdb48d32b29772101000000, ip address: 10.134.60.23, port: 9203, worker_id: 36c2a4d79cefe673a21300a7f5c2f23e54592733be59449e9fa2d464, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:25,474 E 5219 5219] actor_manager.cc:93: Actor handle already exists d7d88f3c4bdb48d32b29772101000000 | |
| [2021-04-29 17:04:29,741 I 5219 5219] core_worker.cc:2064: Finished executing task 489716a0400786d4ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-46eb0426295ec572057c5d98ea6f6d36bde5908bb03bc3115cb2714a_5309.log <== | |
| [2021-04-29 17:03:56,683 E 5309 5309] actor_manager.cc:93: Actor handle already exists aa1e287d5c3eeb531ef24cf401000000 | |
| [2021-04-29 17:03:57,994 I 5309 5309] core_worker.cc:2064: Finished executing task 7c622e164f6c5b65ffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:03:59,416 E 5309 5309] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,416 E 5309 5309] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,466 I 5309 5309] direct_task_transport.cc:47: Creating actor via GCS actor id = : cc0405f9ed19c1eb7927c1dd01000000 | |
| [2021-04-29 17:03:59,472 I 5309 6392] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: cc0405f9ed19c1eb7927c1dd01000000, ip address: 10.134.60.23, port: 9133, worker_id: e61874da9a6e055ab704ce7cf7a6d7596a69387a03a387d45e092a1f, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,316 I 5309 6392] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: cc0405f9ed19c1eb7927c1dd01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,562 I 5309 6392] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: cc0405f9ed19c1eb7927c1dd01000000, ip address: 10.134.60.23, port: 9133, worker_id: e61874da9a6e055ab704ce7cf7a6d7596a69387a03a387d45e092a1f, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:11,136 E 5309 5309] actor_manager.cc:93: Actor handle already exists cc0405f9ed19c1eb7927c1dd01000000 | |
| [2021-04-29 17:04:16,316 I 5309 5309] core_worker.cc:2064: Finished executing task 62183263b14990b8ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-47481a4c785378bf6ea326196a0435c41c68701291c443a69af85e18_9867.log <== | |
| [2021-04-29 17:04:02,049 I 9867 11616] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,072 E 9867 9867] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:02,080 I 9867 11616] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,816 E 9867 9867] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:04,931 I 9867 9867] direct_task_transport.cc:47: Creating actor via GCS actor id = : a4a2f8de756525172128968701000000 | |
| [2021-04-29 17:04:04,938 I 9867 11616] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: a4a2f8de756525172128968701000000, ip address: 10.133.95.40, port: 9108, worker_id: dad1b8f21068dacc0175b8b75ec11056293827caf7aed3d5b6c49502, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,194 I 9867 11616] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: a4a2f8de756525172128968701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,392 I 9867 11616] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: a4a2f8de756525172128968701000000, ip address: 10.133.95.40, port: 9108, worker_id: dad1b8f21068dacc0175b8b75ec11056293827caf7aed3d5b6c49502, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,769 E 9867 9867] actor_manager.cc:93: Actor handle already exists a4a2f8de756525172128968701000000 | |
| [2021-04-29 17:04:29,549 I 9867 9867] core_worker.cc:2064: Finished executing task c2a3ee1c2e711b0affffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-4760c2cbae693ab8da9003bf83a7c76e2be1e32ad229d60fb4bbab1a_9667.log <== | |
| [2021-04-29 17:04:01,206 I 9667 10349] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,233 E 9667 9667] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,241 I 9667 10349] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,903 E 9667 9667] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:01,964 I 9667 9667] direct_task_transport.cc:47: Creating actor via GCS actor id = : d025a4fff0b29dfede42b65701000000 | |
| [2021-04-29 17:04:01,974 I 9667 10349] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: d025a4fff0b29dfede42b65701000000, ip address: 10.134.61.130, port: 9107, worker_id: ca82b8dbb26db8a2818b09ce23aa16a049eb732a0286e6fdec5df4c1, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,526 I 9667 10349] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: d025a4fff0b29dfede42b65701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,390 I 9667 10349] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: d025a4fff0b29dfede42b65701000000, ip address: 10.134.61.130, port: 9107, worker_id: ca82b8dbb26db8a2818b09ce23aa16a049eb732a0286e6fdec5df4c1, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,754 E 9667 9667] actor_manager.cc:93: Actor handle already exists d025a4fff0b29dfede42b65701000000 | |
| [2021-04-29 17:04:29,550 I 9667 9667] core_worker.cc:2064: Finished executing task 01b26d3e85169a47ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-479d7e557123ed51221c7103f146087d79e5123c8faa1d0fb2ffb571_9705.log <== | |
| [2021-04-29 17:04:01,142 I 9705 10335] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,177 E 9705 9705] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,190 I 9705 10335] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,773 E 9705 9705] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:01,857 I 9705 9705] direct_task_transport.cc:47: Creating actor via GCS actor id = : d3546862f8752816f0f5fcdb01000000 | |
| [2021-04-29 17:04:01,860 I 9705 10335] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: d3546862f8752816f0f5fcdb01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,104 I 9705 10335] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: d3546862f8752816f0f5fcdb01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,390 I 9705 10335] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: d3546862f8752816f0f5fcdb01000000, ip address: 10.134.60.23, port: 9119, worker_id: 68b80d200ec914a12cbcf52dc9179668e07ce82c403d8b39fd62d479, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,060 E 9705 9705] actor_manager.cc:93: Actor handle already exists d3546862f8752816f0f5fcdb01000000 | |
| [2021-04-29 17:04:29,451 I 9705 9705] core_worker.cc:2064: Finished executing task de07d597f2b458e1ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-47d81c168cf7e963c05f39afd2d3df69e25dee588e02fe8a93b21574_5348.log <== | |
| [2021-04-29 17:03:46,416 I 5348 5348] core_worker.cc:2064: Finished executing task e276d7cb301c427e2a99e75ddd642d28ac877fa501000000, status=OK | |
| [2021-04-29 17:04:47,755 I 5348 5348] core_worker.cc:2064: Finished executing task c283a4645ab857d62a99e75ddd642d28ac877fa501000000, status=OK | |
| [2021-04-29 17:04:48,959 I 5348 5348] core_worker.cc:2064: Finished executing task 579f61ebd4328d972a99e75ddd642d28ac877fa501000000, status=OK | |
| [2021-04-29 17:04:49,721 I 5348 5348] core_worker.cc:2064: Finished executing task 2c698130f60cd2072a99e75ddd642d28ac877fa501000000, status=OK | |
| [2021-04-29 17:04:51,196 I 5348 5348] core_worker.cc:2064: Finished executing task 793de61ac76f790e2a99e75ddd642d28ac877fa501000000, status=OK | |
| [2021-04-29 17:04:52,341 I 5348 5348] core_worker.cc:2064: Finished executing task 72c8b9b30995c7bd2a99e75ddd642d28ac877fa501000000, status=OK | |
| [2021-04-29 17:04:53,171 I 5348 5348] core_worker.cc:2064: Finished executing task 9b485a4aa677f3662a99e75ddd642d28ac877fa501000000, status=OK | |
| [2021-04-29 17:04:54,121 I 5348 5348] core_worker.cc:2064: Finished executing task 035ea0e5f42600a92a99e75ddd642d28ac877fa501000000, status=OK | |
| [2021-04-29 17:04:56,383 I 5348 5348] core_worker.cc:2064: Finished executing task 75d97068f0df367a2a99e75ddd642d28ac877fa501000000, status=OK | |
| [2021-04-29 17:04:57,719 I 5348 5348] core_worker.cc:2064: Finished executing task 816ca24bd87e1d812a99e75ddd642d28ac877fa501000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-47e6d122755caf283f6c34d58bbc626b82d3afac1e51697cb05ec14e_14482.log <== | |
| [2021-04-29 17:04:05,967 I 14482 15808] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,010 E 14482 14482] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:06,025 I 14482 15808] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,784 E 14482 14482] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:08,885 I 14482 14482] direct_task_transport.cc:47: Creating actor via GCS actor id = : b165f4a5c6cfc12801ae23de01000000 | |
| [2021-04-29 17:04:08,922 I 14482 15808] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: b165f4a5c6cfc12801ae23de01000000, ip address: 10.134.70.227, port: 9223, worker_id: e1389999537aae9e376a1f3ac776608b66ed6796f55f2a695461e9c1, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:19,691 I 14482 15808] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: b165f4a5c6cfc12801ae23de01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:22,911 I 14482 15808] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: b165f4a5c6cfc12801ae23de01000000, ip address: 10.134.70.227, port: 9223, worker_id: e1389999537aae9e376a1f3ac776608b66ed6796f55f2a695461e9c1, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:25,731 E 14482 14482] actor_manager.cc:93: Actor handle already exists b165f4a5c6cfc12801ae23de01000000 | |
| [2021-04-29 17:04:29,762 I 14482 14482] core_worker.cc:2064: Finished executing task d0e38cfd4933ae51ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-48048f758adb84bde3046f5f6fbaa602439bdc0cf849bff6028dbba4_12083.log <== | |
| [2021-04-29 17:04:03,386 I 12083 12498] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,412 E 12083 12083] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:03,424 I 12083 12498] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,868 E 12083 12083] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,943 I 12083 12083] direct_task_transport.cc:47: Creating actor via GCS actor id = : 0c5b7879df169b35de1f05d501000000 | |
| [2021-04-29 17:04:05,968 I 12083 12498] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 0c5b7879df169b35de1f05d501000000, ip address: 10.134.142.44, port: 9154, worker_id: 1f14f392b824550faef14199ad05451a5ca51b2eb00bb3ea1ad092ff, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,405 I 12083 12498] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 0c5b7879df169b35de1f05d501000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,396 I 12083 12498] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 0c5b7879df169b35de1f05d501000000, ip address: 10.134.142.44, port: 9154, worker_id: 1f14f392b824550faef14199ad05451a5ca51b2eb00bb3ea1ad092ff, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,305 E 12083 12083] actor_manager.cc:93: Actor handle already exists 0c5b7879df169b35de1f05d501000000 | |
| [2021-04-29 17:04:29,504 I 12083 12083] core_worker.cc:2064: Finished executing task 1e71d6d695d764aeffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-481f8b7c2f360f753e89f592a1df68d92e4f0a99545b714259b4b244_9754.log <== | |
| [2021-04-29 17:04:01,403 I 9754 10892] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,424 E 9754 9754] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,432 I 9754 10892] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,011 E 9754 9754] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:02,089 I 9754 9754] direct_task_transport.cc:47: Creating actor via GCS actor id = : 6dff7377603da43b7e3560d501000000 | |
| [2021-04-29 17:04:02,097 I 9754 10892] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 6dff7377603da43b7e3560d501000000, ip address: 10.134.60.23, port: 9114, worker_id: a64c4127406ca020589ddd3dfeb8b2fd1f7bb1c48289ebc1591810ee, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,544 I 9754 10892] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 6dff7377603da43b7e3560d501000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,391 I 9754 10892] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 6dff7377603da43b7e3560d501000000, ip address: 10.134.60.23, port: 9114, worker_id: a64c4127406ca020589ddd3dfeb8b2fd1f7bb1c48289ebc1591810ee, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,094 E 9754 9754] actor_manager.cc:93: Actor handle already exists 6dff7377603da43b7e3560d501000000 | |
| [2021-04-29 17:04:29,458 I 9754 9754] core_worker.cc:2064: Finished executing task 73eab34638a94302ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-485a6a0376d62590bd41c003130cb9fa830ffdb3bb8ef3269ea436ee_12707.log <== | |
| [2021-04-29 17:04:04,559 I 12707 13253] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,641 E 12707 12707] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:04,671 I 12707 13253] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,518 E 12707 12707] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:06,686 I 12707 12707] direct_task_transport.cc:47: Creating actor via GCS actor id = : 402649cfac70e8816e9a843201000000 | |
| [2021-04-29 17:04:06,728 I 12707 13253] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 402649cfac70e8816e9a843201000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:13,517 I 12707 13253] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 402649cfac70e8816e9a843201000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,398 I 12707 13253] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 402649cfac70e8816e9a843201000000, ip address: 10.134.61.25, port: 9152, worker_id: ff924b8acc2b2b531bc846b4eb9e78137fee3edcb0bc2512c5b1b864, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,401 E 12707 12707] actor_manager.cc:93: Actor handle already exists 402649cfac70e8816e9a843201000000 | |
| [2021-04-29 17:04:29,520 I 12707 12707] core_worker.cc:2064: Finished executing task e1e78213a7dbed76ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-48859e1f3d8c1b5838ccb4dc5e681a362f8d3dd5247f171d62d23f85_16034.log <== | |
| [2021-04-29 17:04:08,665 I 16034 17878] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,719 E 16034 16034] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,742 I 16034 17878] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,161 E 16034 16034] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,253 I 16034 16034] direct_task_transport.cc:47: Creating actor via GCS actor id = : 8b06290f561690904c4b871701000000 | |
| [2021-04-29 17:04:11,303 I 16034 17878] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 8b06290f561690904c4b871701000000, ip address: 10.134.60.23, port: 9192, worker_id: d1d778c48a164cd1315a70c229f6297a7e459b303c3164f2eeb8e5a6, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,377 I 16034 17878] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 8b06290f561690904c4b871701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,928 I 16034 17878] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8b06290f561690904c4b871701000000, ip address: 10.134.60.23, port: 9192, worker_id: d1d778c48a164cd1315a70c229f6297a7e459b303c3164f2eeb8e5a6, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:27,654 E 16034 16034] actor_manager.cc:93: Actor handle already exists 8b06290f561690904c4b871701000000 | |
| [2021-04-29 17:04:29,819 I 16034 16034] core_worker.cc:2064: Finished executing task 94a91cd44b383b07ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-48cb78a33b4819b006c5c40952ab9b6ad9fd89aae28c87e9cbe3b6c8_18083.log <== | |
| [2021-04-29 17:04:10,144 I 18083 19915] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,219 E 18083 18083] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,241 I 18083 19915] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,603 E 18083 18083] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,704 I 18083 18083] direct_task_transport.cc:47: Creating actor via GCS actor id = : ddca2c8c5e1ff8792f2a2c0801000000 | |
| [2021-04-29 17:04:11,755 I 18083 19915] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: ddca2c8c5e1ff8792f2a2c0801000000, ip address: 10.134.142.44, port: 9201, worker_id: 46e8721e4bbb158c827ee995231be368cbbd050e84ad9c5bdb845f0b, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,386 I 18083 19915] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: ddca2c8c5e1ff8792f2a2c0801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:28,403 I 18083 19915] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: ddca2c8c5e1ff8792f2a2c0801000000, ip address: 10.134.142.44, port: 9201, worker_id: 46e8721e4bbb158c827ee995231be368cbbd050e84ad9c5bdb845f0b, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:29,341 E 18083 18083] actor_manager.cc:93: Actor handle already exists ddca2c8c5e1ff8792f2a2c0801000000 | |
| [2021-04-29 17:04:29,871 I 18083 18083] core_worker.cc:2064: Finished executing task 51d6b1811457657effffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-49227228332f8ee31e3bc2881965defe68117e9012d5bef456ad760a_16063.log <== | |
| [2021-04-29 17:04:08,719 I 16063 17903] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,774 E 16063 16063] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,798 I 16063 17903] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,191 E 16063 16063] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,301 I 16063 16063] direct_task_transport.cc:47: Creating actor via GCS actor id = : 3edb0173ad1a1b9d1926958101000000 | |
| [2021-04-29 17:04:11,327 I 16063 17903] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 3edb0173ad1a1b9d1926958101000000, ip address: 10.134.70.227, port: 9196, worker_id: 6901a28e7da5d5291f4e4f763a313c0381da16d31d6c9c3b11f30d31, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,378 I 16063 17903] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 3edb0173ad1a1b9d1926958101000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:27,366 I 16063 17903] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 3edb0173ad1a1b9d1926958101000000, ip address: 10.134.70.227, port: 9196, worker_id: 6901a28e7da5d5291f4e4f763a313c0381da16d31d6c9c3b11f30d31, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,850 E 16063 16063] actor_manager.cc:93: Actor handle already exists 3edb0173ad1a1b9d1926958101000000 | |
| [2021-04-29 17:04:29,857 I 16063 16063] core_worker.cc:2064: Finished executing task b65ed7dcdc0402c1ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-49bffc290f7a572841aeab120d232c59063730b3555942d44100c119_15892.log <== | |
| [2021-04-29 17:04:08,003 I 15892 16881] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,203 E 15892 15892] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,249 I 15892 16881] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,962 E 15892 15892] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,060 I 15892 15892] direct_task_transport.cc:47: Creating actor via GCS actor id = : 4eda37be679bc16922abd5f001000000 | |
| [2021-04-29 17:04:11,102 I 15892 16881] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 4eda37be679bc16922abd5f001000000, ip address: 10.134.142.16, port: 9195, worker_id: c287d79b9d24aef00088588d28d48fd2aeb0dbcb4addb9ec6af45622, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,371 I 15892 16881] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 4eda37be679bc16922abd5f001000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,008 I 15892 16881] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 4eda37be679bc16922abd5f001000000, ip address: 10.134.142.16, port: 9195, worker_id: c287d79b9d24aef00088588d28d48fd2aeb0dbcb4addb9ec6af45622, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:26,955 E 15892 15892] actor_manager.cc:93: Actor handle already exists 4eda37be679bc16922abd5f001000000 | |
| [2021-04-29 17:04:29,777 I 15892 15892] core_worker.cc:2064: Finished executing task 23d1f76e020a0df1ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-4a077eb31797d2413ba4754c854b730e8f1a0325cf35980cff6a4c4a_12147.log <== | |
| [2021-04-29 17:04:04,426 I 12147 13125] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,467 E 12147 12147] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:04,483 I 12147 13125] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,504 E 12147 12147] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:06,629 I 12147 12147] direct_task_transport.cc:47: Creating actor via GCS actor id = : c8470c8b6d5445d8f7b0f2bb01000000 | |
| [2021-04-29 17:04:06,674 I 12147 13125] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: c8470c8b6d5445d8f7b0f2bb01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:13,369 I 12147 13125] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: c8470c8b6d5445d8f7b0f2bb01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,397 I 12147 13125] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: c8470c8b6d5445d8f7b0f2bb01000000, ip address: 10.134.112.222, port: 9152, worker_id: c8b23903d428b5370cf4a7d3336db758b14d55c41f7bfc85400196f8, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,350 E 12147 12147] actor_manager.cc:93: Actor handle already exists c8470c8b6d5445d8f7b0f2bb01000000 | |
| [2021-04-29 17:04:29,513 I 12147 12147] core_worker.cc:2064: Finished executing task 14aa7c6c8894251bffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-4a09d35dd4525b22c4d2fc5e27762154c29222d22d619a4c8eb969ad_12910.log <== | |
| [2021-04-29 17:04:04,775 I 12910 14416] service_based_accessor.cc:579: Received notification for node id = ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, IsAlive = 1 | |
| [2021-04-29 17:04:04,775 I 12910 14416] service_based_accessor.cc:579: Received notification for node id = d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, IsAlive = 1 | |
| [2021-04-29 17:04:04,775 I 12910 14416] service_based_accessor.cc:579: Received notification for node id = b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, IsAlive = 1 | |
| [2021-04-29 17:04:04,775 I 12910 14416] service_based_accessor.cc:579: Received notification for node id = 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, IsAlive = 1 | |
| [2021-04-29 17:04:04,775 I 12910 14416] service_based_accessor.cc:579: Received notification for node id = 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, IsAlive = 1 | |
| [2021-04-29 17:04:05,981 I 12910 14416] core_worker.cc:717: Exit signal received, this process will exit after all outstanding tasks have finished, exit_type=INTENDED_EXIT | |
| [2021-04-29 17:04:05,981 I 12910 12910] raylet_client.cc:129: RayletClient::Disconnect, exit_type=INTENDED_EXIT, has creation_task_exception_pb_bytes=0 | |
| [2021-04-29 17:04:05,981 I 12910 12910] core_worker.cc:308: Removed worker 4a09d35dd4525b22c4d2fc5e27762154c29222d22d619a4c8eb969ad | |
| [2021-04-29 17:04:05,985 I 12910 12910] core_worker.cc:181: Destructing CoreWorkerProcess. pid: 12910 | |
| [2021-04-29 17:04:05,985 I 12910 12910] io_service_pool.cc:47: IOServicePool is stopped. | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-4a3a07bf437f406749c28c9b32559993bd7487f34b5aedef196772e0_12092.log <== | |
| [2021-04-29 17:04:03,386 I 12092 12497] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,412 E 12092 12092] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:03,424 I 12092 12497] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,870 E 12092 12092] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,972 I 12092 12092] direct_task_transport.cc:47: Creating actor via GCS actor id = : 15469a602aa4ce54c843916001000000 | |
| [2021-04-29 17:04:05,983 I 12092 12497] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 15469a602aa4ce54c843916001000000, ip address: 10.134.112.222, port: 9100, worker_id: d85c4bd0c15acebce870652aa72054522e6c896c682ba7acaf580c92, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,866 I 12092 12497] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 15469a602aa4ce54c843916001000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,394 I 12092 12497] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 15469a602aa4ce54c843916001000000, ip address: 10.134.112.222, port: 9100, worker_id: d85c4bd0c15acebce870652aa72054522e6c896c682ba7acaf580c92, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,198 E 12092 12092] actor_manager.cc:93: Actor handle already exists 15469a602aa4ce54c843916001000000 | |
| [2021-04-29 17:04:29,475 I 12092 12092] core_worker.cc:2064: Finished executing task f2e32283fea8908dffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-4abbcfca6a86f04b3095df02e84149687cb9e13c53bb7439d934dcc0_9927.log <== | |
| [2021-04-29 17:04:02,048 I 9927 11628] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,072 E 9927 9927] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:02,080 I 9927 11628] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,817 E 9927 9927] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:04,906 I 9927 9927] direct_task_transport.cc:47: Creating actor via GCS actor id = : 2a95733aab29a25463a1d2ae01000000 | |
| [2021-04-29 17:04:04,940 I 9927 11628] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 2a95733aab29a25463a1d2ae01000000, ip address: 10.134.60.23, port: 9111, worker_id: 2d6c5db3db79d72d91e613d71574eb99c3375ae99f3221841cbe1a80, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,186 I 9927 11628] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 2a95733aab29a25463a1d2ae01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,393 I 9927 11628] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 2a95733aab29a25463a1d2ae01000000, ip address: 10.134.60.23, port: 9111, worker_id: 2d6c5db3db79d72d91e613d71574eb99c3375ae99f3221841cbe1a80, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,738 E 9927 9927] actor_manager.cc:93: Actor handle already exists 2a95733aab29a25463a1d2ae01000000 | |
| [2021-04-29 17:04:29,547 I 9927 9927] core_worker.cc:2064: Finished executing task a606d25597095384ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-4b04b9114b942e1b6234f854728f960d144aaad11eea4658c7116fdc_9660.log <== | |
| [2021-04-29 17:04:01,175 I 9660 10288] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,208 E 9660 9660] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,216 I 9660 10288] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,775 E 9660 9660] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:01,844 I 9660 9660] direct_task_transport.cc:47: Creating actor via GCS actor id = : 60186f905bee0a61dd98550d01000000 | |
| [2021-04-29 17:04:01,862 I 9660 10288] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 60186f905bee0a61dd98550d01000000, ip address: 10.134.70.227, port: 9110, worker_id: dcf9e41cece0c1c8a2f40ab709c64299153322ae729ebdc8b2a5e04b, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,657 I 9660 10288] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 60186f905bee0a61dd98550d01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,391 I 9660 10288] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 60186f905bee0a61dd98550d01000000, ip address: 10.134.70.227, port: 9110, worker_id: dcf9e41cece0c1c8a2f40ab709c64299153322ae729ebdc8b2a5e04b, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,747 E 9660 9660] actor_manager.cc:93: Actor handle already exists 60186f905bee0a61dd98550d01000000 | |
| [2021-04-29 17:04:29,550 I 9660 9660] core_worker.cc:2064: Finished executing task 5cc96152a0720d49ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-4b3e6491958077ee77b3aea4f5ec97808728d7b9b5fd444e2060e78f_18035.log <== | |
| [2021-04-29 17:04:09,703 I 18035 19840] service_based_accessor.cc:579: Received notification for node id = ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, IsAlive = 1 | |
| [2021-04-29 17:04:09,703 I 18035 19840] service_based_accessor.cc:579: Received notification for node id = d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, IsAlive = 1 | |
| [2021-04-29 17:04:09,703 I 18035 19840] service_based_accessor.cc:579: Received notification for node id = b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, IsAlive = 1 | |
| [2021-04-29 17:04:09,703 I 18035 19840] service_based_accessor.cc:579: Received notification for node id = 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, IsAlive = 1 | |
| [2021-04-29 17:04:09,703 I 18035 19840] service_based_accessor.cc:579: Received notification for node id = 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, IsAlive = 1 | |
| [2021-04-29 17:04:10,733 I 18035 19840] core_worker.cc:717: Exit signal received, this process will exit after all outstanding tasks have finished, exit_type=INTENDED_EXIT | |
| [2021-04-29 17:04:10,733 I 18035 18035] raylet_client.cc:129: RayletClient::Disconnect, exit_type=INTENDED_EXIT, has creation_task_exception_pb_bytes=0 | |
| [2021-04-29 17:04:10,734 I 18035 18035] core_worker.cc:308: Removed worker 4b3e6491958077ee77b3aea4f5ec97808728d7b9b5fd444e2060e78f | |
| [2021-04-29 17:04:10,737 I 18035 18035] core_worker.cc:181: Destructing CoreWorkerProcess. pid: 18035 | |
| [2021-04-29 17:04:10,737 I 18035 18035] io_service_pool.cc:47: IOServicePool is stopped. | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-4b8d437ea471fbd379ed894874e7fa47bce25a5f76ec88386ba9ce4f_7569.log <== | |
| [2021-04-29 17:03:58,834 I 7569 9072] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,858 E 7569 7569] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,864 I 7569 9072] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,588 E 7569 7569] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,635 I 7569 7569] direct_task_transport.cc:47: Creating actor via GCS actor id = : 0edf35d29358545d024fb11601000000 | |
| [2021-04-29 17:03:59,644 I 7569 9072] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 0edf35d29358545d024fb11601000000, ip address: 10.134.70.227, port: 9121, worker_id: 4d412183ba5ba672d99fc701544ca7ef0e12109d23f7ea04b8bf0038, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,628 I 7569 9072] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 0edf35d29358545d024fb11601000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:17,695 I 7569 9072] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 0edf35d29358545d024fb11601000000, ip address: 10.134.70.227, port: 9121, worker_id: 4d412183ba5ba672d99fc701544ca7ef0e12109d23f7ea04b8bf0038, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:21,135 E 7569 7569] actor_manager.cc:93: Actor handle already exists 0edf35d29358545d024fb11601000000 | |
| [2021-04-29 17:04:28,435 I 7569 7569] core_worker.cc:2064: Finished executing task 5b8f175254b3e2f6ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-4bbdbb464aaf0fbb84dc3a4e15ff2f4eb319c55569749b1c7804c7dd_12958.log <== | |
| [2021-04-29 17:04:05,000 I 12958 14725] service_based_accessor.cc:579: Received notification for node id = ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, IsAlive = 1 | |
| [2021-04-29 17:04:05,000 I 12958 14725] service_based_accessor.cc:579: Received notification for node id = d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, IsAlive = 1 | |
| [2021-04-29 17:04:05,000 I 12958 14725] service_based_accessor.cc:579: Received notification for node id = b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, IsAlive = 1 | |
| [2021-04-29 17:04:05,000 I 12958 14725] service_based_accessor.cc:579: Received notification for node id = 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, IsAlive = 1 | |
| [2021-04-29 17:04:05,000 I 12958 14725] service_based_accessor.cc:579: Received notification for node id = 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, IsAlive = 1 | |
| [2021-04-29 17:04:05,982 I 12958 14725] core_worker.cc:717: Exit signal received, this process will exit after all outstanding tasks have finished, exit_type=INTENDED_EXIT | |
| [2021-04-29 17:04:05,982 I 12958 12958] raylet_client.cc:129: RayletClient::Disconnect, exit_type=INTENDED_EXIT, has creation_task_exception_pb_bytes=0 | |
| [2021-04-29 17:04:05,982 I 12958 12958] core_worker.cc:308: Removed worker 4bbdbb464aaf0fbb84dc3a4e15ff2f4eb319c55569749b1c7804c7dd | |
| [2021-04-29 17:04:05,985 I 12958 12958] core_worker.cc:181: Destructing CoreWorkerProcess. pid: 12958 | |
| [2021-04-29 17:04:05,985 I 12958 12958] io_service_pool.cc:47: IOServicePool is stopped. | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-4c34a50b1943887cc59a419d7555e8aacdf62f1fd846c5bcdb959b80_16957.log <== | |
| [2021-04-29 17:04:09,294 I 16957 18655] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,434 E 16957 16957] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,460 I 16957 18655] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,488 E 16957 16957] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,587 I 16957 16957] direct_task_transport.cc:47: Creating actor via GCS actor id = : b53d6466ee3240dd2e91cd2201000000 | |
| [2021-04-29 17:04:11,625 I 16957 18655] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: b53d6466ee3240dd2e91cd2201000000, ip address: 10.134.142.16, port: 9192, worker_id: dd1405d88d01263c61e3f7daafa2b9f480f1473407237a830d82c773, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,383 I 16957 18655] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: b53d6466ee3240dd2e91cd2201000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:25,567 I 16957 18655] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: b53d6466ee3240dd2e91cd2201000000, ip address: 10.134.142.16, port: 9192, worker_id: dd1405d88d01263c61e3f7daafa2b9f480f1473407237a830d82c773, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,449 E 16957 16957] actor_manager.cc:93: Actor handle already exists b53d6466ee3240dd2e91cd2201000000 | |
| [2021-04-29 17:04:29,838 I 16957 16957] core_worker.cc:2064: Finished executing task a6ea2bf21270b439ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-4ca23475ebfd5fb38072d8e513c0eab97b6ced49e304f9ef9f49d739_12917.log <== | |
| [2021-04-29 17:04:04,960 I 12917 14105] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,015 E 12917 12917] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,025 I 12917 14105] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,875 E 12917 12917] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:07,067 I 12917 12917] direct_task_transport.cc:47: Creating actor via GCS actor id = : 2062add8699767ecd7d13a2201000000 | |
| [2021-04-29 17:04:07,105 I 12917 14105] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 2062add8699767ecd7d13a2201000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:15,177 I 12917 14105] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 2062add8699767ecd7d13a2201000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,399 I 12917 14105] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 2062add8699767ecd7d13a2201000000, ip address: 10.133.95.40, port: 9155, worker_id: bde1acc13804a0e95b433378504808d4351ce3b425d1733657cb9c66, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,438 E 12917 12917] actor_manager.cc:93: Actor handle already exists 2062add8699767ecd7d13a2201000000 | |
| [2021-04-29 17:04:29,521 I 12917 12917] core_worker.cc:2064: Finished executing task e526c0c659290e47ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-4cb194e473ea760950f37f79f332707cacd932b3656c20adf4730cc5_15873.log <== | |
| [2021-04-29 17:04:07,728 I 15873 16504] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:07,789 E 15873 15873] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:07,848 I 15873 16504] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,495 E 15873 15873] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:10,578 I 15873 15873] direct_task_transport.cc:47: Creating actor via GCS actor id = : 95c1dc725735a45ec2b63c0601000000 | |
| [2021-04-29 17:04:10,591 I 15873 16504] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 95c1dc725735a45ec2b63c0601000000, ip address: 10.134.142.16, port: 9209, worker_id: 1dccf1ccb99be3b0cfc43c6d32b6f4b8ab5c7e0a02ec7ffb29328127, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,367 I 15873 16504] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 95c1dc725735a45ec2b63c0601000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:22,948 I 15873 16504] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 95c1dc725735a45ec2b63c0601000000, ip address: 10.134.142.16, port: 9209, worker_id: 1dccf1ccb99be3b0cfc43c6d32b6f4b8ab5c7e0a02ec7ffb29328127, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:25,741 E 15873 15873] actor_manager.cc:93: Actor handle already exists 95c1dc725735a45ec2b63c0601000000 | |
| [2021-04-29 17:04:29,765 I 15873 15873] core_worker.cc:2064: Finished executing task 0984f130f3ec27c6ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-4cdbf14f62290d0e9348b1966519e22e13b35fab59bf51964abf1245_12826.log <== | |
| [2021-04-29 17:04:04,789 I 12826 13697] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,850 E 12826 12826] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:04,876 I 12826 13697] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,644 E 12826 12826] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:06,851 I 12826 12826] direct_task_transport.cc:47: Creating actor via GCS actor id = : a436684d2c07363495a5ac2b01000000 | |
| [2021-04-29 17:04:06,866 I 12826 13697] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: a436684d2c07363495a5ac2b01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:13,881 I 12826 13697] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: a436684d2c07363495a5ac2b01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,398 I 12826 13697] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: a436684d2c07363495a5ac2b01000000, ip address: 10.134.61.25, port: 9154, worker_id: e1902ed3b67d5a855cc0ffa41037c80b5bfb38dc35a11fee42bcf784, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,405 E 12826 12826] actor_manager.cc:93: Actor handle already exists a436684d2c07363495a5ac2b01000000 | |
| [2021-04-29 17:04:29,522 I 12826 12826] core_worker.cc:2064: Finished executing task e56623b0a0bb55cfffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-4d2bb283c5f88c6f2fb0438cafce0298dc9fdca24ff55d5f8ae5ef87_9682.log <== | |
| [2021-04-29 17:04:01,230 I 9682 10530] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,255 E 9682 9682] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,264 I 9682 10530] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,905 E 9682 9682] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:01,956 I 9682 9682] direct_task_transport.cc:47: Creating actor via GCS actor id = : 5a68665327438a550f190cb501000000 | |
| [2021-04-29 17:04:01,975 I 9682 10530] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 5a68665327438a550f190cb501000000, ip address: 10.134.142.16, port: 9118, worker_id: 1b7efa0fb13dd7d1864d9631a9c3e10bb8757f6fd66ffe98a090f452, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,458 I 9682 10530] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 5a68665327438a550f190cb501000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,390 I 9682 10530] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 5a68665327438a550f190cb501000000, ip address: 10.134.142.16, port: 9118, worker_id: 1b7efa0fb13dd7d1864d9631a9c3e10bb8757f6fd66ffe98a090f452, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,049 E 9682 9682] actor_manager.cc:93: Actor handle already exists 5a68665327438a550f190cb501000000 | |
| [2021-04-29 17:04:29,452 I 9682 9682] core_worker.cc:2064: Finished executing task c814a1570a3836f0ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-4d3098160a584dc450ab3cb562fc917981708264aa7092bb32906d51_9937.log <== | |
| [2021-04-29 17:04:01,669 I 9937 11724] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,688 E 9937 9937] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,694 I 9937 11724] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,194 E 9937 9937] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:02,249 I 9937 9937] direct_task_transport.cc:47: Creating actor via GCS actor id = : 0bd9c17a61afd19fedc6a17801000000 | |
| [2021-04-29 17:04:02,257 I 9937 11724] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 0bd9c17a61afd19fedc6a17801000000, ip address: 10.134.112.222, port: 9116, worker_id: 9bd204bc1243f9d745b5b474673a8475bf1c60b8c255eec3fa0855d7, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,546 I 9937 11724] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 0bd9c17a61afd19fedc6a17801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,391 I 9937 11724] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 0bd9c17a61afd19fedc6a17801000000, ip address: 10.134.112.222, port: 9116, worker_id: 9bd204bc1243f9d745b5b474673a8475bf1c60b8c255eec3fa0855d7, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,112 E 9937 9937] actor_manager.cc:93: Actor handle already exists 0bd9c17a61afd19fedc6a17801000000 | |
| [2021-04-29 17:04:29,459 I 9937 9937] core_worker.cc:2064: Finished executing task 78271dd337963a65ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-4d454986667f029e9ad113674ac3192881b13a2d90cde0870a1e9cb8_10435.log <== | |
| [2021-04-29 17:04:01,759 I 10435 11894] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,760 E 10435 10435] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,769 I 10435 11894] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,036 E 10435 10435] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:04,133 I 10435 10435] direct_task_transport.cc:47: Creating actor via GCS actor id = : f012888da83472422b42cab701000000 | |
| [2021-04-29 17:04:04,146 I 10435 11894] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: f012888da83472422b42cab701000000, ip address: 10.134.112.222, port: 9112, worker_id: a99b3d520af2e870be86c7a7a8cdc4a861a32ede6daa57325caf4c9f, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,241 I 10435 11894] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: f012888da83472422b42cab701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,392 I 10435 11894] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: f012888da83472422b42cab701000000, ip address: 10.134.112.222, port: 9112, worker_id: a99b3d520af2e870be86c7a7a8cdc4a861a32ede6daa57325caf4c9f, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,172 E 10435 10435] actor_manager.cc:93: Actor handle already exists f012888da83472422b42cab701000000 | |
| [2021-04-29 17:04:29,473 I 10435 10435] core_worker.cc:2064: Finished executing task c4db464dc1628bfaffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-4d70b0d0d4d1994bcd2ee51f69a2fbba237ee752a6450c65b33add2c_5238.log <== | |
| [2021-04-29 17:03:59,587 E 5238 5238] actor_manager.cc:93: Actor handle already exists 3cebe197da5ff70d26d6b6f901000000 | |
| [2021-04-29 17:04:01,765 I 5238 5238] core_worker.cc:2064: Finished executing task a29503c395235072ffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:01,808 E 5238 5238] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,809 E 5238 5238] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,875 I 5238 5238] direct_task_transport.cc:47: Creating actor via GCS actor id = : 21f9c171defc02bcfef1194401000000 | |
| [2021-04-29 17:04:01,885 I 5238 6130] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 21f9c171defc02bcfef1194401000000, ip address: 10.134.60.23, port: 9118, worker_id: aa2a96440a61bb1c82de06876ae1111990840fc32df0159f5ce6d5c8, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,307 I 5238 6130] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 21f9c171defc02bcfef1194401000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,390 I 5238 6130] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 21f9c171defc02bcfef1194401000000, ip address: 10.134.60.23, port: 9118, worker_id: aa2a96440a61bb1c82de06876ae1111990840fc32df0159f5ce6d5c8, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,079 E 5238 5238] actor_manager.cc:93: Actor handle already exists 21f9c171defc02bcfef1194401000000 | |
| [2021-04-29 17:04:29,454 I 5238 5238] core_worker.cc:2064: Finished executing task dbb86ed72da47ad3ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-4daf663b3229aac34c550d66c44ed4987712862467420e10291d8990_12086.log <== | |
| [2021-04-29 17:04:03,359 I 12086 12404] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,388 E 12086 12086] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:03,397 I 12086 12404] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,865 E 12086 12086] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,964 I 12086 12086] direct_task_transport.cc:47: Creating actor via GCS actor id = : 7876474353f3b8e3060c228301000000 | |
| [2021-04-29 17:04:05,966 I 12086 12404] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 7876474353f3b8e3060c228301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,339 I 12086 12404] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 7876474353f3b8e3060c228301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,396 I 12086 12404] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 7876474353f3b8e3060c228301000000, ip address: 10.134.142.44, port: 9151, worker_id: 05ac3b0745f6de60396f022f2e8598b9642b296ad60a8380938ad87a, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,283 E 12086 12086] actor_manager.cc:93: Actor handle already exists 7876474353f3b8e3060c228301000000 | |
| [2021-04-29 17:04:29,488 I 12086 12086] core_worker.cc:2064: Finished executing task 85932742529cceb7ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-4dc452ab8a8e578e96b538ce1e13402c3fd1359dfd3d626f0994b5db_15903.log <== | |
| [2021-04-29 17:04:07,910 I 15903 16879] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,151 E 15903 15903] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,202 I 15903 16879] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,926 E 15903 15903] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,027 I 15903 15903] direct_task_transport.cc:47: Creating actor via GCS actor id = : 6c398591d7e40021a39f8e3101000000 | |
| [2021-04-29 17:04:11,064 I 15903 16879] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 6c398591d7e40021a39f8e3101000000, ip address: 10.134.142.44, port: 9220, worker_id: fb5e98eb6ec8e3c499457a2c01b4f2fee57111c9295a2d943f65d098, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,369 I 15903 16879] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 6c398591d7e40021a39f8e3101000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,958 I 15903 16879] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 6c398591d7e40021a39f8e3101000000, ip address: 10.134.142.44, port: 9220, worker_id: fb5e98eb6ec8e3c499457a2c01b4f2fee57111c9295a2d943f65d098, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:27,659 E 15903 15903] actor_manager.cc:93: Actor handle already exists 6c398591d7e40021a39f8e3101000000 | |
| [2021-04-29 17:04:29,811 I 15903 15903] core_worker.cc:2064: Finished executing task b1cb6d4c4f4a5c6dffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-4e14a8b060958a188f76b043d9e8335f90b107fb4e1e19459931eb47_12860.log <== | |
| [2021-04-29 17:04:04,960 I 12860 14049] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,016 E 12860 12860] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,044 I 12860 14049] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,877 E 12860 12860] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:07,068 I 12860 12860] direct_task_transport.cc:47: Creating actor via GCS actor id = : 5acbca2c4adf03b7c52be51501000000 | |
| [2021-04-29 17:04:07,105 I 12860 14049] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 5acbca2c4adf03b7c52be51501000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:15,387 I 12860 14049] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 5acbca2c4adf03b7c52be51501000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,396 I 12860 14049] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 5acbca2c4adf03b7c52be51501000000, ip address: 10.134.61.130, port: 9235, worker_id: a8ad676d5543b58bb8e5edfea51eb93ad6f7c97c38f6be2b19eba0bc, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,310 E 12860 12860] actor_manager.cc:93: Actor handle already exists 5acbca2c4adf03b7c52be51501000000 | |
| [2021-04-29 17:04:29,489 I 12860 12860] core_worker.cc:2064: Finished executing task 99f2a31beec76442ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-4e741016a0d67641fd1b959fa92e237977500dba4c5d2d7037ce8f81_3999.log <== | |
| [2021-04-29 17:03:44,986 I 3999 3999] core_worker.cc:2064: Finished executing task 433c4365f8880161fbd81f73e48cc88fe3cddd6101000000, status=OK | |
| [2021-04-29 17:04:45,253 I 3999 3999] core_worker.cc:2064: Finished executing task 74ef82080894c2ecfbd81f73e48cc88fe3cddd6101000000, status=OK | |
| [2021-04-29 17:04:49,653 I 3999 3999] core_worker.cc:2064: Finished executing task 2ef80bbb4cb60425fbd81f73e48cc88fe3cddd6101000000, status=OK | |
| [2021-04-29 17:04:51,166 I 3999 3999] core_worker.cc:2064: Finished executing task 4a1f901997b886a3fbd81f73e48cc88fe3cddd6101000000, status=OK | |
| [2021-04-29 17:04:51,647 I 3999 3999] core_worker.cc:2064: Finished executing task a834407365f82c93fbd81f73e48cc88fe3cddd6101000000, status=OK | |
| [2021-04-29 17:04:52,817 I 3999 3999] core_worker.cc:2064: Finished executing task 2bb75d8a1ad68aa0fbd81f73e48cc88fe3cddd6101000000, status=OK | |
| [2021-04-29 17:04:54,765 I 3999 3999] core_worker.cc:2064: Finished executing task 881ae5c3bd33a296fbd81f73e48cc88fe3cddd6101000000, status=OK | |
| [2021-04-29 17:04:56,288 I 3999 3999] core_worker.cc:2064: Finished executing task 4a47156c680e69aefbd81f73e48cc88fe3cddd6101000000, status=OK | |
| [2021-04-29 17:04:57,642 I 3999 3999] core_worker.cc:2064: Finished executing task 81b90e5c3f566919fbd81f73e48cc88fe3cddd6101000000, status=OK | |
| [2021-04-29 17:04:57,682 I 3999 3999] core_worker.cc:2064: Finished executing task f75f7c83af2e8708fbd81f73e48cc88fe3cddd6101000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-4f5e4db19bebf85c3024dad887611ffc6dd542cbf8be7aa7cfd44e4d_18033.log <== | |
| [2021-04-29 17:04:09,703 I 18033 19841] service_based_accessor.cc:579: Received notification for node id = ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, IsAlive = 1 | |
| [2021-04-29 17:04:09,703 I 18033 19841] service_based_accessor.cc:579: Received notification for node id = d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, IsAlive = 1 | |
| [2021-04-29 17:04:09,703 I 18033 19841] service_based_accessor.cc:579: Received notification for node id = b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, IsAlive = 1 | |
| [2021-04-29 17:04:09,703 I 18033 19841] service_based_accessor.cc:579: Received notification for node id = 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, IsAlive = 1 | |
| [2021-04-29 17:04:09,703 I 18033 19841] service_based_accessor.cc:579: Received notification for node id = 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, IsAlive = 1 | |
| [2021-04-29 17:04:10,733 I 18033 19841] core_worker.cc:717: Exit signal received, this process will exit after all outstanding tasks have finished, exit_type=INTENDED_EXIT | |
| [2021-04-29 17:04:10,733 I 18033 18033] raylet_client.cc:129: RayletClient::Disconnect, exit_type=INTENDED_EXIT, has creation_task_exception_pb_bytes=0 | |
| [2021-04-29 17:04:10,734 I 18033 18033] core_worker.cc:308: Removed worker 4f5e4db19bebf85c3024dad887611ffc6dd542cbf8be7aa7cfd44e4d | |
| [2021-04-29 17:04:10,737 I 18033 18033] core_worker.cc:181: Destructing CoreWorkerProcess. pid: 18033 | |
| [2021-04-29 17:04:10,737 I 18033 18033] io_service_pool.cc:47: IOServicePool is stopped. | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-4f61385c50e4e64412a0806ed2c235ed0a64eb3bf98c32c56d380c3f_15922.log <== | |
| [2021-04-29 17:04:08,003 I 15922 16976] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,202 E 15922 15922] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,249 I 15922 16976] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,962 E 15922 15922] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,058 I 15922 15922] direct_task_transport.cc:47: Creating actor via GCS actor id = : 0856c4002ade7529b14e2d4301000000 | |
| [2021-04-29 17:04:11,103 I 15922 16976] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 0856c4002ade7529b14e2d4301000000, ip address: 10.134.142.44, port: 9221, worker_id: 232ba2d242496b8fa7fc3ef0b2de5cdc32627d00cd2eb11cfcd4f7e8, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,369 I 15922 16976] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 0856c4002ade7529b14e2d4301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:25,021 I 15922 16976] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 0856c4002ade7529b14e2d4301000000, ip address: 10.134.142.44, port: 9221, worker_id: 232ba2d242496b8fa7fc3ef0b2de5cdc32627d00cd2eb11cfcd4f7e8, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:27,703 E 15922 15922] actor_manager.cc:93: Actor handle already exists 0856c4002ade7529b14e2d4301000000 | |
| [2021-04-29 17:04:29,821 I 15922 15922] core_worker.cc:2064: Finished executing task 9ba36e4b9b49da16ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-4fa7ddcf83fcc7d0b77ba4d67767ec476efef56c9a1af547c88e1969_15953.log <== | |
| [2021-04-29 17:04:08,201 I 15953 17389] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,416 E 15953 15953] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,444 I 15953 17389] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,029 E 15953 15953] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,133 I 15953 15953] direct_task_transport.cc:47: Creating actor via GCS actor id = : 9b123621bdbdfb331f4a883501000000 | |
| [2021-04-29 17:04:11,171 I 15953 17389] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 9b123621bdbdfb331f4a883501000000, ip address: 10.134.61.130, port: 9214, worker_id: 0de00dbc27db43d2488193139a0421db7b40d0950d06810d4b30fb5c, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,373 I 15953 17389] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 9b123621bdbdfb331f4a883501000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:25,508 I 15953 17389] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 9b123621bdbdfb331f4a883501000000, ip address: 10.134.61.130, port: 9214, worker_id: 0de00dbc27db43d2488193139a0421db7b40d0950d06810d4b30fb5c, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,442 E 15953 15953] actor_manager.cc:93: Actor handle already exists 9b123621bdbdfb331f4a883501000000 | |
| [2021-04-29 17:04:29,835 I 15953 15953] core_worker.cc:2064: Finished executing task 6d5977d315d97c19ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-4fc18bea1efbb18d3432b01fd6d42f88cb05d09012b12908b267a3ee_16045.log <== | |
| [2021-04-29 17:04:08,626 I 16045 17846] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,691 E 16045 16045] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,717 I 16045 17846] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,160 E 16045 16045] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,253 I 16045 16045] direct_task_transport.cc:47: Creating actor via GCS actor id = : 20f58f64e89389fadf8c447f01000000 | |
| [2021-04-29 17:04:11,303 I 16045 17846] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 20f58f64e89389fadf8c447f01000000, ip address: 10.134.142.16, port: 9200, worker_id: ea10856d24878f37859b8bc6437b62aea0cde45e5ece4c294b7dc908, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,377 I 16045 17846] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 20f58f64e89389fadf8c447f01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,912 I 16045 17846] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 20f58f64e89389fadf8c447f01000000, ip address: 10.134.142.16, port: 9200, worker_id: ea10856d24878f37859b8bc6437b62aea0cde45e5ece4c294b7dc908, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:27,651 E 16045 16045] actor_manager.cc:93: Actor handle already exists 20f58f64e89389fadf8c447f01000000 | |
| [2021-04-29 17:04:29,813 I 16045 16045] core_worker.cc:2064: Finished executing task 1ec052c792bda690ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-5099dea1c9d150cdb2b039148a0ebef0175ac279526c7b05f8551a90_5366.log <== | |
| [2021-04-29 17:03:46,413 I 5366 5366] core_worker.cc:2064: Finished executing task e510a424110f368ba82936f57c63efdc52c5def701000000, status=OK | |
| [2021-04-29 17:04:37,898 I 5366 5366] core_worker.cc:2064: Finished executing task cd5a413226374c7aa82936f57c63efdc52c5def701000000, status=OK | |
| [2021-04-29 17:04:47,999 I 5366 5366] core_worker.cc:2064: Finished executing task 19169d5a0a46065ca82936f57c63efdc52c5def701000000, status=OK | |
| [2021-04-29 17:04:49,537 I 5366 5366] core_worker.cc:2064: Finished executing task a9227f98caec8851a82936f57c63efdc52c5def701000000, status=OK | |
| [2021-04-29 17:04:51,819 I 5366 5366] core_worker.cc:2064: Finished executing task 07a7308f5e999c47a82936f57c63efdc52c5def701000000, status=OK | |
| [2021-04-29 17:04:52,351 I 5366 5366] core_worker.cc:2064: Finished executing task 0972cd2f2ede44f4a82936f57c63efdc52c5def701000000, status=OK | |
| [2021-04-29 17:04:52,966 I 5366 5366] core_worker.cc:2064: Finished executing task 1881b3e9ecbaff55a82936f57c63efdc52c5def701000000, status=OK | |
| [2021-04-29 17:04:55,504 I 5366 5366] core_worker.cc:2064: Finished executing task 346c01bd5c9c2d6ea82936f57c63efdc52c5def701000000, status=OK | |
| [2021-04-29 17:04:55,959 I 5366 5366] core_worker.cc:2064: Finished executing task c15afe04dfdd9219a82936f57c63efdc52c5def701000000, status=OK | |
| [2021-04-29 17:04:57,714 I 5366 5366] core_worker.cc:2064: Finished executing task 4d9fabbc7174054ea82936f57c63efdc52c5def701000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-50e78dadc69667fc539be87a6d100763247fc0bfbadb59f7ee011048_5299.log <== | |
| [2021-04-29 17:03:58,039 E 5299 5299] actor_manager.cc:93: Actor handle already exists fe2d0306af8f2d321bac6ba901000000 | |
| [2021-04-29 17:03:59,657 I 5299 5299] core_worker.cc:2064: Finished executing task f5964f917f9a7611ffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:03:59,665 E 5299 5299] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,665 E 5299 5299] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,731 I 5299 5299] direct_task_transport.cc:47: Creating actor via GCS actor id = : d96e5b6d4d0a5fe52487300a01000000 | |
| [2021-04-29 17:03:59,745 I 5299 6307] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: d96e5b6d4d0a5fe52487300a01000000, ip address: 10.134.60.23, port: 9129, worker_id: 955bdb4c87b104a5ce1216534b994ed229a3b0648f29ebb942fefd77, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,723 I 5299 6307] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: d96e5b6d4d0a5fe52487300a01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:19,263 I 5299 6307] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: d96e5b6d4d0a5fe52487300a01000000, ip address: 10.134.60.23, port: 9129, worker_id: 955bdb4c87b104a5ce1216534b994ed229a3b0648f29ebb942fefd77, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,033 E 5299 5299] actor_manager.cc:93: Actor handle already exists d96e5b6d4d0a5fe52487300a01000000 | |
| [2021-04-29 17:04:28,906 I 5299 5299] core_worker.cc:2064: Finished executing task b7890c8906d42957ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-50f11d407b97647037c44d1498fb9ade13ec78f7d3631519890ecd28_12942.log <== | |
| [2021-04-29 17:04:05,162 I 12942 14532] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,173 E 12942 12942] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,178 I 12942 14532] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:07,361 E 12942 12942] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:07,438 I 12942 12942] direct_task_transport.cc:47: Creating actor via GCS actor id = : cd0766f1baf9a18ce6faa6cb01000000 | |
| [2021-04-29 17:04:07,461 I 12942 14532] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: cd0766f1baf9a18ce6faa6cb01000000, ip address: 10.133.95.40, port: 13723, worker_id: NIL_ID, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:16,918 I 12942 14532] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: cd0766f1baf9a18ce6faa6cb01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,400 I 12942 14532] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: cd0766f1baf9a18ce6faa6cb01000000, ip address: 10.133.95.40, port: 9169, worker_id: 67f9afb08b1e99c19c9bcc0e27793b4a4a6f2193965f1a30679f4144, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,493 E 12942 12942] actor_manager.cc:93: Actor handle already exists cd0766f1baf9a18ce6faa6cb01000000 | |
| [2021-04-29 17:04:29,533 I 12942 12942] core_worker.cc:2064: Finished executing task 05d0aa8cda7f08a4ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-51261c876c1a466be1aa1ec3034ac27485113ad77abb12a757d989af_5373.log <== | |
| [2021-04-29 17:03:46,160 E 5373 5373] actor_manager.cc:93: Actor handle already exists ff125b7bd2b8fb7149bc306301000000 | |
| [2021-04-29 17:03:46,261 I 5373 5373] core_worker.cc:2064: Finished executing task ffffffffffffffff0aeefb151d6809f3f1d8a9f601000000, status=OK | |
| [2021-04-29 17:03:46,261 I 5373 5373] direct_actor_transport.cc:497: Actor creation task finished, task_id: ffffffffffffffff0aeefb151d6809f3f1d8a9f601000000, actor_id: 0aeefb151d6809f3f1d8a9f601000000 | |
| [2021-04-29 17:03:46,270 I 5373 5373] core_worker.cc:2064: Finished executing task dce7e58adc4b0c540aeefb151d6809f3f1d8a9f601000000, status=OK | |
| [2021-04-29 17:03:46,272 I 5373 6992] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 0aeefb151d6809f3f1d8a9f601000000, ip address: 10.133.100.138, port: 9239, worker_id: 51261c876c1a466be1aa1ec3034ac27485113ad77abb12a757d989af, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:46,274 I 5373 5373] core_worker.cc:2064: Finished executing task b0ec528ca2ce0b860aeefb151d6809f3f1d8a9f601000000, status=OK | |
| [2021-04-29 17:03:46,290 I 5373 5373] core_worker.cc:2064: Finished executing task 2c4bce83479063400aeefb151d6809f3f1d8a9f601000000, status=OK | |
| [2021-04-29 17:03:46,595 I 5373 5373] core_worker.cc:2064: Finished executing task 38cee6f113910f5e0aeefb151d6809f3f1d8a9f601000000, status=OK | |
| [2021-04-29 17:03:46,606 I 5373 5373] core_worker.cc:2064: Finished executing task 080f58b5a9604e070aeefb151d6809f3f1d8a9f601000000, status=OK | |
| [2021-04-29 17:03:51,305 I 5373 5373] core_worker.cc:2064: Finished executing task 4aad33a593fbb31e0aeefb151d6809f3f1d8a9f601000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-514bca6434da78395d008c3acc5ab2b01d2acd4bb4d016faa71bf043_12931.log <== | |
| [2021-04-29 17:04:05,000 I 12931 14713] service_based_accessor.cc:579: Received notification for node id = ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, IsAlive = 1 | |
| [2021-04-29 17:04:05,000 I 12931 14713] service_based_accessor.cc:579: Received notification for node id = d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, IsAlive = 1 | |
| [2021-04-29 17:04:05,000 I 12931 14713] service_based_accessor.cc:579: Received notification for node id = b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, IsAlive = 1 | |
| [2021-04-29 17:04:05,000 I 12931 14713] service_based_accessor.cc:579: Received notification for node id = 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, IsAlive = 1 | |
| [2021-04-29 17:04:05,000 I 12931 14713] service_based_accessor.cc:579: Received notification for node id = 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, IsAlive = 1 | |
| [2021-04-29 17:04:05,982 I 12931 14713] core_worker.cc:717: Exit signal received, this process will exit after all outstanding tasks have finished, exit_type=INTENDED_EXIT | |
| [2021-04-29 17:04:05,982 I 12931 12931] raylet_client.cc:129: RayletClient::Disconnect, exit_type=INTENDED_EXIT, has creation_task_exception_pb_bytes=0 | |
| [2021-04-29 17:04:05,982 I 12931 12931] core_worker.cc:308: Removed worker 514bca6434da78395d008c3acc5ab2b01d2acd4bb4d016faa71bf043 | |
| [2021-04-29 17:04:05,986 I 12931 12931] core_worker.cc:181: Destructing CoreWorkerProcess. pid: 12931 | |
| [2021-04-29 17:04:05,986 I 12931 12931] io_service_pool.cc:47: IOServicePool is stopped. | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-51967bdfb84c42c01b003c645b59f46cd82e2a3fe85c0afdf723d2e6_9797.log <== | |
| [2021-04-29 17:04:02,216 I 9797 11378] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,232 E 9797 9797] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:02,240 I 9797 11378] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,346 E 9797 9797] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,422 I 9797 9797] direct_task_transport.cc:47: Creating actor via GCS actor id = : cee0b055099e3a80f05da07701000000 | |
| [2021-04-29 17:04:05,431 I 9797 11378] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: cee0b055099e3a80f05da07701000000, ip address: 10.134.112.222, port: 9110, worker_id: 2c388275fd67e6cc47fd0628f22339763f4d3f6fa78fc953fef2b8f1, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,655 I 9797 11378] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: cee0b055099e3a80f05da07701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,394 I 9797 11378] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: cee0b055099e3a80f05da07701000000, ip address: 10.134.112.222, port: 9110, worker_id: 2c388275fd67e6cc47fd0628f22339763f4d3f6fa78fc953fef2b8f1, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,211 E 9797 9797] actor_manager.cc:93: Actor handle already exists cee0b055099e3a80f05da07701000000 | |
| [2021-04-29 17:04:29,481 I 9797 9797] core_worker.cc:2064: Finished executing task dc81b8b525a90080ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-51ab2f578d7120f09d5a70ba65786b42e8504dd90b32cda5dd3a19ba_5375.log <== | |
| [2021-04-29 17:04:02,543 E 5375 5375] actor_manager.cc:93: Actor handle already exists 61eb54e92f26d32bbbb0392301000000 | |
| [2021-04-29 17:04:10,420 I 5375 5375] core_worker.cc:2064: Finished executing task 3cf191332f3be7eaffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:10,424 E 5375 5375] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,424 E 5375 5375] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,509 I 5375 5375] direct_task_transport.cc:47: Creating actor via GCS actor id = : bc655bde9a3b3354320ea52e01000000 | |
| [2021-04-29 17:04:10,539 I 5375 7014] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: bc655bde9a3b3354320ea52e01000000, ip address: 10.134.60.23, port: 9202, worker_id: c05d6c96cba6fb54d62aaab75d3fd8767a648a8cf0fbb47e012a9b9b, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,366 I 5375 7014] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: bc655bde9a3b3354320ea52e01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:22,889 I 5375 7014] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: bc655bde9a3b3354320ea52e01000000, ip address: 10.134.60.23, port: 9202, worker_id: c05d6c96cba6fb54d62aaab75d3fd8767a648a8cf0fbb47e012a9b9b, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:25,706 E 5375 5375] actor_manager.cc:93: Actor handle already exists bc655bde9a3b3354320ea52e01000000 | |
| [2021-04-29 17:04:29,758 I 5375 5375] core_worker.cc:2064: Finished executing task 99768264477d8e6affffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-51ceba1f3c087f94235fbfb24767bc94df0e3facca1f0c7141f5af77_17466.log <== | |
| [2021-04-29 17:04:09,492 I 17466 19092] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,673 E 17466 17466] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,703 I 17466 19092] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,556 E 17466 17466] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,661 I 17466 17466] direct_task_transport.cc:47: Creating actor via GCS actor id = : 127cc1e93c98cb2ee2534d2601000000 | |
| [2021-04-29 17:04:11,719 I 17466 19092] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 127cc1e93c98cb2ee2534d2601000000, ip address: 10.134.70.227, port: 9195, worker_id: 9425353d1758606c2972efb7d87ca33ca298bf5cb7bb2496a984b47e, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,384 I 17466 19092] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 127cc1e93c98cb2ee2534d2601000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:28,125 I 17466 19092] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 127cc1e93c98cb2ee2534d2601000000, ip address: 10.134.70.227, port: 9195, worker_id: 9425353d1758606c2972efb7d87ca33ca298bf5cb7bb2496a984b47e, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,995 E 17466 17466] actor_manager.cc:93: Actor handle already exists 127cc1e93c98cb2ee2534d2601000000 | |
| [2021-04-29 17:04:29,857 I 17466 17466] core_worker.cc:2064: Finished executing task ece8061e9516843dffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-524cee672bb1e19d0f8279c1aad3bbd60791d6cea80dd9d49a3630b4_7410.log <== | |
| [2021-04-29 17:03:58,036 I 7410 7709] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,064 E 7410 7410] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,070 I 7410 7709] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,676 E 7410 7410] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:58,762 I 7410 7410] direct_task_transport.cc:47: Creating actor via GCS actor id = : b55a5dc61fd4d1ad440dd72901000000 | |
| [2021-04-29 17:03:58,771 I 7410 7709] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: b55a5dc61fd4d1ad440dd72901000000, ip address: 10.134.60.23, port: 9139, worker_id: dd733244586a037e58055a66183fcd55727299ceb4ce38f97fd10391, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,323 I 7410 7709] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: b55a5dc61fd4d1ad440dd72901000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,548 I 7410 7709] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: b55a5dc61fd4d1ad440dd72901000000, ip address: 10.134.60.23, port: 9139, worker_id: dd733244586a037e58055a66183fcd55727299ceb4ce38f97fd10391, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:06,030 E 7410 7410] actor_manager.cc:93: Actor handle already exists b55a5dc61fd4d1ad440dd72901000000 | |
| [2021-04-29 17:04:11,925 I 7410 7410] core_worker.cc:2064: Finished executing task ecfd5328c6370e94ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-524f23c9fcabef744896aa0701f005137e607b7e11b04b25ce56f5e2_14224.log <== | |
| [2021-04-29 17:04:05,911 I 14224 15758] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,968 E 14224 14224] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,981 I 14224 15758] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,771 E 14224 14224] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:08,871 I 14224 14224] direct_task_transport.cc:47: Creating actor via GCS actor id = : ad1e9226fcdb8f84fec1d10c01000000 | |
| [2021-04-29 17:04:08,923 I 14224 15758] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: ad1e9226fcdb8f84fec1d10c01000000, ip address: 10.134.61.25, port: 9181, worker_id: b23f340b911b1fc93cb4225a556489097a4091e0a611d2a4e6cbabdc, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:19,382 I 14224 15758] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: ad1e9226fcdb8f84fec1d10c01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,401 I 14224 15758] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: ad1e9226fcdb8f84fec1d10c01000000, ip address: 10.134.61.25, port: 9181, worker_id: b23f340b911b1fc93cb4225a556489097a4091e0a611d2a4e6cbabdc, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,503 E 14224 14224] actor_manager.cc:93: Actor handle already exists ad1e9226fcdb8f84fec1d10c01000000 | |
| [2021-04-29 17:04:29,534 I 14224 14224] core_worker.cc:2064: Finished executing task d9ff272afa46342affffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-525b23d603eac0addfa3965428b3945659d6b78c8f8212bb95a20293_12901.log <== | |
| [2021-04-29 17:04:04,776 I 12901 14390] service_based_accessor.cc:579: Received notification for node id = ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, IsAlive = 1 | |
| [2021-04-29 17:04:04,776 I 12901 14390] service_based_accessor.cc:579: Received notification for node id = d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, IsAlive = 1 | |
| [2021-04-29 17:04:04,776 I 12901 14390] service_based_accessor.cc:579: Received notification for node id = b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, IsAlive = 1 | |
| [2021-04-29 17:04:04,776 I 12901 14390] service_based_accessor.cc:579: Received notification for node id = 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, IsAlive = 1 | |
| [2021-04-29 17:04:04,776 I 12901 14390] service_based_accessor.cc:579: Received notification for node id = 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, IsAlive = 1 | |
| [2021-04-29 17:04:05,981 I 12901 14390] core_worker.cc:717: Exit signal received, this process will exit after all outstanding tasks have finished, exit_type=INTENDED_EXIT | |
| [2021-04-29 17:04:05,981 I 12901 12901] raylet_client.cc:129: RayletClient::Disconnect, exit_type=INTENDED_EXIT, has creation_task_exception_pb_bytes=0 | |
| [2021-04-29 17:04:05,981 I 12901 12901] core_worker.cc:308: Removed worker 525b23d603eac0addfa3965428b3945659d6b78c8f8212bb95a20293 | |
| [2021-04-29 17:04:05,984 I 12901 12901] core_worker.cc:181: Destructing CoreWorkerProcess. pid: 12901 | |
| [2021-04-29 17:04:05,985 I 12901 12901] io_service_pool.cc:47: IOServicePool is stopped. | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-52740246935d76014467250e6dbf2e4fce79b78adf0fe49c4216e79f_12143.log <== | |
| [2021-04-29 17:04:03,611 I 12143 12788] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,613 E 12143 12143] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:03,615 I 12143 12788] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,309 E 12143 12143] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:06,467 I 12143 12143] direct_task_transport.cc:47: Creating actor via GCS actor id = : ee282357a510b926e14f7c5401000000 | |
| [2021-04-29 17:04:06,485 I 12143 12788] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: ee282357a510b926e14f7c5401000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:12,240 I 12143 12788] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: ee282357a510b926e14f7c5401000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,395 I 12143 12788] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: ee282357a510b926e14f7c5401000000, ip address: 10.134.61.25, port: 9102, worker_id: 5cd1516b806039c9f9090292f24aa23a4ff17f123708c37c86778691, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,771 E 12143 12143] actor_manager.cc:93: Actor handle already exists ee282357a510b926e14f7c5401000000 | |
| [2021-04-29 17:04:29,547 I 12143 12143] core_worker.cc:2064: Finished executing task 9e76ee6037b3ed1dffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-52b17cc3f347c29a944c9d179c6c1d28576626db67aa7c598600d30a_7465.log <== | |
| [2021-04-29 17:03:58,240 I 7465 8280] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,242 E 7465 7465] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,243 I 7465 8280] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,712 E 7465 7465] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:58,804 I 7465 7465] direct_task_transport.cc:47: Creating actor via GCS actor id = : 8af2dc1747decc9ff0c2356501000000 | |
| [2021-04-29 17:03:58,819 I 7465 8280] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 8af2dc1747decc9ff0c2356501000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,898 I 7465 8280] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 8af2dc1747decc9ff0c2356501000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,137 I 7465 8280] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8af2dc1747decc9ff0c2356501000000, ip address: 10.134.70.227, port: 9126, worker_id: 396079f6d25ca50a6fc3de75c7cb60413d00b3ebb42418f9749a7c1e, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:06,438 E 7465 7465] actor_manager.cc:93: Actor handle already exists 8af2dc1747decc9ff0c2356501000000 | |
| [2021-04-29 17:04:12,122 I 7465 7465] core_worker.cc:2064: Finished executing task f8a5e1557caee0c3ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-52e565fc0bfa93b0a41dc9cb106e3b9702b6dca171c5f4b8fa981132_9684.log <== | |
| [2021-04-29 17:04:01,192 I 9684 10436] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,219 E 9684 9684] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,230 I 9684 10436] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,777 E 9684 9684] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:01,844 I 9684 9684] direct_task_transport.cc:47: Creating actor via GCS actor id = : f08225d4283ad99b67adbd4601000000 | |
| [2021-04-29 17:04:01,862 I 9684 10436] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: f08225d4283ad99b67adbd4601000000, ip address: 10.133.95.40, port: 9120, worker_id: 810f7b441277599b4805b8a85b1eea8a244079334ef8a1ea1367a739, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,612 I 9684 10436] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: f08225d4283ad99b67adbd4601000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,390 I 9684 10436] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: f08225d4283ad99b67adbd4601000000, ip address: 10.133.95.40, port: 9120, worker_id: 810f7b441277599b4805b8a85b1eea8a244079334ef8a1ea1367a739, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,749 E 9684 9684] actor_manager.cc:93: Actor handle already exists f08225d4283ad99b67adbd4601000000 | |
| [2021-04-29 17:04:29,544 I 9684 9684] core_worker.cc:2064: Finished executing task 0e2469fedddd8ad6ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-533c3b690ca46e5dbf654b68b51205bdd91830be50524e74a0d88732_15908.log <== | |
| [2021-04-29 17:04:07,995 I 15908 16980] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,268 E 15908 15908] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,330 I 15908 16980] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,964 E 15908 15908] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,059 I 15908 15908] direct_task_transport.cc:47: Creating actor via GCS actor id = : d70b48e21dc995077120fa4b01000000 | |
| [2021-04-29 17:04:11,102 I 15908 16980] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: d70b48e21dc995077120fa4b01000000, ip address: 10.134.70.227, port: 9207, worker_id: 081d0d449f9807ebaab2623227ba171f7cde9726812d4b8cf5929534, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,370 I 15908 16980] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: d70b48e21dc995077120fa4b01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:25,084 I 15908 16980] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: d70b48e21dc995077120fa4b01000000, ip address: 10.134.70.227, port: 9207, worker_id: 081d0d449f9807ebaab2623227ba171f7cde9726812d4b8cf5929534, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,003 E 15908 15908] actor_manager.cc:93: Actor handle already exists d70b48e21dc995077120fa4b01000000 | |
| [2021-04-29 17:04:29,816 I 15908 15908] core_worker.cc:2064: Finished executing task 98c70fe78a688578ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-53659245f58a5c3083a4691de6f284a878d51f59f8138cd35e5efc28_16708.log <== | |
| [2021-04-29 17:04:09,293 I 16708 18728] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,463 E 16708 16708] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,492 I 16708 18728] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,522 E 16708 16708] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,628 I 16708 16708] direct_task_transport.cc:47: Creating actor via GCS actor id = : 86c162085421499a0730607601000000 | |
| [2021-04-29 17:04:11,667 I 16708 18728] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 86c162085421499a0730607601000000, ip address: 10.134.112.222, port: 9185, worker_id: a50c061e50ae2879ce6366795b1b902de5ffa0125a2bca0203ad2b9d, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,384 I 16708 18728] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 86c162085421499a0730607601000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:25,642 I 16708 18728] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 86c162085421499a0730607601000000, ip address: 10.134.112.222, port: 9185, worker_id: a50c061e50ae2879ce6366795b1b902de5ffa0125a2bca0203ad2b9d, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,457 E 16708 16708] actor_manager.cc:93: Actor handle already exists 86c162085421499a0730607601000000 | |
| [2021-04-29 17:04:29,839 I 16708 16708] core_worker.cc:2064: Finished executing task 4c6b7009aa3890f4ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-5365a9f13f87b9eb24b30636a7b476b608a5d284e42b9f1057512d7d_12787.log <== | |
| [2021-04-29 17:04:04,817 I 12787 13786] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,865 E 12787 12787] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:04,891 I 12787 13786] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,646 E 12787 12787] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:06,852 I 12787 12787] direct_task_transport.cc:47: Creating actor via GCS actor id = : 1fe79900d7e50230612f467c01000000 | |
| [2021-04-29 17:04:06,865 I 12787 13786] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 1fe79900d7e50230612f467c01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:13,954 I 12787 13786] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 1fe79900d7e50230612f467c01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,397 I 12787 13786] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1fe79900d7e50230612f467c01000000, ip address: 10.134.70.227, port: 9230, worker_id: 13fa188bf32c9a9203cee809958c9eb3cb820ecfffd75b4ae113f85d, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,359 E 12787 12787] actor_manager.cc:93: Actor handle already exists 1fe79900d7e50230612f467c01000000 | |
| [2021-04-29 17:04:29,521 I 12787 12787] core_worker.cc:2064: Finished executing task db788c77cdc7d972ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-538825eae7db93ab141968bfdb7528997e978b33a603b08f26de6a70_7416.log <== | |
| [2021-04-29 17:03:57,977 I 7416 7784] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:57,980 E 7416 7416] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:57,986 I 7416 7784] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,641 E 7416 7416] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:58,701 I 7416 7416] direct_task_transport.cc:47: Creating actor via GCS actor id = : a928d6c01eed2267bbfb0f1901000000 | |
| [2021-04-29 17:03:58,713 I 7416 7784] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: a928d6c01eed2267bbfb0f1901000000, ip address: 10.134.112.222, port: 9139, worker_id: 4e35953a0c4bcfc6e611c4d825239a610dfe4445991e67b90f7568c2, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,887 I 7416 7784] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: a928d6c01eed2267bbfb0f1901000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,549 I 7416 7784] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: a928d6c01eed2267bbfb0f1901000000, ip address: 10.134.112.222, port: 9139, worker_id: 4e35953a0c4bcfc6e611c4d825239a610dfe4445991e67b90f7568c2, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:06,256 E 7416 7416] actor_manager.cc:93: Actor handle already exists a928d6c01eed2267bbfb0f1901000000 | |
| [2021-04-29 17:04:12,024 I 7416 7416] core_worker.cc:2064: Finished executing task 241c984afc096d69ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-53f33d485b9730d25c0ec4f645a6f7d1c587da8a58fef6394ef19d1d_12090.log <== | |
| [2021-04-29 17:04:03,373 I 12090 12392] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,399 E 12090 12090] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:03,409 I 12090 12392] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,866 E 12090 12090] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,943 I 12090 12090] direct_task_transport.cc:47: Creating actor via GCS actor id = : 5e5e817a14586102d7b7c9c001000000 | |
| [2021-04-29 17:04:05,968 I 12090 12392] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 5e5e817a14586102d7b7c9c001000000, ip address: 10.133.95.40, port: 9101, worker_id: 28bac2c2ac38167a2f512e5687f753baa12a378ed0be3d5308e3db59, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,947 I 12090 12392] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 5e5e817a14586102d7b7c9c001000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,394 I 12090 12392] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 5e5e817a14586102d7b7c9c001000000, ip address: 10.133.95.40, port: 9101, worker_id: 28bac2c2ac38167a2f512e5687f753baa12a378ed0be3d5308e3db59, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,207 E 12090 12090] actor_manager.cc:93: Actor handle already exists 5e5e817a14586102d7b7c9c001000000 | |
| [2021-04-29 17:04:29,479 I 12090 12090] core_worker.cc:2064: Finished executing task 1b390c2b8a3d5f11ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-544bc88c171f9bffff2244300ec4c0346535069b688f39569beb071b_5326.log <== | |
| [2021-04-29 17:03:46,429 I 5326 5326] core_worker.cc:2064: Finished executing task 005635bb046e450de95ce44a3b410dc523c450c901000000, status=OK | |
| [2021-04-29 17:04:48,817 I 5326 5326] core_worker.cc:2064: Finished executing task 5ff50f8cbe3df1d5e95ce44a3b410dc523c450c901000000, status=OK | |
| [2021-04-29 17:04:49,027 I 5326 5326] core_worker.cc:2064: Finished executing task 334161fb47be39cae95ce44a3b410dc523c450c901000000, status=OK | |
| [2021-04-29 17:04:50,944 I 5326 5326] core_worker.cc:2064: Finished executing task 3d5ad75c34041d99e95ce44a3b410dc523c450c901000000, status=OK | |
| [2021-04-29 17:04:51,457 I 5326 5326] core_worker.cc:2064: Finished executing task ab2d85cf1812e30fe95ce44a3b410dc523c450c901000000, status=OK | |
| [2021-04-29 17:04:53,324 I 5326 5326] core_worker.cc:2064: Finished executing task 9c862a3ec0aedbc7e95ce44a3b410dc523c450c901000000, status=OK | |
| [2021-04-29 17:04:54,349 I 5326 5326] core_worker.cc:2064: Finished executing task a4f4c52b46cc0df1e95ce44a3b410dc523c450c901000000, status=OK | |
| [2021-04-29 17:04:56,268 I 5326 5326] core_worker.cc:2064: Finished executing task 721020ecd0197cd1e95ce44a3b410dc523c450c901000000, status=OK | |
| [2021-04-29 17:04:56,648 I 5326 5326] core_worker.cc:2064: Finished executing task 9298b8b1d259b3b5e95ce44a3b410dc523c450c901000000, status=OK | |
| [2021-04-29 17:04:57,616 I 5326 5326] core_worker.cc:2064: Finished executing task d3d56dbf9ed4057be95ce44a3b410dc523c450c901000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-545e060291fe17b9cf53440411069c8afac59b56983dd93a459415b1_16024.log <== | |
| [2021-04-29 17:04:08,580 I 16024 17739] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,666 E 16024 16024] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,691 I 16024 17739] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,159 E 16024 16024] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,253 I 16024 16024] direct_task_transport.cc:47: Creating actor via GCS actor id = : 611d275d65d44349146491c201000000 | |
| [2021-04-29 17:04:11,303 I 16024 17739] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 611d275d65d44349146491c201000000, ip address: 10.134.112.222, port: 9195, worker_id: 708fbbcf3b2019a92dfba758a23dfbedae6302b73aa10f69de588244, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,377 I 16024 17739] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 611d275d65d44349146491c201000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,898 I 16024 17739] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 611d275d65d44349146491c201000000, ip address: 10.134.112.222, port: 9195, worker_id: 708fbbcf3b2019a92dfba758a23dfbedae6302b73aa10f69de588244, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:27,634 E 16024 16024] actor_manager.cc:93: Actor handle already exists 611d275d65d44349146491c201000000 | |
| [2021-04-29 17:04:29,817 I 16024 16024] core_worker.cc:2064: Finished executing task 39636d079a8f0839ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-5492a89621decf278c138b6a8b59ba4d4b8809c712c50a21086c1e97_13282.log <== | |
| [2021-04-29 17:04:06,150 I 13282 15024] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,168 E 13282 13282] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:06,199 I 13282 15024] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,828 E 13282 13282] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:08,962 I 13282 13282] direct_task_transport.cc:47: Creating actor via GCS actor id = : bf10e601d5fffe7b7f74176901000000 | |
| [2021-04-29 17:04:08,998 I 13282 15024] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: bf10e601d5fffe7b7f74176901000000, ip address: 10.133.95.40, port: 9183, worker_id: baaf3e8b1afdfe8b6bc2b2ba2ba056f472a09e717c1d46838e696df5, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:20,662 I 13282 15024] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: bf10e601d5fffe7b7f74176901000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,401 I 13282 15024] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: bf10e601d5fffe7b7f74176901000000, ip address: 10.133.95.40, port: 9183, worker_id: baaf3e8b1afdfe8b6bc2b2ba2ba056f472a09e717c1d46838e696df5, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,550 E 13282 13282] actor_manager.cc:93: Actor handle already exists bf10e601d5fffe7b7f74176901000000 | |
| [2021-04-29 17:04:29,537 I 13282 13282] core_worker.cc:2064: Finished executing task 1a3f6e5be5f90329ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-556964fedbdc7ba026d9a86cce06435f7c27f94b2f03b070f6d56640_16681.log <== | |
| [2021-04-29 17:04:09,224 I 16681 18489] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,350 E 16681 16681] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,380 I 16681 18489] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,437 E 16681 16681] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,535 I 16681 16681] direct_task_transport.cc:47: Creating actor via GCS actor id = : ce0a822de25d679353a9230e01000000 | |
| [2021-04-29 17:04:11,554 I 16681 18489] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: ce0a822de25d679353a9230e01000000, ip address: 10.133.95.40, port: 9189, worker_id: dab5ed3f54967fbd1a325d265df7857a41ee2358a9748aa9ba00226f, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,381 I 16681 18489] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: ce0a822de25d679353a9230e01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:25,354 I 16681 18489] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: ce0a822de25d679353a9230e01000000, ip address: 10.133.95.40, port: 9189, worker_id: dab5ed3f54967fbd1a325d265df7857a41ee2358a9748aa9ba00226f, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,369 E 16681 16681] actor_manager.cc:93: Actor handle already exists ce0a822de25d679353a9230e01000000 | |
| [2021-04-29 17:04:29,835 I 16681 16681] core_worker.cc:2064: Finished executing task 0f3117bc21629689ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-55a7e579ab1998e1fd47ba79e0a9660d8cba2ba56c95c080591bd384_5273.log <== | |
| [2021-04-29 17:03:58,943 E 5273 5273] actor_manager.cc:93: Actor handle already exists 08ccf890c0f63dd3b24df4f801000000 | |
| [2021-04-29 17:04:00,648 I 5273 5273] core_worker.cc:2064: Finished executing task ab64d1d71e706cffffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:00,654 E 5273 5273] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:00,654 E 5273 5273] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:00,733 I 5273 5273] direct_task_transport.cc:47: Creating actor via GCS actor id = : de6097a09b761865c2816eb301000000 | |
| [2021-04-29 17:04:00,751 I 5273 6203] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: de6097a09b761865c2816eb301000000, ip address: 10.134.142.44, port: 9112, worker_id: 2df17eac160c661e4d2300d9759ec48ba591b43cedf7cb3d9d78da5b, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,733 I 5273 6203] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: de6097a09b761865c2816eb301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,379 I 5273 6203] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: de6097a09b761865c2816eb301000000, ip address: 10.134.142.44, port: 9112, worker_id: 2df17eac160c661e4d2300d9759ec48ba591b43cedf7cb3d9d78da5b, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,001 E 5273 5273] actor_manager.cc:93: Actor handle already exists de6097a09b761865c2816eb301000000 | |
| [2021-04-29 17:04:29,439 I 5273 5273] core_worker.cc:2064: Finished executing task 619e55fae086ea32ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-55f24bc5f192fa26fdf122a77f249020400b5fba4876cc81e25f7030_15928.log <== | |
| [2021-04-29 17:04:08,145 I 15928 17240] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,379 E 15928 15928] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,412 I 15928 17240] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,002 E 15928 15928] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,093 I 15928 15928] direct_task_transport.cc:47: Creating actor via GCS actor id = : 3c7e1c6dc091c782c1d99e9201000000 | |
| [2021-04-29 17:04:11,138 I 15928 17240] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 3c7e1c6dc091c782c1d99e9201000000, ip address: 10.134.61.25, port: 9197, worker_id: 68268cb346a84584bd3fe9a1eec543a3f89c964e3154629ec9ae0f54, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,372 I 15928 17240] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 3c7e1c6dc091c782c1d99e9201000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,383 I 15928 17240] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 3c7e1c6dc091c782c1d99e9201000000, ip address: 10.134.61.25, port: 9197, worker_id: 68268cb346a84584bd3fe9a1eec543a3f89c964e3154629ec9ae0f54, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:27,095 E 15928 15928] actor_manager.cc:93: Actor handle already exists 3c7e1c6dc091c782c1d99e9201000000 | |
| [2021-04-29 17:04:29,789 I 15928 15928] core_worker.cc:2064: Finished executing task 54f6a0a756be1813ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-567df7a801a81b6944fe1f29f8ada10555eee1face542f862905829f_16020.log <== | |
| [2021-04-29 17:04:08,415 I 16020 17615] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,507 E 16020 16020] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,537 I 16020 17615] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,054 E 16020 16020] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,164 I 16020 16020] direct_task_transport.cc:47: Creating actor via GCS actor id = : bbe5deceffc8a313f044a50101000000 | |
| [2021-04-29 17:04:11,194 I 16020 17615] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: bbe5deceffc8a313f044a50101000000, ip address: 10.134.60.23, port: 9193, worker_id: 43546c94abc893404031af28044689803ee4f5bd2d63c04e2026566e, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,375 I 16020 17615] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: bbe5deceffc8a313f044a50101000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,725 I 16020 17615] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: bbe5deceffc8a313f044a50101000000, ip address: 10.134.60.23, port: 9193, worker_id: 43546c94abc893404031af28044689803ee4f5bd2d63c04e2026566e, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:27,574 E 16020 16020] actor_manager.cc:93: Actor handle already exists bbe5deceffc8a313f044a50101000000 | |
| [2021-04-29 17:04:29,806 I 16020 16020] core_worker.cc:2064: Finished executing task f11ccc70a46fb32effffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-56813a7d38517593567f88035b9cb7f604123e2c0f258d456afb636e_7553.log <== | |
| [2021-04-29 17:03:58,834 I 7553 9078] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,849 E 7553 7553] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,856 I 7553 9078] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,587 E 7553 7553] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,639 I 7553 7553] direct_task_transport.cc:47: Creating actor via GCS actor id = : ab734b5e3b4570cbbf60924d01000000 | |
| [2021-04-29 17:03:59,644 I 7553 9078] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: ab734b5e3b4570cbbf60924d01000000, ip address: 10.134.142.16, port: 9133, worker_id: b6ec7812c704dff81b6eb072b52259c9cf9db9fae0566bff5ac693fd, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,699 I 7553 9078] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: ab734b5e3b4570cbbf60924d01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:17,279 I 7553 9078] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: ab734b5e3b4570cbbf60924d01000000, ip address: 10.134.142.16, port: 9133, worker_id: b6ec7812c704dff81b6eb072b52259c9cf9db9fae0566bff5ac693fd, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:20,120 E 7553 7553] actor_manager.cc:93: Actor handle already exists ab734b5e3b4570cbbf60924d01000000 | |
| [2021-04-29 17:04:28,157 I 7553 7553] core_worker.cc:2064: Finished executing task aeb19104c814ed6cffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-5687562c70c74f254393608070ddbed39151c88e63dad2aa8418cb9f_4055.log <== | |
| [2021-04-29 17:04:02,351 E 4055 4055] actor_manager.cc:93: Actor handle already exists f266678ad87f2cf921841eb301000000 | |
| [2021-04-29 17:04:10,137 I 4055 4055] core_worker.cc:2064: Finished executing task f72ba50bcf096161ffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:10,141 E 4055 4055] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,142 E 4055 4055] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,243 I 4055 4055] direct_task_transport.cc:47: Creating actor via GCS actor id = : a2e1126bee423676026a383e01000000 | |
| [2021-04-29 17:04:10,279 I 4055 4799] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: a2e1126bee423676026a383e01000000, ip address: 10.134.142.44, port: 12853, worker_id: NIL_ID, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,363 I 4055 4799] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: a2e1126bee423676026a383e01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,398 I 4055 4799] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: a2e1126bee423676026a383e01000000, ip address: 10.134.142.44, port: 9217, worker_id: de74ec13ff73e9868cc41b1773a81c67334ddd9c9d30ae922d34a02e, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:27,097 E 4055 4055] actor_manager.cc:93: Actor handle already exists a2e1126bee423676026a383e01000000 | |
| [2021-04-29 17:04:29,791 I 4055 4055] core_worker.cc:2064: Finished executing task a65920128917daefffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-573a06c8db2ec350bab9a071efea3c658260cc95faabc7e1173b6eed_14369.log <== | |
| [2021-04-29 17:04:05,703 I 14369 15558] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,731 E 14369 14369] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,795 I 14369 15558] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,240 E 14369 14369] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:08,407 I 14369 14369] direct_task_transport.cc:47: Creating actor via GCS actor id = : 8f55b55556ef965eba51acfc01000000 | |
| [2021-04-29 17:04:08,445 I 14369 15558] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 8f55b55556ef965eba51acfc01000000, ip address: 10.134.142.44, port: 9226, worker_id: 9639ade4df3e6b8aafe9a7fd98d5516f9637d52df3b91b17dea16a85, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:19,064 I 14369 15558] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 8f55b55556ef965eba51acfc01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:22,769 I 14369 15558] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8f55b55556ef965eba51acfc01000000, ip address: 10.134.142.44, port: 9226, worker_id: 9639ade4df3e6b8aafe9a7fd98d5516f9637d52df3b91b17dea16a85, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:25,472 E 14369 14369] actor_manager.cc:93: Actor handle already exists 8f55b55556ef965eba51acfc01000000 | |
| [2021-04-29 17:04:29,740 I 14369 14369] core_worker.cc:2064: Finished executing task edbcafb5b5667672ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-578ab6ccdcbeb4bc8af29f6007810950f3d863b9213511b33b563aba_12836.log <== | |
| [2021-04-29 17:04:04,789 I 12836 13711] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,846 E 12836 12836] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:04,879 I 12836 13711] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,645 E 12836 12836] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:06,851 I 12836 12836] direct_task_transport.cc:47: Creating actor via GCS actor id = : 428c26d83ef28fe9c5d1204b01000000 | |
| [2021-04-29 17:04:06,866 I 12836 13711] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 428c26d83ef28fe9c5d1204b01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:13,807 I 12836 13711] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 428c26d83ef28fe9c5d1204b01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,398 I 12836 13711] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 428c26d83ef28fe9c5d1204b01000000, ip address: 10.134.112.222, port: 9153, worker_id: fbb520c4f8aaccbdb28269a533b714da6796f33b6a08655cc85c7375, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,392 E 12836 12836] actor_manager.cc:93: Actor handle already exists 428c26d83ef28fe9c5d1204b01000000 | |
| [2021-04-29 17:04:29,521 I 12836 12836] core_worker.cc:2064: Finished executing task 227f6e8e863e8a74ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-57900e7d7d8316641d29a24bb7b3c7d75c66ab385942e0d42e9f03be_18593.log <== | |
| [2021-04-29 17:04:10,205 I 18593 20057] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,315 E 18593 18593] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,338 I 18593 20057] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,625 E 18593 18593] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,744 I 18593 18593] direct_task_transport.cc:47: Creating actor via GCS actor id = : aef1b9d156d084a0731159d101000000 | |
| [2021-04-29 17:04:11,779 I 18593 20057] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: aef1b9d156d084a0731159d101000000, ip address: 10.133.95.40, port: 9176, worker_id: cb2fc078be05e9a21dd38c08a7752444c4a866cd7025643101f7e48c, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,389 I 18593 20057] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: aef1b9d156d084a0731159d101000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:28,160 I 18593 20057] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: aef1b9d156d084a0731159d101000000, ip address: 10.133.95.40, port: 9176, worker_id: cb2fc078be05e9a21dd38c08a7752444c4a866cd7025643101f7e48c, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:29,003 E 18593 18593] actor_manager.cc:93: Actor handle already exists aef1b9d156d084a0731159d101000000 | |
| [2021-04-29 17:04:29,859 I 18593 18593] core_worker.cc:2064: Finished executing task e67a8fb17aafe222ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-57dddc6ba12f82a806baef68c0489f10db021718fc16223e56b6e53d_5293.log <== | |
| [2021-04-29 17:03:59,580 E 5293 5293] actor_manager.cc:93: Actor handle already exists 8bb3f12fbcbfc5fa34aafba701000000 | |
| [2021-04-29 17:04:01,767 I 5293 5293] core_worker.cc:2064: Finished executing task 1dc7ab52653f9f86ffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:01,777 E 5293 5293] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,778 E 5293 5293] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,842 I 5293 6149] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: b2ac1e3430e7e53bed0835a201000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,858 I 5293 5293] direct_task_transport.cc:47: Creating actor via GCS actor id = : b2ac1e3430e7e53bed0835a201000000 | |
| [2021-04-29 17:04:02,444 I 5293 6149] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: b2ac1e3430e7e53bed0835a201000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,390 I 5293 6149] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: b2ac1e3430e7e53bed0835a201000000, ip address: 10.134.70.227, port: 9109, worker_id: 982ce7a0735320e6634c62e8bce69b134d177872b3daae669026dfdd, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,085 E 5293 5293] actor_manager.cc:93: Actor handle already exists b2ac1e3430e7e53bed0835a201000000 | |
| [2021-04-29 17:04:29,454 I 5293 5293] core_worker.cc:2064: Finished executing task 56383e37125b080affffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-58400242c046d7676433e40e0e47cec21caf77e69ada84c3ee974d89_9675.log <== | |
| [2021-04-29 17:04:01,231 I 9675 10539] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,256 E 9675 9675] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,264 I 9675 10539] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,906 E 9675 9675] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:01,968 I 9675 9675] direct_task_transport.cc:47: Creating actor via GCS actor id = : aae0d8d4bcfbfd788229b16201000000 | |
| [2021-04-29 17:04:01,974 I 9675 10539] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: aae0d8d4bcfbfd788229b16201000000, ip address: 10.134.142.44, port: 9108, worker_id: 33bdfae26011ba7fb3953c20efa9836ea9c91e793bba7deaaaf618db, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,528 I 9675 10539] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: aae0d8d4bcfbfd788229b16201000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,390 I 9675 10539] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: aae0d8d4bcfbfd788229b16201000000, ip address: 10.134.142.44, port: 9108, worker_id: 33bdfae26011ba7fb3953c20efa9836ea9c91e793bba7deaaaf618db, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,039 E 9675 9675] actor_manager.cc:93: Actor handle already exists aae0d8d4bcfbfd788229b16201000000 | |
| [2021-04-29 17:04:29,442 I 9675 9675] core_worker.cc:2064: Finished executing task c2b97e2ade46c7ecffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-589197c364e5bbaa512651b484352976a65332710bc991861b27fdb8_7415.log <== | |
| [2021-04-29 17:03:57,982 I 7415 7820] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:57,983 E 7415 7415] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,006 I 7415 7820] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,661 E 7415 7415] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:58,728 I 7415 7415] direct_task_transport.cc:47: Creating actor via GCS actor id = : 3e169e485ac922b36c7de95c01000000 | |
| [2021-04-29 17:03:58,745 I 7415 7820] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 3e169e485ac922b36c7de95c01000000, ip address: 10.134.142.44, port: 9128, worker_id: 991fc3a9fcf70084178baf2fa67b25c0874a5e9d884c6ed8ccffb925, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,960 I 7415 7820] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 3e169e485ac922b36c7de95c01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,549 I 7415 7820] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 3e169e485ac922b36c7de95c01000000, ip address: 10.134.142.44, port: 9128, worker_id: 991fc3a9fcf70084178baf2fa67b25c0874a5e9d884c6ed8ccffb925, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:06,138 E 7415 7415] actor_manager.cc:93: Actor handle already exists 3e169e485ac922b36c7de95c01000000 | |
| [2021-04-29 17:04:11,957 I 7415 7415] core_worker.cc:2064: Finished executing task 9616ee2c610084fdffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-58f50750d4ef7bfab0f6e4060775c56ea3f07110981d18a71ef60abe_7523.log <== | |
| [2021-04-29 17:03:59,057 I 7523 8877] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,067 E 7523 7523] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,073 I 7523 8877] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,617 E 7523 7523] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,677 I 7523 7523] direct_task_transport.cc:47: Creating actor via GCS actor id = : 2ada00984bdd25dfeff158f101000000 | |
| [2021-04-29 17:03:59,686 I 7523 8877] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 2ada00984bdd25dfeff158f101000000, ip address: 10.134.61.25, port: 9129, worker_id: c4f0398cec2491ded2d225993f1d2b0a9c8c677d721bd94def3830f4, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,721 I 7523 8877] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 2ada00984bdd25dfeff158f101000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:18,830 I 7523 8877] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 2ada00984bdd25dfeff158f101000000, ip address: 10.134.61.25, port: 9129, worker_id: c4f0398cec2491ded2d225993f1d2b0a9c8c677d721bd94def3830f4, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:21,726 E 7523 7523] actor_manager.cc:93: Actor handle already exists 2ada00984bdd25dfeff158f101000000 | |
| [2021-04-29 17:04:28,842 I 7523 7523] core_worker.cc:2064: Finished executing task 704b55f42489fc44ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-5910a246b33493bdcd8d8cb9c8dcd470751db46059aa909fe28a8fd0_14013.log <== | |
| [2021-04-29 17:04:05,630 I 14013 15321] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,666 E 14013 14013] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,677 I 14013 15321] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:07,969 E 14013 14013] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:08,195 I 14013 14013] direct_task_transport.cc:47: Creating actor via GCS actor id = : 705167408a57843601892b4301000000 | |
| [2021-04-29 17:04:08,251 I 14013 15321] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 705167408a57843601892b4301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:18,690 I 14013 15321] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 705167408a57843601892b4301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,400 I 14013 15321] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 705167408a57843601892b4301000000, ip address: 10.134.61.25, port: 9167, worker_id: cf96a2e7c666dbd0a3c3915d9f7c79cc4dfbf1e9b2a12fd98ca16351, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,469 E 14013 14013] actor_manager.cc:93: Actor handle already exists 705167408a57843601892b4301000000 | |
| [2021-04-29 17:04:29,539 I 14013 14013] core_worker.cc:2064: Finished executing task 98564c0f209ee4beffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-59619cb9e9aa958d17251b8cd812f8bbc6aa1c308048565cbd6c7219_4124.log <== | |
| [2021-04-29 17:03:36,359 I 4124 5007] direct_task_transport.cc:189: Task queue empty, canceling lease request | |
| [2021-04-29 17:03:36,652 I 4124 5007] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: e2210617ebcbee4a0c47919d01000000, ip address: 10.133.100.138, port: 9150, worker_id: cbc2355f624dcc5e56357250514b9ea10b8c28a6c25b757892675b11, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:36,654 E 4124 4124] actor_manager.cc:93: Actor handle already exists e2210617ebcbee4a0c47919d01000000 | |
| [2021-04-29 17:03:37,027 E 4124 4124] actor_manager.cc:93: Actor handle already exists 5fcfdec7dbd20ae44c215a0801000000 | |
| [2021-04-29 17:03:37,027 E 4124 4124] actor_manager.cc:93: Actor handle already exists 5fcfdec7dbd20ae44c215a0801000000 | |
| [2021-04-29 17:03:37,045 I 4124 5007] direct_task_transport.cc:189: Task queue empty, canceling lease request | |
| [2021-04-29 17:03:37,342 I 4124 5007] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:37,344 E 4124 4124] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:05:08,999 I 4124 4124] core_worker.cc:2064: Finished executing task 54cc81a39af1e4fa5fcfdec7dbd20ae44c215a0801000000, status=OK | |
| [2021-04-29 17:05:09,000 I 4124 4124] core_worker.cc:2064: Finished executing task e7b04d2c8c8440645fcfdec7dbd20ae44c215a0801000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-5a63c8e2ab22379ebbe5a43d9637d0cecb1ce304705edd15bff98283_7427.log <== | |
| [2021-04-29 17:03:58,237 I 7427 8258] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,238 E 7427 7427] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,239 I 7427 8258] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,711 E 7427 7427] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:58,801 I 7427 7427] direct_task_transport.cc:47: Creating actor via GCS actor id = : 1b5c0ef341af5c3bde1ecfb801000000 | |
| [2021-04-29 17:03:58,820 I 7427 8258] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 1b5c0ef341af5c3bde1ecfb801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,618 I 7427 8258] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 1b5c0ef341af5c3bde1ecfb801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,809 I 7427 8258] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1b5c0ef341af5c3bde1ecfb801000000, ip address: 10.134.142.16, port: 9136, worker_id: 2c7a5345899dc4524008c5eab430c0bbe26e4504ad61760c19663e37, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:06,378 E 7427 7427] actor_manager.cc:93: Actor handle already exists 1b5c0ef341af5c3bde1ecfb801000000 | |
| [2021-04-29 17:04:12,107 I 7427 7427] core_worker.cc:2064: Finished executing task 77aec7c4d6182006ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-5a776b996a0d384b6af9a820bf5cb989821ba5b8e26431184d77d93e_9785.log <== | |
| [2021-04-29 17:04:02,216 I 9785 11415] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,232 E 9785 9785] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:02,240 I 9785 11415] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,346 E 9785 9785] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,427 I 9785 9785] direct_task_transport.cc:47: Creating actor via GCS actor id = : cccb1239efcaf36f8df1cb7201000000 | |
| [2021-04-29 17:04:05,447 I 9785 11415] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: cccb1239efcaf36f8df1cb7201000000, ip address: 10.134.61.130, port: 13217, worker_id: NIL_ID, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,759 I 9785 11415] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: cccb1239efcaf36f8df1cb7201000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,396 I 9785 11415] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: cccb1239efcaf36f8df1cb7201000000, ip address: 10.134.61.130, port: 9143, worker_id: 19e22ca39bba3dfa21a8866074c10815e64f473f70dbcafbe2c001ed, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,289 E 9785 9785] actor_manager.cc:93: Actor handle already exists cccb1239efcaf36f8df1cb7201000000 | |
| [2021-04-29 17:04:29,486 I 9785 9785] core_worker.cc:2064: Finished executing task 8b781b5bc1b54e92ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-5a98ec745297ecb05a70389a4412c17f58190b637acd3dd3a8bc006e_5218.log <== | |
| [2021-04-29 17:04:01,948 E 5218 5218] actor_manager.cc:93: Actor handle already exists 69c25be18189d61fc51298ce01000000 | |
| [2021-04-29 17:04:09,232 I 5218 5218] core_worker.cc:2064: Finished executing task fca2af9a56b4f1abffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:09,243 E 5218 5218] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,243 E 5218 5218] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,377 I 5218 5218] direct_task_transport.cc:47: Creating actor via GCS actor id = : ba5ade614c3fe6c792dab79101000000 | |
| [2021-04-29 17:04:09,421 I 5218 5956] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: ba5ade614c3fe6c792dab79101000000, ip address: 10.134.142.16, port: 9191, worker_id: 8f1fe1da1fd676ba0e5a1fdd6579c81bbe5a7384873455d101062e9d, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,349 I 5218 5956] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: ba5ade614c3fe6c792dab79101000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,401 I 5218 5956] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: ba5ade614c3fe6c792dab79101000000, ip address: 10.134.142.16, port: 9191, worker_id: 8f1fe1da1fd676ba0e5a1fdd6579c81bbe5a7384873455d101062e9d, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,560 E 5218 5218] actor_manager.cc:93: Actor handle already exists ba5ade614c3fe6c792dab79101000000 | |
| [2021-04-29 17:04:29,537 I 5218 5218] core_worker.cc:2064: Finished executing task b59cff17a7b79abbffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-5aea30b943f2d09f8d73054bd56c347d5346dd58970fa89b19287963_9788.log <== | |
| [2021-04-29 17:04:02,440 I 9788 11120] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,457 E 9788 9788] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:02,465 I 9788 11120] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,436 E 9788 9788] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,534 I 9788 9788] direct_task_transport.cc:47: Creating actor via GCS actor id = : dc4bb32bad953a10ee81ba4601000000 | |
| [2021-04-29 17:04:05,542 I 9788 11120] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: dc4bb32bad953a10ee81ba4601000000, ip address: 10.133.95.40, port: 9110, worker_id: bbc43f65de646dce16fd65083f051e0466efca7a0c4fdc991f68167a, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:07,511 I 9788 11120] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: dc4bb32bad953a10ee81ba4601000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,394 I 9788 11120] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: dc4bb32bad953a10ee81ba4601000000, ip address: 10.133.95.40, port: 9110, worker_id: bbc43f65de646dce16fd65083f051e0466efca7a0c4fdc991f68167a, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,775 E 9788 9788] actor_manager.cc:93: Actor handle already exists dc4bb32bad953a10ee81ba4601000000 | |
| [2021-04-29 17:04:29,554 I 9788 9788] core_worker.cc:2064: Finished executing task a738bde4febdbb6affffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-5bb72d84173c511f24f819d50603a8dc41f8212c648e0c3f148ec24d_7579.log <== | |
| [2021-04-29 17:03:58,880 I 7579 9005] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,910 E 7579 7579] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,916 I 7579 9005] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,603 E 7579 7579] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,662 I 7579 7579] direct_task_transport.cc:47: Creating actor via GCS actor id = : 382c14f3844ea0344393a80601000000 | |
| [2021-04-29 17:03:59,672 I 7579 9005] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 382c14f3844ea0344393a80601000000, ip address: 10.134.60.23, port: 9131, worker_id: f178d761610792d81c12316854121041275435a935542f8063813f73, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,718 I 7579 9005] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 382c14f3844ea0344393a80601000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:18,579 I 7579 9005] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 382c14f3844ea0344393a80601000000, ip address: 10.134.60.23, port: 9131, worker_id: f178d761610792d81c12316854121041275435a935542f8063813f73, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:21,655 E 7579 7579] actor_manager.cc:93: Actor handle already exists 382c14f3844ea0344393a80601000000 | |
| [2021-04-29 17:04:28,800 I 7579 7579] core_worker.cc:2064: Finished executing task 6b9df2abd64be432ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-5bd785e80b4f171479bf45476e08234e5dd663c3cf6fc246177de75c_14413.log <== | |
| [2021-04-29 17:04:05,946 I 14413 15817] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,011 E 14413 14413] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:06,025 I 14413 15817] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,785 E 14413 14413] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:08,961 I 14413 14413] direct_task_transport.cc:47: Creating actor via GCS actor id = : 608d9a944a81e06c3190f55901000000 | |
| [2021-04-29 17:04:08,966 I 14413 15817] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 608d9a944a81e06c3190f55901000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:20,492 I 14413 15817] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 608d9a944a81e06c3190f55901000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:22,929 I 14413 15817] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 608d9a944a81e06c3190f55901000000, ip address: 10.134.142.44, port: 9211, worker_id: fd93ade0e1ade762c0f58f2af392c4bbfc1b992ba444cade8fb26e7b, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:25,735 E 14413 14413] actor_manager.cc:93: Actor handle already exists 608d9a944a81e06c3190f55901000000 | |
| [2021-04-29 17:04:29,766 I 14413 14413] core_worker.cc:2064: Finished executing task ca1a06959c59c9f6ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-5cf8ee2437f95e565ff0dcecfecfc408b150c80452e223a986a72a7f_4125.log <== | |
| [2021-04-29 18:18:11,980 I 4125 4125] core_worker.cc:2064: Finished executing task 57f2bef385220f689b43df895dbca74cebf7c1e401000000, status=OK | |
| [2021-04-29 18:18:12,018 I 4125 5011] reference_count.cc:955: Tried to remove an object location for an object 57f2bef385220f689b43df895dbca74cebf7c1e40100000004000000 that doesn't exist in the reference table | |
| [2021-04-29 18:18:12,018 I 4125 5011] reference_count.cc:955: Tried to remove an object location for an object 57f2bef385220f689b43df895dbca74cebf7c1e40100000003000000 that doesn't exist in the reference table | |
| [2021-04-29 18:18:12,019 I 4125 5011] reference_count.cc:955: Tried to remove an object location for an object 57f2bef385220f689b43df895dbca74cebf7c1e40100000003000000 that doesn't exist in the reference table | |
| [2021-04-29 18:18:12,019 I 4125 5011] reference_count.cc:955: Tried to remove an object location for an object 57f2bef385220f689b43df895dbca74cebf7c1e40100000004000000 that doesn't exist in the reference table | |
| [2021-04-29 18:18:18,965 I 4125 4125] core_worker.cc:2064: Finished executing task c72a716a03ad38389b43df895dbca74cebf7c1e401000000, status=OK | |
| [2021-04-29 18:18:19,026 I 4125 5011] reference_count.cc:955: Tried to remove an object location for an object c72a716a03ad38389b43df895dbca74cebf7c1e40100000004000000 that doesn't exist in the reference table | |
| [2021-04-29 18:18:19,027 I 4125 5011] reference_count.cc:955: Tried to remove an object location for an object c72a716a03ad38389b43df895dbca74cebf7c1e40100000003000000 that doesn't exist in the reference table | |
| [2021-04-29 18:18:19,027 I 4125 5011] reference_count.cc:955: Tried to remove an object location for an object c72a716a03ad38389b43df895dbca74cebf7c1e40100000004000000 that doesn't exist in the reference table | |
| [2021-04-29 18:18:19,027 I 4125 5011] reference_count.cc:955: Tried to remove an object location for an object c72a716a03ad38389b43df895dbca74cebf7c1e40100000003000000 that doesn't exist in the reference table | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-5d4831d8c9dff8619652d6751a665ae710a7c089f6d4277d4b1e1413_15910.log <== | |
| [2021-04-29 17:04:07,995 I 15910 16981] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,268 E 15910 15910] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,330 I 15910 16981] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,965 E 15910 15910] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,059 I 15910 15910] direct_task_transport.cc:47: Creating actor via GCS actor id = : 4099fa4658e0c5393a76da4e01000000 | |
| [2021-04-29 17:04:11,102 I 15910 16981] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 4099fa4658e0c5393a76da4e01000000, ip address: 10.134.142.16, port: 9198, worker_id: f80efe26c500784cb590d5c5bd6c52a48e761aa292a7de3728834903, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,370 I 15910 16981] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 4099fa4658e0c5393a76da4e01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,153 I 15910 16981] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 4099fa4658e0c5393a76da4e01000000, ip address: 10.134.142.16, port: 9198, worker_id: f80efe26c500784cb590d5c5bd6c52a48e761aa292a7de3728834903, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:26,998 E 15910 15910] actor_manager.cc:93: Actor handle already exists 4099fa4658e0c5393a76da4e01000000 | |
| [2021-04-29 17:04:29,781 I 15910 15910] core_worker.cc:2064: Finished executing task d8b7a9b9278793a1ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-5d90a3ea3e5d66ac12bebb07b9c3783bb29793c1a8dafe7825c2a71a_3996.log <== | |
| [2021-04-29 17:03:44,987 I 3996 3996] core_worker.cc:2064: Finished executing task fb8857645cc3a393cb6d5272554fce6f425869e101000000, status=OK | |
| [2021-04-29 17:04:48,203 I 3996 3996] core_worker.cc:2064: Finished executing task b14974dedc713d6dcb6d5272554fce6f425869e101000000, status=OK | |
| [2021-04-29 17:04:50,447 I 3996 3996] core_worker.cc:2064: Finished executing task bd85e233103a2823cb6d5272554fce6f425869e101000000, status=OK | |
| [2021-04-29 17:04:51,202 I 3996 3996] core_worker.cc:2064: Finished executing task c0d7d0ad38d8b39ccb6d5272554fce6f425869e101000000, status=OK | |
| [2021-04-29 17:04:52,042 I 3996 3996] core_worker.cc:2064: Finished executing task 5b9d0e5769c9ed06cb6d5272554fce6f425869e101000000, status=OK | |
| [2021-04-29 17:04:52,287 I 3996 3996] core_worker.cc:2064: Finished executing task 97f4598ddc4e979ccb6d5272554fce6f425869e101000000, status=OK | |
| [2021-04-29 17:04:54,669 I 3996 3996] core_worker.cc:2064: Finished executing task eb4551c009bc5daacb6d5272554fce6f425869e101000000, status=OK | |
| [2021-04-29 17:04:55,619 I 3996 3996] core_worker.cc:2064: Finished executing task fe694587b3c755e5cb6d5272554fce6f425869e101000000, status=OK | |
| [2021-04-29 17:04:55,941 I 3996 3996] core_worker.cc:2064: Finished executing task 6b1385fa40c4dba8cb6d5272554fce6f425869e101000000, status=OK | |
| [2021-04-29 17:04:57,786 I 3996 3996] core_worker.cc:2064: Finished executing task 1963d70e157ecd8acb6d5272554fce6f425869e101000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-5db881d0b4f463f6f5530f0ff30245b34b42eac2d46e8b5c7c6dd7e8_4043.log <== | |
| [2021-04-29 17:03:58,821 E 4043 4043] actor_manager.cc:93: Actor handle already exists e8297d3faad638320e56fb8501000000 | |
| [2021-04-29 17:04:00,452 I 4043 4043] core_worker.cc:2064: Finished executing task d31a41b20f6ab417ffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:00,489 E 4043 4043] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:00,490 E 4043 4043] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:00,572 I 4043 4043] direct_task_transport.cc:47: Creating actor via GCS actor id = : 1b617c0b4f933dce68634c8901000000 | |
| [2021-04-29 17:04:00,587 I 4043 4843] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 1b617c0b4f933dce68634c8901000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,733 I 4043 4843] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 1b617c0b4f933dce68634c8901000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,376 I 4043 4843] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1b617c0b4f933dce68634c8901000000, ip address: 10.134.142.16, port: 9122, worker_id: 1417b7ba7db80b715ce6e401d52fe9eaf68e749b23f8e68b7e8a7808, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,995 E 4043 4043] actor_manager.cc:93: Actor handle already exists 1b617c0b4f933dce68634c8901000000 | |
| [2021-04-29 17:04:29,433 I 4043 4043] core_worker.cc:2064: Finished executing task 7a84727c40f8ea76ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-5e557b43dfadf56048d92dc98db52d49959ba3287b59c01e933a4fb5_15932.log <== | |
| [2021-04-29 17:04:08,065 I 15932 16958] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,268 E 15932 15932] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,330 I 15932 16958] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,964 E 15932 15932] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,059 I 15932 15932] direct_task_transport.cc:47: Creating actor via GCS actor id = : c6b87179c86c9179eb50156101000000 | |
| [2021-04-29 17:04:11,102 I 15932 16958] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: c6b87179c86c9179eb50156101000000, ip address: 10.133.95.40, port: 9198, worker_id: d7d69a2571dc6e1fc266e76cb8872b18c80b5ac7f5bb99a50c5e6852, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,371 I 15932 16958] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: c6b87179c86c9179eb50156101000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,074 I 15932 16958] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: c6b87179c86c9179eb50156101000000, ip address: 10.133.95.40, port: 9198, worker_id: d7d69a2571dc6e1fc266e76cb8872b18c80b5ac7f5bb99a50c5e6852, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:26,965 E 15932 15932] actor_manager.cc:93: Actor handle already exists c6b87179c86c9179eb50156101000000 | |
| [2021-04-29 17:04:29,780 I 15932 15932] core_worker.cc:2064: Finished executing task 3e3e893bca0798edffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-5ed402006767df50e9fb70a0535ae9597e684cca915ae2bae52b7977_7506.log <== | |
| [2021-04-29 17:03:59,238 I 7506 8756] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,256 E 7506 7506] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,262 I 7506 8756] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,686 E 7506 7506] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,757 I 7506 7506] direct_task_transport.cc:47: Creating actor via GCS actor id = : 9f14aa5781c9c5266341a1a401000000 | |
| [2021-04-29 17:03:59,775 I 7506 8756] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 9f14aa5781c9c5266341a1a401000000, ip address: 10.134.142.44, port: 9118, worker_id: 65839dc24f9ff94e54a2f1902c17cad53a1b921b07b393187a5d7aa0, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,724 I 7506 8756] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 9f14aa5781c9c5266341a1a401000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:19,263 I 7506 8756] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 9f14aa5781c9c5266341a1a401000000, ip address: 10.134.142.44, port: 9118, worker_id: 65839dc24f9ff94e54a2f1902c17cad53a1b921b07b393187a5d7aa0, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,060 E 7506 7506] actor_manager.cc:93: Actor handle already exists 9f14aa5781c9c5266341a1a401000000 | |
| [2021-04-29 17:04:28,913 I 7506 7506] core_worker.cc:2064: Finished executing task 9b495a4c16eda0cdffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-5ee315d05588f87b566de33ac9bc70a81fe1efb1beb4f02d32b383a9_10113.log <== | |
| [2021-04-29 17:04:01,909 I 10113 11813] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,931 E 10113 10113] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,945 I 10113 11813] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,477 E 10113 10113] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:04,550 I 10113 10113] direct_task_transport.cc:47: Creating actor via GCS actor id = : bc6bd86c88acfc709f35b53b01000000 | |
| [2021-04-29 17:04:04,558 I 10113 11813] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: bc6bd86c88acfc709f35b53b01000000, ip address: 10.133.95.40, port: 9114, worker_id: f322971152640c296baf133e85b4c54f04be7a1f8658848cea914092, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,064 I 10113 11813] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: bc6bd86c88acfc709f35b53b01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,392 I 10113 11813] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: bc6bd86c88acfc709f35b53b01000000, ip address: 10.133.95.40, port: 9114, worker_id: f322971152640c296baf133e85b4c54f04be7a1f8658848cea914092, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,140 E 10113 10113] actor_manager.cc:93: Actor handle already exists bc6bd86c88acfc709f35b53b01000000 | |
| [2021-04-29 17:04:29,471 I 10113 10113] core_worker.cc:2064: Finished executing task d35b7819c83fbe44ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-5fe5d9d602441535bec4f941db24e7879b07ee3ed591487e13a03e67_15936.log <== | |
| [2021-04-29 17:04:08,146 I 15936 17153] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,308 E 15936 15936] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,414 I 15936 17153] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,999 E 15936 15936] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,093 I 15936 15936] direct_task_transport.cc:47: Creating actor via GCS actor id = : 7bb12b392d5cab4bcf3d04cb01000000 | |
| [2021-04-29 17:04:11,138 I 15936 17153] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 7bb12b392d5cab4bcf3d04cb01000000, ip address: 10.134.70.227, port: 9211, worker_id: 632e081c621957250691ff6c945e530b96be6a07925ebe197a8f9176, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,372 I 15936 17153] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 7bb12b392d5cab4bcf3d04cb01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:25,213 I 15936 17153] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 7bb12b392d5cab4bcf3d04cb01000000, ip address: 10.134.70.227, port: 9211, worker_id: 632e081c621957250691ff6c945e530b96be6a07925ebe197a8f9176, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,193 E 15936 15936] actor_manager.cc:93: Actor handle already exists 7bb12b392d5cab4bcf3d04cb01000000 | |
| [2021-04-29 17:04:29,823 I 15936 15936] core_worker.cc:2064: Finished executing task cc69369f92f34ac7ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-600141f3aca774b6d3220b186c21b1ed284f232548583844b694ac9f_5260.log <== | |
| [2021-04-29 17:03:58,261 E 5260 5260] actor_manager.cc:93: Actor handle already exists 2b9286d835a644e2f31759f701000000 | |
| [2021-04-29 17:03:59,747 I 5260 5260] core_worker.cc:2064: Finished executing task e7b91489f940addcffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:03:59,800 E 5260 5260] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,800 E 5260 5260] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,862 I 5260 5260] direct_task_transport.cc:47: Creating actor via GCS actor id = : 89c07cb42c2acaec6e20930901000000 | |
| [2021-04-29 17:03:59,875 I 5260 6227] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 89c07cb42c2acaec6e20930901000000, ip address: 10.134.142.44, port: 9117, worker_id: 2a3a858dcf5986c4df2cf15dbeea0fa089e415ea8d8e6501807a6c26, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,727 I 5260 6227] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 89c07cb42c2acaec6e20930901000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,360 I 5260 6227] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 89c07cb42c2acaec6e20930901000000, ip address: 10.134.142.44, port: 9117, worker_id: 2a3a858dcf5986c4df2cf15dbeea0fa089e415ea8d8e6501807a6c26, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,887 E 5260 5260] actor_manager.cc:93: Actor handle already exists 89c07cb42c2acaec6e20930901000000 | |
| [2021-04-29 17:04:29,409 I 5260 5260] core_worker.cc:2064: Finished executing task 82ba9781159a061affffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-620c3211da26d098a7dca01b7b8931ce4062e35c1489d48c11b9434f_7574.log <== | |
| [2021-04-29 17:03:58,389 I 7574 9215] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,390 E 7574 7574] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,391 I 7574 9215] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,755 E 7574 7574] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:58,868 I 7574 7574] direct_task_transport.cc:47: Creating actor via GCS actor id = : fcd15ec230134af200c72cee01000000 | |
| [2021-04-29 17:03:58,879 I 7574 9215] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: fcd15ec230134af200c72cee01000000, ip address: 10.134.61.130, port: 9125, worker_id: 9b3d1cdaa1024ad3ddd094c76005b8b7f026c4040acf7a8dfde80d8f, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:00,269 I 7574 9215] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: fcd15ec230134af200c72cee01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,172 I 7574 9215] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: fcd15ec230134af200c72cee01000000, ip address: 10.134.61.130, port: 9125, worker_id: 9b3d1cdaa1024ad3ddd094c76005b8b7f026c4040acf7a8dfde80d8f, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:07,309 E 7574 7574] actor_manager.cc:93: Actor handle already exists fcd15ec230134af200c72cee01000000 | |
| [2021-04-29 17:04:12,630 I 7574 7574] core_worker.cc:2064: Finished executing task c6bba8c6c24a9357ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-62d68a171ed0d2af1562e75e11498e2d9920c0a69b58189bc6a674a0_15934.log <== | |
| [2021-04-29 17:04:08,065 I 15934 16988] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,268 E 15934 15934] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,330 I 15934 16988] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,964 E 15934 15934] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,059 I 15934 15934] direct_task_transport.cc:47: Creating actor via GCS actor id = : cfcfb33ddfedde7dd239f62401000000 | |
| [2021-04-29 17:04:11,102 I 15934 16988] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: cfcfb33ddfedde7dd239f62401000000, ip address: 10.134.61.25, port: 9193, worker_id: 77ceefa8df6662a2c3777da89ce53abed30d80b2e3f9a8407b3e2cd7, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,370 I 15934 16988] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: cfcfb33ddfedde7dd239f62401000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,123 I 15934 16988] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: cfcfb33ddfedde7dd239f62401000000, ip address: 10.134.61.25, port: 9193, worker_id: 77ceefa8df6662a2c3777da89ce53abed30d80b2e3f9a8407b3e2cd7, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:26,992 E 15934 15934] actor_manager.cc:93: Actor handle already exists cfcfb33ddfedde7dd239f62401000000 | |
| [2021-04-29 17:04:29,778 I 15934 15934] core_worker.cc:2064: Finished executing task 5da7ede874d3f1dcffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-62fb7f2ef35ed10cf981e7e97f1de6cbca207cd9047036470c61e908_7565.log <== | |
| [2021-04-29 17:03:58,804 I 7565 9088] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,840 E 7565 7565] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,847 I 7565 9088] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,556 E 7565 7565] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,601 I 7565 7565] direct_task_transport.cc:47: Creating actor via GCS actor id = : e5e3522456481fc0b8d2cd7a01000000 | |
| [2021-04-29 17:03:59,608 I 7565 9088] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: e5e3522456481fc0b8d2cd7a01000000, ip address: 10.134.112.222, port: 9131, worker_id: 54f7e6704abeaf9b24a835cf8d1d74f8a1602f20138f27a7ad7c5996, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,556 I 7565 9088] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: e5e3522456481fc0b8d2cd7a01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:16,597 I 7565 9088] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: e5e3522456481fc0b8d2cd7a01000000, ip address: 10.134.112.222, port: 9131, worker_id: 54f7e6704abeaf9b24a835cf8d1d74f8a1602f20138f27a7ad7c5996, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:17,859 E 7565 7565] actor_manager.cc:93: Actor handle already exists e5e3522456481fc0b8d2cd7a01000000 | |
| [2021-04-29 17:04:25,749 I 7565 7565] core_worker.cc:2064: Finished executing task f8275180d32f9265ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-633d93406cf381ef0a9725b17fb65aa3b6b37cb958686f842933bef9_5312.log <== | |
| [2021-04-29 17:03:46,423 I 5312 5312] core_worker.cc:2064: Finished executing task a0b89669cc11382daf2fda9d4fb1897251f5dab101000000, status=OK | |
| [2021-04-29 17:04:41,858 I 5312 5312] core_worker.cc:2064: Finished executing task f1f5d1c489218c24af2fda9d4fb1897251f5dab101000000, status=OK | |
| [2021-04-29 17:04:49,674 I 5312 5312] core_worker.cc:2064: Finished executing task 1d452ae509905040af2fda9d4fb1897251f5dab101000000, status=OK | |
| [2021-04-29 17:04:50,201 I 5312 5312] core_worker.cc:2064: Finished executing task a23a143364d09a54af2fda9d4fb1897251f5dab101000000, status=OK | |
| [2021-04-29 17:04:51,429 I 5312 5312] core_worker.cc:2064: Finished executing task f74e0f1d2d29d76daf2fda9d4fb1897251f5dab101000000, status=OK | |
| [2021-04-29 17:04:52,426 I 5312 5312] core_worker.cc:2064: Finished executing task 6ca548df96f9ec7eaf2fda9d4fb1897251f5dab101000000, status=OK | |
| [2021-04-29 17:04:54,659 I 5312 5312] core_worker.cc:2064: Finished executing task 85a9f253e1a14e34af2fda9d4fb1897251f5dab101000000, status=OK | |
| [2021-04-29 17:04:55,501 I 5312 5312] core_worker.cc:2064: Finished executing task e0883cf2e52558d3af2fda9d4fb1897251f5dab101000000, status=OK | |
| [2021-04-29 17:04:57,738 I 5312 5312] core_worker.cc:2064: Finished executing task 2b1dfaa3a9e735baaf2fda9d4fb1897251f5dab101000000, status=OK | |
| [2021-04-29 17:04:58,135 I 5312 5312] core_worker.cc:2064: Finished executing task 652597ffa7ff9dc4af2fda9d4fb1897251f5dab101000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-6343c537bca014238de15d870822449c17280856c5d11d56183ff679_5263.log <== | |
| [2021-04-29 17:03:59,661 E 5263 5263] actor_manager.cc:93: Actor handle already exists f0c5889db61c1cd44160ae1501000000 | |
| [2021-04-29 17:04:02,023 I 5263 5263] core_worker.cc:2064: Finished executing task 98852cf3716c8d02ffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:02,032 E 5263 5263] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:02,032 E 5263 5263] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:02,090 I 5263 5263] direct_task_transport.cc:47: Creating actor via GCS actor id = : 964ada869294c67d03caab7801000000 | |
| [2021-04-29 17:04:02,097 I 5263 6093] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 964ada869294c67d03caab7801000000, ip address: 10.134.70.227, port: 9104, worker_id: 41f71504af0bee4a5a513e45053feb222a6be25c16141fd78165fac5, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,544 I 5263 6093] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 964ada869294c67d03caab7801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,391 I 5263 6093] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 964ada869294c67d03caab7801000000, ip address: 10.134.70.227, port: 9104, worker_id: 41f71504af0bee4a5a513e45053feb222a6be25c16141fd78165fac5, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,107 E 5263 5263] actor_manager.cc:93: Actor handle already exists 964ada869294c67d03caab7801000000 | |
| [2021-04-29 17:04:29,461 I 5263 5263] core_worker.cc:2064: Finished executing task 45851c073a67d86affffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-642996b9c5e35264bece6f91c0b694b20611eb4b310247129696fce8_5223.log <== | |
| [2021-04-29 17:04:02,479 E 5223 5223] actor_manager.cc:93: Actor handle already exists 7df6a600ffb88a9ef191a96101000000 | |
| [2021-04-29 17:04:10,308 I 5223 5223] core_worker.cc:2064: Finished executing task 7ddd3330d244cafcffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:10,330 E 5223 5223] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,331 E 5223 5223] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,449 I 5223 5223] direct_task_transport.cc:47: Creating actor via GCS actor id = : 53e10432a9d8a77f66a3e5f701000000 | |
| [2021-04-29 17:04:10,484 I 5223 5763] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 53e10432a9d8a77f66a3e5f701000000, ip address: 10.134.61.25, port: 9204, worker_id: fd3e6205d0b1f126a6bc4fd66ce713c16a5fb3a828f6dfe6131e482e, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,366 I 5223 5763] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 53e10432a9d8a77f66a3e5f701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:22,814 I 5223 5763] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 53e10432a9d8a77f66a3e5f701000000, ip address: 10.134.61.25, port: 9204, worker_id: fd3e6205d0b1f126a6bc4fd66ce713c16a5fb3a828f6dfe6131e482e, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:25,485 E 5223 5223] actor_manager.cc:93: Actor handle already exists 53e10432a9d8a77f66a3e5f701000000 | |
| [2021-04-29 17:04:29,746 I 5223 5223] core_worker.cc:2064: Finished executing task 09070a9bd848d3f8ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-642afc9d6b2d05ce6e73af5755f8dac61a5f93afc8c1757de4441950_5307.log <== | |
| [2021-04-29 17:03:58,296 E 5307 5307] actor_manager.cc:93: Actor handle already exists ed22adf55bbb0b76bde70ec601000000 | |
| [2021-04-29 17:03:59,753 I 5307 5307] core_worker.cc:2064: Finished executing task 6d6c9367d967367cffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:03:59,769 E 5307 5307] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,770 E 5307 5307] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,840 I 5307 5307] direct_task_transport.cc:47: Creating actor via GCS actor id = : 895494ecf53c9cb4f7b461ad01000000 | |
| [2021-04-29 17:03:59,845 I 5307 6257] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 895494ecf53c9cb4f7b461ad01000000, ip address: 10.134.142.16, port: 9126, worker_id: fa313ac5185e2c75c0a4e75b9895b814e45d24424ebf7f1187cc674a, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,727 I 5307 6257] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 895494ecf53c9cb4f7b461ad01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,355 I 5307 6257] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 895494ecf53c9cb4f7b461ad01000000, ip address: 10.134.142.16, port: 9126, worker_id: fa313ac5185e2c75c0a4e75b9895b814e45d24424ebf7f1187cc674a, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,882 E 5307 5307] actor_manager.cc:93: Actor handle already exists 895494ecf53c9cb4f7b461ad01000000 | |
| [2021-04-29 17:04:29,404 I 5307 5307] core_worker.cc:2064: Finished executing task 1e4494588a06d613ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-64ac49fecae673043308eb2fcca0e73298224a7a1d468ec32f64a9ff_13985.log <== | |
| [2021-04-29 17:04:05,690 I 13985 15298] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,717 E 13985 13985] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,795 I 13985 15298] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,238 E 13985 13985] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:08,415 I 13985 13985] direct_task_transport.cc:47: Creating actor via GCS actor id = : dbf850f85a4c866e1cb6456801000000 | |
| [2021-04-29 17:04:08,444 I 13985 15298] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: dbf850f85a4c866e1cb6456801000000, ip address: 10.134.142.16, port: 22629, worker_id: NIL_ID, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:19,187 I 13985 15298] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: dbf850f85a4c866e1cb6456801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,400 I 13985 15298] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: dbf850f85a4c866e1cb6456801000000, ip address: 10.134.142.16, port: 9162, worker_id: 08ede9f6dbca18eeed4c1e23b53adac41cdc0d2967283181db95ef61, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,484 E 13985 13985] actor_manager.cc:93: Actor handle already exists dbf850f85a4c866e1cb6456801000000 | |
| [2021-04-29 17:04:29,529 I 13985 13985] core_worker.cc:2064: Finished executing task 2ac3ad779460cfe0ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-64f5e3f287f97c1a9bff5c0f3c3eb9f386ba47cd682622801847a48b_5222.log <== | |
| [2021-04-29 17:04:01,921 E 5222 5222] actor_manager.cc:93: Actor handle already exists 45e7d5f5b655b01edc64fe1c01000000 | |
| [2021-04-29 17:04:09,214 I 5222 5222] core_worker.cc:2064: Finished executing task 902f200819b37226ffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:09,226 E 5222 5222] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,226 E 5222 5222] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,339 I 5222 5222] direct_task_transport.cc:47: Creating actor via GCS actor id = : fb244326a70ec8c578e736ae01000000 | |
| [2021-04-29 17:04:09,380 I 5222 6008] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: fb244326a70ec8c578e736ae01000000, ip address: 10.134.112.222, port: 9190, worker_id: 912d6bbece5c4fec066abe262dea0ccaad17c3bc38017dff3c30abae, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,348 I 5222 6008] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: fb244326a70ec8c578e736ae01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,401 I 5222 6008] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: fb244326a70ec8c578e736ae01000000, ip address: 10.134.112.222, port: 9190, worker_id: 912d6bbece5c4fec066abe262dea0ccaad17c3bc38017dff3c30abae, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,711 E 5222 5222] actor_manager.cc:93: Actor handle already exists fb244326a70ec8c578e736ae01000000 | |
| [2021-04-29 17:04:29,537 I 5222 5222] core_worker.cc:2064: Finished executing task 8701bcee6a3200a5ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-65495eb820db9f953bb2412d0e9e6112c48a8e24276672f9bf076d35_17361.log <== | |
| [2021-04-29 17:04:08,965 I 17361 18911] service_based_accessor.cc:579: Received notification for node id = ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, IsAlive = 1 | |
| [2021-04-29 17:04:08,965 I 17361 18911] service_based_accessor.cc:579: Received notification for node id = d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, IsAlive = 1 | |
| [2021-04-29 17:04:08,965 I 17361 18911] service_based_accessor.cc:579: Received notification for node id = b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, IsAlive = 1 | |
| [2021-04-29 17:04:08,966 I 17361 18911] service_based_accessor.cc:579: Received notification for node id = 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, IsAlive = 1 | |
| [2021-04-29 17:04:08,966 I 17361 18911] service_based_accessor.cc:579: Received notification for node id = 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, IsAlive = 1 | |
| [2021-04-29 17:04:10,129 I 17361 18911] core_worker.cc:717: Exit signal received, this process will exit after all outstanding tasks have finished, exit_type=INTENDED_EXIT | |
| [2021-04-29 17:04:10,129 I 17361 17361] raylet_client.cc:129: RayletClient::Disconnect, exit_type=INTENDED_EXIT, has creation_task_exception_pb_bytes=0 | |
| [2021-04-29 17:04:10,129 I 17361 17361] core_worker.cc:308: Removed worker 65495eb820db9f953bb2412d0e9e6112c48a8e24276672f9bf076d35 | |
| [2021-04-29 17:04:10,134 I 17361 17361] core_worker.cc:181: Destructing CoreWorkerProcess. pid: 17361 | |
| [2021-04-29 17:04:10,134 I 17361 17361] io_service_pool.cc:47: IOServicePool is stopped. | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-65f29259657d917e294d2e846fa90a7f39fe8a8b8ad21c56787630ba_13854.log <== | |
| [2021-04-29 17:04:05,703 I 13854 15284] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,729 E 13854 13854] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,795 I 13854 15284] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,237 E 13854 13854] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:08,428 I 13854 13854] direct_task_transport.cc:47: Creating actor via GCS actor id = : 71088ff5b936128df2fc241e01000000 | |
| [2021-04-29 17:04:08,444 I 13854 15284] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 71088ff5b936128df2fc241e01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:19,247 I 13854 15284] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 71088ff5b936128df2fc241e01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:22,632 I 13854 15284] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 71088ff5b936128df2fc241e01000000, ip address: 10.134.142.44, port: 9225, worker_id: 42d50c924731a7dde2911b1af571b69807b77911e20ad683069445a8, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:25,405 E 13854 13854] actor_manager.cc:93: Actor handle already exists 71088ff5b936128df2fc241e01000000 | |
| [2021-04-29 17:04:29,727 I 13854 13854] core_worker.cc:2064: Finished executing task 83feeb313407297fffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-669874e7bcadaa273b64758ec465ac0e6422bdad34660aa209e2528b_10097.log <== | |
| [2021-04-29 17:04:01,695 I 10097 11830] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,704 E 10097 10097] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,707 I 10097 11830] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,288 E 10097 10097] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:02,339 I 10097 10097] direct_task_transport.cc:47: Creating actor via GCS actor id = : 1ce376c48e74f0029070399001000000 | |
| [2021-04-29 17:04:02,348 I 10097 11830] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 1ce376c48e74f0029070399001000000, ip address: 10.134.70.227, port: 9102, worker_id: 71f83d2a015966d5687a367281e8c210811f158ab6f96303635a4160, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,547 I 10097 11830] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 1ce376c48e74f0029070399001000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,391 I 10097 11830] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ce376c48e74f0029070399001000000, ip address: 10.134.70.227, port: 9102, worker_id: 71f83d2a015966d5687a367281e8c210811f158ab6f96303635a4160, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,745 E 10097 10097] actor_manager.cc:93: Actor handle already exists 1ce376c48e74f0029070399001000000 | |
| [2021-04-29 17:04:29,540 I 10097 10097] core_worker.cc:2064: Finished executing task 608f4b8a6b865ee5ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-66fb4cdc32285c8594cfa989025deee157047bd4192b729b1e04110e_12784.log <== | |
| [2021-04-29 17:04:04,838 I 12784 13647] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,882 E 12784 12784] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:04,913 I 12784 13647] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,804 E 12784 12784] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:06,918 I 12784 12784] direct_task_transport.cc:47: Creating actor via GCS actor id = : 0559656c46d59a6e1637c15301000000 | |
| [2021-04-29 17:04:06,987 I 12784 13647] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 0559656c46d59a6e1637c15301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:14,329 I 12784 13647] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 0559656c46d59a6e1637c15301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,398 I 12784 13647] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 0559656c46d59a6e1637c15301000000, ip address: 10.134.60.23, port: 9152, worker_id: 80e6a7460b3ce1b9ef7a7d24565ec95efb2009acc1fb3bed882e4cdb, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,385 E 12784 12784] actor_manager.cc:93: Actor handle already exists 0559656c46d59a6e1637c15301000000 | |
| [2021-04-29 17:04:29,520 I 12784 12784] core_worker.cc:2064: Finished executing task e408d424476c9d35ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-670d7fa0d232dd3eabbd808c656998c22e96ebf378a81bff04e8f5bf_5230.log <== | |
| [2021-04-29 17:04:01,978 E 5230 5230] actor_manager.cc:93: Actor handle already exists 96ee1b25b203b06524a5a87301000000 | |
| [2021-04-29 17:04:09,254 I 5230 5230] core_worker.cc:2064: Finished executing task d1ad63ad323bbedfffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:09,296 E 5230 5230] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,297 E 5230 5230] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,417 I 5230 5230] direct_task_transport.cc:47: Creating actor via GCS actor id = : 3e799ba14843122a36aed82601000000 | |
| [2021-04-29 17:04:09,461 I 5230 5972] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 3e799ba14843122a36aed82601000000, ip address: 10.134.70.227, port: 9221, worker_id: a4fa4f116634b35893b693253c5156dcad61ea90c4b483b6f25864aa, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,351 I 5230 5972] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 3e799ba14843122a36aed82601000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,266 I 5230 5972] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 3e799ba14843122a36aed82601000000, ip address: 10.134.70.227, port: 9221, worker_id: a4fa4f116634b35893b693253c5156dcad61ea90c4b483b6f25864aa, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:27,078 E 5230 5230] actor_manager.cc:93: Actor handle already exists 3e799ba14843122a36aed82601000000 | |
| [2021-04-29 17:04:29,790 I 5230 5230] core_worker.cc:2064: Finished executing task caf8faaa0bbf59ffffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-673fb5d338b8da374dd75fb4eaaa7addab52bfd9042d866e24663711_4000.log <== | |
| [2021-04-29 17:03:59,732 E 4000 4000] actor_manager.cc:93: Actor handle already exists 8d80ba6871d2275593da8b2a01000000 | |
| [2021-04-29 17:04:02,209 I 4000 4000] core_worker.cc:2064: Finished executing task 05a861001434384dffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:02,221 E 4000 4000] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:02,222 E 4000 4000] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:02,277 I 4000 4000] direct_task_transport.cc:47: Creating actor via GCS actor id = : 6fedd0f225f5178cb3f2f3e701000000 | |
| [2021-04-29 17:04:02,287 I 4000 4292] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 6fedd0f225f5178cb3f2f3e701000000, ip address: 10.134.61.25, port: 9113, worker_id: e318cd938e6be168b391c7c131ab836509d9528296e786b61d8c7ed4, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,546 I 4000 4292] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 6fedd0f225f5178cb3f2f3e701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,391 I 4000 4292] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 6fedd0f225f5178cb3f2f3e701000000, ip address: 10.134.61.25, port: 9113, worker_id: e318cd938e6be168b391c7c131ab836509d9528296e786b61d8c7ed4, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,114 E 4000 4000] actor_manager.cc:93: Actor handle already exists 6fedd0f225f5178cb3f2f3e701000000 | |
| [2021-04-29 17:04:29,459 I 4000 4000] core_worker.cc:2064: Finished executing task 6c65821e117266e9ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-67432fe556c61eb74f2bcb8818e39ab79565d50e5a7bec5bb365df06_7492.log <== | |
| [2021-04-29 17:03:59,480 I 7492 8391] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,500 E 7492 7492] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,505 I 7492 8391] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,825 E 7492 7492] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,910 I 7492 7492] direct_task_transport.cc:47: Creating actor via GCS actor id = : 7dfca4a305352a8e59e6d3c701000000 | |
| [2021-04-29 17:03:59,918 I 7492 8391] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 7dfca4a305352a8e59e6d3c701000000, ip address: 10.134.142.44, port: 12853, worker_id: NIL_ID, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,728 I 7492 8391] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 7dfca4a305352a8e59e6d3c701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,360 I 7492 8391] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 7dfca4a305352a8e59e6d3c701000000, ip address: 10.134.142.44, port: 9115, worker_id: 758e1ae8bdad1404b741531e7d86bf38b488c4f5efb0cb64015b6fd4, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,895 E 7492 7492] actor_manager.cc:93: Actor handle already exists 7dfca4a305352a8e59e6d3c701000000 | |
| [2021-04-29 17:04:29,412 I 7492 7492] core_worker.cc:2064: Finished executing task 6bb671f9a6351efaffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-678e7ccb3aa4a50acd55f686598d57fc5bccabcc79f905074b126e65_7460.log <== | |
| [2021-04-29 17:03:59,473 I 7460 8482] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,488 E 7460 7460] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,493 I 7460 8482] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,824 E 7460 7460] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,895 I 7460 7460] direct_task_transport.cc:47: Creating actor via GCS actor id = : 0a53abb010319117fc9dee7201000000 | |
| [2021-04-29 17:03:59,902 I 7460 8482] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 0a53abb010319117fc9dee7201000000, ip address: 10.134.70.227, port: 9114, worker_id: f274d092182a5db7299aa65f7c915ac5ad9e811d45db6a3c961f13d3, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,728 I 7460 8482] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 0a53abb010319117fc9dee7201000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,360 I 7460 8482] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 0a53abb010319117fc9dee7201000000, ip address: 10.134.70.227, port: 9114, worker_id: f274d092182a5db7299aa65f7c915ac5ad9e811d45db6a3c961f13d3, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,893 E 7460 7460] actor_manager.cc:93: Actor handle already exists 0a53abb010319117fc9dee7201000000 | |
| [2021-04-29 17:04:29,412 I 7460 7460] core_worker.cc:2064: Finished executing task 8212b3e6aa2bc1e1ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-67bc6ca90ae1772e275bbb529398e55d98b0e0b9861e11095d5dc970_7530.log <== | |
| [2021-04-29 17:03:59,178 I 7530 8804] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,197 E 7530 7530] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,204 I 7530 8804] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,662 E 7530 7530] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,718 I 7530 7530] direct_task_transport.cc:47: Creating actor via GCS actor id = : 41212ab2fe3a12a3e73c0c9301000000 | |
| [2021-04-29 17:03:59,729 I 7530 8804] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 41212ab2fe3a12a3e73c0c9301000000, ip address: 10.134.61.130, port: 9117, worker_id: 5f54eaddf8f1a13488e2342dd35a3842f65ff209fdba19f93374cfc8, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,723 I 7530 8804] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 41212ab2fe3a12a3e73c0c9301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:19,263 I 7530 8804] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 41212ab2fe3a12a3e73c0c9301000000, ip address: 10.134.61.130, port: 9117, worker_id: 5f54eaddf8f1a13488e2342dd35a3842f65ff209fdba19f93374cfc8, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,036 E 7530 7530] actor_manager.cc:93: Actor handle already exists 41212ab2fe3a12a3e73c0c9301000000 | |
| [2021-04-29 17:04:28,907 I 7530 7530] core_worker.cc:2064: Finished executing task 10fd1d3d6fe48e7affffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-681bd520193bbff9c790d9bb9c93ffe47a8cfc2575dbcc8bd1234eb9_5207.log <== | |
| [2021-04-29 17:04:10,524 I 5207 5207] direct_task_transport.cc:47: Creating actor via GCS actor id = : 75545e3012b246a8264c979901000000 | |
| [2021-04-29 17:04:10,539 I 5207 5525] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 75545e3012b246a8264c979901000000, ip address: 10.134.142.44, port: 9210, worker_id: 0b28183358a1795ba0f62caa16167f28ef311f699f31e703383efe28, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,366 I 5207 5525] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 75545e3012b246a8264c979901000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,661 I 5207 5525] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 75545e3012b246a8264c979901000000, ip address: 10.134.142.44, port: 9210, worker_id: 0b28183358a1795ba0f62caa16167f28ef311f699f31e703383efe28, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:27,467 E 5207 5207] actor_manager.cc:93: Actor handle already exists 75545e3012b246a8264c979901000000 | |
| [2021-04-29 17:04:29,801 I 5207 5207] core_worker.cc:2064: Finished executing task c19ef12f0c809b23ffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:52:18,463 I 5207 5525] actor_manager.cc:127: received notification on actor, state: DEAD, actor_id: d61ced7ddc924cda5c353b5f01000000, ip address: 10.134.61.25, port: 9140, worker_id: 37307554699c9ff814689169f3e10379e9f581ef19357b0fe38ef162, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:52:18,463 I 5207 5525] actor_manager.cc:141: Creation task formatted exception: , actor_id: d61ced7ddc924cda5c353b5f01000000 | |
| [2021-04-29 17:52:18,463 I 5207 5525] direct_actor_transport.cc:204: Failing pending tasks for actor d61ced7ddc924cda5c353b5f01000000 | |
| [2021-04-29 17:52:18,463 I 5207 5525] direct_actor_transport.cc:222: Failing tasks waiting for death info, size=0, actor_id=d61ced7ddc924cda5c353b5f01000000 | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-68360578a64512857904867196cb4e2752e156285187105e5c53687a_12103.log <== | |
| [2021-04-29 17:04:03,425 I 12103 12597] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,453 E 12103 12103] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:03,463 I 12103 12597] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,872 E 12103 12103] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,943 I 12103 12103] direct_task_transport.cc:47: Creating actor via GCS actor id = : 82eabc087661390781fd830b01000000 | |
| [2021-04-29 17:04:05,968 I 12103 12597] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 82eabc087661390781fd830b01000000, ip address: 10.134.61.130, port: 13217, worker_id: NIL_ID, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,100 I 12103 12597] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 82eabc087661390781fd830b01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,395 I 12103 12597] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 82eabc087661390781fd830b01000000, ip address: 10.134.61.130, port: 9147, worker_id: 94afb4409d50231c7880af67716561357890c3cac1a0821d24a68291, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,269 E 12103 12103] actor_manager.cc:93: Actor handle already exists 82eabc087661390781fd830b01000000 | |
| [2021-04-29 17:04:29,491 I 12103 12103] core_worker.cc:2064: Finished executing task 41f14ec18fe0d0f3ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-6845b111af51a8b7ff259edf9fbabaf70ea6a09c263ab0e214fda473_12842.log <== | |
| [2021-04-29 17:04:04,913 I 12842 13936] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,962 E 12842 12842] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:04,980 I 12842 13936] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,817 E 12842 12842] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:06,964 I 12842 12842] direct_task_transport.cc:47: Creating actor via GCS actor id = : 660a7f9d60eb9ba3e033ac6a01000000 | |
| [2021-04-29 17:04:07,055 I 12842 13936] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 660a7f9d60eb9ba3e033ac6a01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:14,488 I 12842 13936] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 660a7f9d60eb9ba3e033ac6a01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,397 I 12842 13936] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 660a7f9d60eb9ba3e033ac6a01000000, ip address: 10.134.142.44, port: 9229, worker_id: 80f8691178fc57e5c1be2b213e40434cbf918c3c585592974a71b971, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,364 E 12842 12842] actor_manager.cc:93: Actor handle already exists 660a7f9d60eb9ba3e033ac6a01000000 | |
| [2021-04-29 17:04:29,512 I 12842 12842] core_worker.cc:2064: Finished executing task a8cd202143897222ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-68c524b3233817a3421d6e33929f714929f1e437403e9ef3725ce8c8_7452.log <== | |
| [2021-04-29 17:03:58,164 I 7452 8213] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,164 E 7452 7452] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,167 I 7452 8213] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,709 E 7452 7452] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:58,782 I 7452 7452] direct_task_transport.cc:47: Creating actor via GCS actor id = : 4a26861a4740249bb188b21d01000000 | |
| [2021-04-29 17:03:58,787 I 7452 8213] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 4a26861a4740249bb188b21d01000000, ip address: 10.134.112.222, port: 9137, worker_id: 9587ebbb5aa0981869f35d7790495a43807aed3896a83415a599baf6, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,415 I 7452 8213] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 4a26861a4740249bb188b21d01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,748 I 7452 8213] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 4a26861a4740249bb188b21d01000000, ip address: 10.134.112.222, port: 9137, worker_id: 9587ebbb5aa0981869f35d7790495a43807aed3896a83415a599baf6, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:06,327 E 7452 7452] actor_manager.cc:93: Actor handle already exists 4a26861a4740249bb188b21d01000000 | |
| [2021-04-29 17:04:12,071 I 7452 7452] core_worker.cc:2064: Finished executing task 0810906c28e4012dffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-698e1f1ed40437b69f9dcd735701493551c09d61dbad12642014279d_15885.log <== | |
| [2021-04-29 17:04:07,912 I 15885 16767] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,081 E 15885 15885] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,144 I 15885 16767] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,849 E 15885 15885] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:10,956 I 15885 15885] direct_task_transport.cc:47: Creating actor via GCS actor id = : 0ff0284e9b53b68511092acb01000000 | |
| [2021-04-29 17:04:10,989 I 15885 16767] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 0ff0284e9b53b68511092acb01000000, ip address: 10.134.61.25, port: 9201, worker_id: a267bb94ecaf5d109569616a2df40b362aef60b0349731e766e20a0d, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,368 I 15885 16767] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 0ff0284e9b53b68511092acb01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:23,907 I 15885 16767] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 0ff0284e9b53b68511092acb01000000, ip address: 10.134.61.25, port: 9201, worker_id: a267bb94ecaf5d109569616a2df40b362aef60b0349731e766e20a0d, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:26,921 E 15885 15885] actor_manager.cc:93: Actor handle already exists 0ff0284e9b53b68511092acb01000000 | |
| [2021-04-29 17:04:29,771 I 15885 15885] core_worker.cc:2064: Finished executing task 63b8f215c6d5227fffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-69d857b87cf88dbc896c104313a81afe74c02d437cea0e167559b2b7_5345.log <== | |
| [2021-04-29 17:03:57,040 E 5345 5345] actor_manager.cc:93: Actor handle already exists 859ae02bf27060dfa9e3b31201000000 | |
| [2021-04-29 17:03:58,594 I 5345 5345] core_worker.cc:2064: Finished executing task 9636d2bd3ce8a195ffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:03:58,600 E 5345 5345] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,600 E 5345 5345] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,633 I 5345 5345] direct_task_transport.cc:47: Creating actor via GCS actor id = : 7be0af708e0a50c7982e7a5a01000000 | |
| [2021-04-29 17:03:58,640 I 5345 6391] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 7be0af708e0a50c7982e7a5a01000000, ip address: 10.134.142.16, port: 9137, worker_id: 6b5f13bb41fd1fb702a09957fa3c75952d7b793ed20d2a9107cc2c3d, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,662 I 5345 6391] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 7be0af708e0a50c7982e7a5a01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,547 I 5345 6391] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 7be0af708e0a50c7982e7a5a01000000, ip address: 10.134.142.16, port: 9137, worker_id: 6b5f13bb41fd1fb702a09957fa3c75952d7b793ed20d2a9107cc2c3d, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:05,969 E 5345 5345] actor_manager.cc:93: Actor handle already exists 7be0af708e0a50c7982e7a5a01000000 | |
| [2021-04-29 17:04:11,905 I 5345 5345] core_worker.cc:2064: Finished executing task 3a54f79a8815c8d0ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-6a02d34dab0c4c104ace2291c266a8daf0f6a1e6c933963c2fa44f89_18079.log <== | |
| [2021-04-29 17:04:09,992 I 18079 19865] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,081 E 18079 18079] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,105 I 18079 19865] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,590 E 18079 18079] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,715 I 18079 18079] direct_task_transport.cc:47: Creating actor via GCS actor id = : f0094b75cbb674d94741771701000000 | |
| [2021-04-29 17:04:11,754 I 18079 19865] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: f0094b75cbb674d94741771701000000, ip address: 10.134.142.44, port: 9196, worker_id: af64e3b2c6a0880377a9b51996eb2ab31f836ad6ce57ac9e4b2cbca4, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,385 I 18079 19865] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: f0094b75cbb674d94741771701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:28,358 I 18079 19865] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: f0094b75cbb674d94741771701000000, ip address: 10.134.142.44, port: 9196, worker_id: af64e3b2c6a0880377a9b51996eb2ab31f836ad6ce57ac9e4b2cbca4, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:29,051 E 18079 18079] actor_manager.cc:93: Actor handle already exists f0094b75cbb674d94741771701000000 | |
| [2021-04-29 17:04:29,861 I 18079 18079] core_worker.cc:2064: Finished executing task da262dbc5f34fc9affffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-6af68f6c6c5c763c5605fb59777a35210750b6db393781159d0d9613_13280.log <== | |
| [2021-04-29 17:04:05,325 I 13280 15011] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,326 E 13280 13280] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,327 I 13280 15011] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:07,706 E 13280 13280] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:07,896 I 13280 13280] direct_task_transport.cc:47: Creating actor via GCS actor id = : 5643d24cae60dee22041945901000000 | |
| [2021-04-29 17:04:08,002 I 13280 15011] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 5643d24cae60dee22041945901000000, ip address: 10.134.142.44, port: 9227, worker_id: 71d777fb994c81d5945114f7b4434de8c6ee281a1a5f76d617a015ab, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:17,400 I 13280 15011] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 5643d24cae60dee22041945901000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,401 I 13280 15011] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 5643d24cae60dee22041945901000000, ip address: 10.134.142.44, port: 9227, worker_id: 71d777fb994c81d5945114f7b4434de8c6ee281a1a5f76d617a015ab, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,537 E 13280 13280] actor_manager.cc:93: Actor handle already exists 5643d24cae60dee22041945901000000 | |
| [2021-04-29 17:04:29,538 I 13280 13280] core_worker.cc:2064: Finished executing task 15f8870a0e9bcd9cffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-6afcaca1c8c9a25dc87230abd617f1e7e553c9846e3da807ce14be46_12799.log <== | |
| [2021-04-29 17:04:04,851 I 12799 13691] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,906 E 12799 12799] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:04,938 I 12799 13691] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,814 E 12799 12799] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:06,972 I 12799 12799] direct_task_transport.cc:47: Creating actor via GCS actor id = : 25f45fb02f6e33ceb800f25301000000 | |
| [2021-04-29 17:04:07,055 I 12799 13691] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 25f45fb02f6e33ceb800f25301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:14,567 I 12799 13691] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 25f45fb02f6e33ceb800f25301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,396 I 12799 13691] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 25f45fb02f6e33ceb800f25301000000, ip address: 10.134.61.130, port: 9233, worker_id: f1e69993c2e398dc8c4cfe52bd82719186a31992946a86c28c8be7f2, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,312 E 12799 12799] actor_manager.cc:93: Actor handle already exists 25f45fb02f6e33ceb800f25301000000 | |
| [2021-04-29 17:04:29,507 I 12799 12799] core_worker.cc:2064: Finished executing task 5898e8ba1f74eb89ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-6b3ff4b593a6d84f0eced93976fe76fd723a54c0a2855645748a013e_12809.log <== | |
| [2021-04-29 17:04:04,818 I 12809 13692] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,882 E 12809 12809] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:04,913 I 12809 13692] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,806 E 12809 12809] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:06,920 I 12809 12809] direct_task_transport.cc:47: Creating actor via GCS actor id = : c77f11d20e3918cb2d2a377701000000 | |
| [2021-04-29 17:04:06,987 I 12809 13692] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: c77f11d20e3918cb2d2a377701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:14,408 I 12809 13692] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: c77f11d20e3918cb2d2a377701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,398 I 12809 13692] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: c77f11d20e3918cb2d2a377701000000, ip address: 10.134.112.222, port: 9154, worker_id: 152b51f5b95c0c6c139b7892a8d26efbe0375fd4e469576607b27c5d, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,381 E 12809 12809] actor_manager.cc:93: Actor handle already exists c77f11d20e3918cb2d2a377701000000 | |
| [2021-04-29 17:04:29,516 I 12809 12809] core_worker.cc:2064: Finished executing task 869dee667c54b4acffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-6b41ed0fd8f5c2ea76534165f8da7e130cc90b25106a021710de9227_7474.log <== | |
| [2021-04-29 17:03:59,534 I 7474 8364] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,551 E 7474 7474] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,557 I 7474 8364] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,827 E 7474 7474] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,908 I 7474 7474] direct_task_transport.cc:47: Creating actor via GCS actor id = : fabc6dfd28a173c8e004f3fe01000000 | |
| [2021-04-29 17:03:59,918 I 7474 8364] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: fabc6dfd28a173c8e004f3fe01000000, ip address: 10.134.61.25, port: 9123, worker_id: 7de80433fa771fb831c7aff4a65f1766d9379df90e42e200f4978aa7, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,728 I 7474 8364] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: fabc6dfd28a173c8e004f3fe01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,363 I 7474 8364] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: fabc6dfd28a173c8e004f3fe01000000, ip address: 10.134.61.25, port: 9123, worker_id: 7de80433fa771fb831c7aff4a65f1766d9379df90e42e200f4978aa7, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,902 E 7474 7474] actor_manager.cc:93: Actor handle already exists fabc6dfd28a173c8e004f3fe01000000 | |
| [2021-04-29 17:04:29,416 I 7474 7474] core_worker.cc:2064: Finished executing task a407e74f54a49edaffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-6b7df3fca7ba4b9814ac27b04cee405ff371f74482f662b82556565f_5241.log <== | |
| [2021-04-29 17:03:58,836 E 5241 5241] actor_manager.cc:93: Actor handle already exists e751eb6aca708fc78ba66f3901000000 | |
| [2021-04-29 17:04:00,466 I 5241 5241] core_worker.cc:2064: Finished executing task e64a76023a9f9035ffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:00,493 E 5241 5241] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:00,493 E 5241 5241] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:00,555 I 5241 5241] direct_task_transport.cc:47: Creating actor via GCS actor id = : 7a8596bc55d140e1b9032e6c01000000 | |
| [2021-04-29 17:04:00,560 I 5241 6184] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 7a8596bc55d140e1b9032e6c01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,733 I 5241 6184] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 7a8596bc55d140e1b9032e6c01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,377 I 5241 6184] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 7a8596bc55d140e1b9032e6c01000000, ip address: 10.134.70.227, port: 9112, worker_id: cb5b4041769260681478dccef7f8e99dc46ad7512beb8f13d75e3432, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,999 E 5241 5241] actor_manager.cc:93: Actor handle already exists 7a8596bc55d140e1b9032e6c01000000 | |
| [2021-04-29 17:04:29,435 I 5241 5241] core_worker.cc:2064: Finished executing task 71d28cccc54a04caffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-6b8905b7b75f81ab8a8e707bf9e311a04f53b6de16933aab517ac3bf_13016.log <== | |
| [2021-04-29 17:04:06,224 I 13016 14794] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,280 E 13016 13016] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:06,302 I 13016 14794] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,849 E 13016 13016] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:08,961 I 13016 13016] direct_task_transport.cc:47: Creating actor via GCS actor id = : 8b9e1fbb7a9623e1cbe6ecbc01000000 | |
| [2021-04-29 17:04:08,998 I 13016 14794] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 8b9e1fbb7a9623e1cbe6ecbc01000000, ip address: 10.134.70.227, port: 9224, worker_id: aa2eb5c5acca66ca2ce45585009793c209580e1e64bf21c81dd9e238, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:20,550 I 13016 14794] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 8b9e1fbb7a9623e1cbe6ecbc01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,057 I 13016 14794] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8b9e1fbb7a9623e1cbe6ecbc01000000, ip address: 10.134.70.227, port: 9224, worker_id: aa2eb5c5acca66ca2ce45585009793c209580e1e64bf21c81dd9e238, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:26,962 E 13016 13016] actor_manager.cc:93: Actor handle already exists 8b9e1fbb7a9623e1cbe6ecbc01000000 | |
| [2021-04-29 17:04:29,775 I 13016 13016] core_worker.cc:2064: Finished executing task b9ebe156564c1afaffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-6c2cca3e324f51e16cdddee297e56e9ca08f3fd67a4b7951829c1f90_18581.log <== | |
| [2021-04-29 17:04:10,243 I 18581 20082] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,351 E 18581 18581] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,373 I 18581 20082] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,627 E 18581 18581] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,751 I 18581 18581] direct_task_transport.cc:47: Creating actor via GCS actor id = : 1888384c2f0d40b3e48defdf01000000 | |
| [2021-04-29 17:04:11,779 I 18581 20082] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 1888384c2f0d40b3e48defdf01000000, ip address: 10.134.61.25, port: 9179, worker_id: 40323255ff6468b580cfccfaddc1897161e88a70f616401f82af644e, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,389 I 18581 20082] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 1888384c2f0d40b3e48defdf01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:27,826 I 18581 20082] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1888384c2f0d40b3e48defdf01000000, ip address: 10.134.61.25, port: 9179, worker_id: 40323255ff6468b580cfccfaddc1897161e88a70f616401f82af644e, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,976 E 18581 18581] actor_manager.cc:93: Actor handle already exists 1888384c2f0d40b3e48defdf01000000 | |
| [2021-04-29 17:04:29,860 I 18581 18581] core_worker.cc:2064: Finished executing task 2997c650a5d997cbffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-6c450d37d2890a088f63ea31c59f43f1ce0c49d734f60e02f3c02328_18032.log <== | |
| [2021-04-29 17:04:09,969 I 18032 19771] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,022 E 18032 18032] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,043 I 18032 19771] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,587 E 18032 18032] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,703 I 18032 18032] direct_task_transport.cc:47: Creating actor via GCS actor id = : 825bb557da050ce1b4aefd0501000000 | |
| [2021-04-29 17:04:11,715 I 18032 19771] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 825bb557da050ce1b4aefd0501000000, ip address: 10.134.112.222, port: 9180, worker_id: 60fa6a6b3b5a307b8f9b33d879a646f3061fe2550c45cd66614b06e1, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,385 I 18032 19771] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 825bb557da050ce1b4aefd0501000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:27,651 I 18032 19771] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 825bb557da050ce1b4aefd0501000000, ip address: 10.134.112.222, port: 9180, worker_id: 60fa6a6b3b5a307b8f9b33d879a646f3061fe2550c45cd66614b06e1, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,936 E 18032 18032] actor_manager.cc:93: Actor handle already exists 825bb557da050ce1b4aefd0501000000 | |
| [2021-04-29 17:04:29,854 I 18032 18032] core_worker.cc:2064: Finished executing task fd78686232e0deffffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-6cb1407eb6b610cee3fc456e9a15fd90108ca009cf9cfa63b322e82f_12572.log <== | |
| [2021-04-29 17:04:04,547 I 12572 13199] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,597 E 12572 12572] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:04,616 I 12572 13199] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,513 E 12572 12572] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:06,625 I 12572 12572] direct_task_transport.cc:47: Creating actor via GCS actor id = : 0a865d4b2d4f614266f3d68801000000 | |
| [2021-04-29 17:04:06,676 I 12572 13199] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 0a865d4b2d4f614266f3d68801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:13,225 I 12572 13199] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 0a865d4b2d4f614266f3d68801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,396 I 12572 13199] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 0a865d4b2d4f614266f3d68801000000, ip address: 10.134.70.227, port: 9231, worker_id: 35a9bb2c6a1428e294acdf9c6ad782a321b14a8eeabc42d651673e10, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,322 E 12572 12572] actor_manager.cc:93: Actor handle already exists 0a865d4b2d4f614266f3d68801000000 | |
| [2021-04-29 17:04:29,497 I 12572 12572] core_worker.cc:2064: Finished executing task 220245dde24a7a45ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-6cbdd9c6c4f12bc4749b6cc3afb10b32cf37cbd88e57da8ff4450666_5304.log <== | |
| [2021-04-29 17:03:56,625 E 5304 5304] actor_manager.cc:93: Actor handle already exists 75c352976472366ff283fc8d01000000 | |
| [2021-04-29 17:03:57,965 I 5304 5304] core_worker.cc:2064: Finished executing task f4ba0959d801c2d0ffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:03:57,985 E 5304 5304] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:57,986 E 5304 5304] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,079 I 5304 5304] direct_task_transport.cc:47: Creating actor via GCS actor id = : d8031e1b722f2e6cc2cff59d01000000 | |
| [2021-04-29 17:03:58,084 I 5304 6402] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: d8031e1b722f2e6cc2cff59d01000000, ip address: 10.134.61.130, port: 9126, worker_id: b343c0d3bd242334f0e4106aed02b1a79b82813d0d3437fdb37e99a5, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,112 I 5304 6402] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: d8031e1b722f2e6cc2cff59d01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,734 I 5304 6402] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: d8031e1b722f2e6cc2cff59d01000000, ip address: 10.134.61.130, port: 9126, worker_id: b343c0d3bd242334f0e4106aed02b1a79b82813d0d3437fdb37e99a5, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:02,585 E 5304 5304] actor_manager.cc:93: Actor handle already exists d8031e1b722f2e6cc2cff59d01000000 | |
| [2021-04-29 17:04:10,480 I 5304 5304] core_worker.cc:2064: Finished executing task 9fead47a4199a264ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-6cbfb5eedd86037431f8e92d1a0bf6d6d66f2cc627f03529cb771af3_7517.log <== | |
| [2021-04-29 17:03:59,131 I 7517 8836] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,148 E 7517 7517] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,154 I 7517 8836] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,642 E 7517 7517] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,699 I 7517 7517] direct_task_transport.cc:47: Creating actor via GCS actor id = : 38bb4f73526c3defd3d50a1001000000 | |
| [2021-04-29 17:03:59,716 I 7517 8836] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 38bb4f73526c3defd3d50a1001000000, ip address: 10.134.60.23, port: 9128, worker_id: 10f848da669f0545139716abeb98877668a95f31676948fa1b9a08a4, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,722 I 7517 8836] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 38bb4f73526c3defd3d50a1001000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:19,263 I 7517 8836] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 38bb4f73526c3defd3d50a1001000000, ip address: 10.134.60.23, port: 9128, worker_id: 10f848da669f0545139716abeb98877668a95f31676948fa1b9a08a4, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,038 E 7517 7517] actor_manager.cc:93: Actor handle already exists 38bb4f73526c3defd3d50a1001000000 | |
| [2021-04-29 17:04:28,907 I 7517 7517] core_worker.cc:2064: Finished executing task f468566a6b44fbd7ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-6d1f47c0a98c32c0224e2fdc3a627b663c361fcd0fbf90cc2e3702bc_7424.log <== | |
| [2021-04-29 17:03:58,015 I 7424 7830] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,052 E 7424 7424] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,060 I 7424 7830] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,675 E 7424 7424] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:58,738 I 7424 7424] direct_task_transport.cc:47: Creating actor via GCS actor id = : 7d06052c4f710366903318b401000000 | |
| [2021-04-29 17:03:58,744 I 7424 7830] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 7d06052c4f710366903318b401000000, ip address: 10.134.61.25, port: 9138, worker_id: e735480a8c1a8b0566913ba9f03f7fd7486257a049346a768e783c8e, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,035 I 7424 7830] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 7d06052c4f710366903318b401000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,549 I 7424 7830] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 7d06052c4f710366903318b401000000, ip address: 10.134.61.25, port: 9138, worker_id: e735480a8c1a8b0566913ba9f03f7fd7486257a049346a768e783c8e, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:06,206 E 7424 7424] actor_manager.cc:93: Actor handle already exists 7d06052c4f710366903318b401000000 | |
| [2021-04-29 17:04:12,030 I 7424 7424] core_worker.cc:2064: Finished executing task d277c2d463c5ba3dffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-6dc69a60893337357991cbfc4049515ac0d595b079f748e62bfd5442_5288.log <== | |
| [2021-04-29 17:03:58,057 E 5288 5288] actor_manager.cc:93: Actor handle already exists 719af0750297b3e51ec534c701000000 | |
| [2021-04-29 17:03:59,688 I 5288 5288] core_worker.cc:2064: Finished executing task 4ae6da0935013634ffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:03:59,742 E 5288 5288] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,742 E 5288 5288] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,831 I 5288 5288] direct_task_transport.cc:47: Creating actor via GCS actor id = : 91579d4566a342cf73f6177001000000 | |
| [2021-04-29 17:03:59,833 I 5288 6267] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 91579d4566a342cf73f6177001000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,726 I 5288 6267] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 91579d4566a342cf73f6177001000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,222 I 5288 6267] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 91579d4566a342cf73f6177001000000, ip address: 10.134.112.222, port: 9128, worker_id: 9774de5065e24af8128ea04975988a49539320d19668a13b9cae059b, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,866 E 5288 5288] actor_manager.cc:93: Actor handle already exists 91579d4566a342cf73f6177001000000 | |
| [2021-04-29 17:04:29,399 I 5288 5288] core_worker.cc:2064: Finished executing task c44e199ca925a27dffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-6dde3c73ecf8f216da97ca56e22d1e171cb0d557fcd23d74815558c8_18599.log <== | |
| [2021-04-29 17:04:10,143 I 18599 20038] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,294 E 18599 18599] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,312 I 18599 20038] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,624 E 18599 18599] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,744 I 18599 18599] direct_task_transport.cc:47: Creating actor via GCS actor id = : 7da0a7dbd4223f6dac0e539201000000 | |
| [2021-04-29 17:04:11,779 I 18599 20038] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 7da0a7dbd4223f6dac0e539201000000, ip address: 10.134.142.16, port: 9187, worker_id: b2bae379381d716d85151682f24da8de140000c98a9841e3059b701b, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,389 I 18599 20038] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 7da0a7dbd4223f6dac0e539201000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:27,554 I 18599 20038] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 7da0a7dbd4223f6dac0e539201000000, ip address: 10.134.142.16, port: 9187, worker_id: b2bae379381d716d85151682f24da8de140000c98a9841e3059b701b, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,893 E 18599 18599] actor_manager.cc:93: Actor handle already exists 7da0a7dbd4223f6dac0e539201000000 | |
| [2021-04-29 17:04:29,858 I 18599 18599] core_worker.cc:2064: Finished executing task 7361f5ea29312b12ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-6deda5016711744192d21ec99fdb15636bb775219370f3a5cd2cfd09_5233.log <== | |
| [2021-04-29 17:04:02,406 E 5233 5233] actor_manager.cc:93: Actor handle already exists 0c3f962871739f338e857c7d01000000 | |
| [2021-04-29 17:04:10,193 I 5233 5233] core_worker.cc:2064: Finished executing task b14a9ef231b78e39ffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:10,209 E 5233 5233] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,209 E 5233 5233] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,309 I 5233 5233] direct_task_transport.cc:47: Creating actor via GCS actor id = : 2af66eebee71f57277a7b31401000000 | |
| [2021-04-29 17:04:10,338 I 5233 5898] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 2af66eebee71f57277a7b31401000000, ip address: 10.134.61.130, port: 9223, worker_id: d40f16771ea1000449f70bd6f3f9d998e492e39ea70d338e32b2f2cc, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,364 I 5233 5898] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 2af66eebee71f57277a7b31401000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,231 I 5233 5898] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 2af66eebee71f57277a7b31401000000, ip address: 10.134.61.130, port: 9223, worker_id: d40f16771ea1000449f70bd6f3f9d998e492e39ea70d338e32b2f2cc, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:27,069 E 5233 5233] actor_manager.cc:93: Actor handle already exists 2af66eebee71f57277a7b31401000000 | |
| [2021-04-29 17:04:29,785 I 5233 5233] core_worker.cc:2064: Finished executing task 370b38b6f6ecc49dffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-6df2eb0da148c279781cdab4460d840b2c973f048a30da36c187ac49_7584.log <== | |
| [2021-04-29 17:03:58,772 I 7584 9204] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,810 E 7584 7584] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,819 I 7584 9204] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,500 E 7584 7584] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,553 I 7584 7584] direct_task_transport.cc:47: Creating actor via GCS actor id = : f597ef1d5ee2dcf2e8505acd01000000 | |
| [2021-04-29 17:03:59,557 I 7584 9204] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: f597ef1d5ee2dcf2e8505acd01000000, ip address: 10.134.60.23, port: 9132, worker_id: 007e5d561e1500486922ab80591048bb912a519eb882800c9d0cf05a, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,479 I 7584 9204] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: f597ef1d5ee2dcf2e8505acd01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:13,679 I 7584 9204] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: f597ef1d5ee2dcf2e8505acd01000000, ip address: 10.134.60.23, port: 9132, worker_id: 007e5d561e1500486922ab80591048bb912a519eb882800c9d0cf05a, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:15,396 E 7584 7584] actor_manager.cc:93: Actor handle already exists f597ef1d5ee2dcf2e8505acd01000000 | |
| [2021-04-29 17:04:22,295 I 7584 7584] core_worker.cc:2064: Finished executing task 98c65c602d8fa016ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-6ea0f76d1947804ca046b91aa17b98f07b1b739332c9d55cb67441a1_5372.log <== | |
| [2021-04-29 17:03:46,980 E 5372 5372] actor_manager.cc:93: Actor handle already exists ff125b7bd2b8fb7149bc306301000000 | |
| [2021-04-29 17:03:47,082 I 5372 5372] core_worker.cc:2064: Finished executing task ffffffffffffffffe23f529dd546638e8db390a801000000, status=OK | |
| [2021-04-29 17:03:47,082 I 5372 5372] direct_actor_transport.cc:497: Actor creation task finished, task_id: ffffffffffffffffe23f529dd546638e8db390a801000000, actor_id: e23f529dd546638e8db390a801000000 | |
| [2021-04-29 17:03:47,084 I 5372 6918] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: e23f529dd546638e8db390a801000000, ip address: 10.133.100.138, port: 9235, worker_id: 6ea0f76d1947804ca046b91aa17b98f07b1b739332c9d55cb67441a1, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:47,085 I 5372 5372] core_worker.cc:2064: Finished executing task f0d5f68933499b37e23f529dd546638e8db390a801000000, status=OK | |
| [2021-04-29 17:03:47,149 I 5372 5372] core_worker.cc:2064: Finished executing task 962ed55c6842d8cfe23f529dd546638e8db390a801000000, status=OK | |
| [2021-04-29 17:03:47,367 I 5372 5372] core_worker.cc:2064: Finished executing task 262a1c2d4fc2ce9ee23f529dd546638e8db390a801000000, status=OK | |
| [2021-04-29 17:03:47,703 I 5372 5372] core_worker.cc:2064: Finished executing task 084497e66b854886e23f529dd546638e8db390a801000000, status=OK | |
| [2021-04-29 17:03:47,719 I 5372 5372] core_worker.cc:2064: Finished executing task b07770e3bbc08c68e23f529dd546638e8db390a801000000, status=OK | |
| [2021-04-29 17:03:51,075 I 5372 5372] core_worker.cc:2064: Finished executing task a978c5f49e0f849be23f529dd546638e8db390a801000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-6ea9a5ca451e58b1adfde67a7226a121450318d6e9bbef25edee3909_18631.log <== | |
| [2021-04-29 17:04:10,180 I 18631 20066] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,316 E 18631 18631] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,337 I 18631 20066] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,626 E 18631 18631] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,745 I 18631 18631] direct_task_transport.cc:47: Creating actor via GCS actor id = : 5b504155a3c359ed3a75ab4301000000 | |
| [2021-04-29 17:04:11,779 I 18631 20066] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 5b504155a3c359ed3a75ab4301000000, ip address: 10.134.60.23, port: 9172, worker_id: 7a247c87448520d3fc03351a41e0a4f6472dc9b30f7b18ddce3c89ee, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,389 I 18631 20066] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 5b504155a3c359ed3a75ab4301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:28,178 I 18631 20066] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 5b504155a3c359ed3a75ab4301000000, ip address: 10.134.60.23, port: 9172, worker_id: 7a247c87448520d3fc03351a41e0a4f6472dc9b30f7b18ddce3c89ee, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:29,007 E 18631 18631] actor_manager.cc:93: Actor handle already exists 5b504155a3c359ed3a75ab4301000000 | |
| [2021-04-29 17:04:29,867 I 18631 18631] core_worker.cc:2064: Finished executing task 5473935290e24987ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-6ede44a15f7e60ce417069ae6df94916222379df26f6f95893b84e1e_17282.log <== | |
| [2021-04-29 17:04:08,966 I 17282 18881] service_based_accessor.cc:579: Received notification for node id = ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, IsAlive = 1 | |
| [2021-04-29 17:04:08,966 I 17282 18881] service_based_accessor.cc:579: Received notification for node id = d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, IsAlive = 1 | |
| [2021-04-29 17:04:08,966 I 17282 18881] service_based_accessor.cc:579: Received notification for node id = b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, IsAlive = 1 | |
| [2021-04-29 17:04:08,966 I 17282 18881] service_based_accessor.cc:579: Received notification for node id = 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, IsAlive = 1 | |
| [2021-04-29 17:04:08,966 I 17282 18881] service_based_accessor.cc:579: Received notification for node id = 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, IsAlive = 1 | |
| [2021-04-29 17:04:10,128 I 17282 18881] core_worker.cc:717: Exit signal received, this process will exit after all outstanding tasks have finished, exit_type=INTENDED_EXIT | |
| [2021-04-29 17:04:10,128 I 17282 17282] raylet_client.cc:129: RayletClient::Disconnect, exit_type=INTENDED_EXIT, has creation_task_exception_pb_bytes=0 | |
| [2021-04-29 17:04:10,128 I 17282 17282] core_worker.cc:308: Removed worker 6ede44a15f7e60ce417069ae6df94916222379df26f6f95893b84e1e | |
| [2021-04-29 17:04:10,133 I 17282 17282] core_worker.cc:181: Destructing CoreWorkerProcess. pid: 17282 | |
| [2021-04-29 17:04:10,133 I 17282 17282] io_service_pool.cc:47: IOServicePool is stopped. | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-6fa39b426a3da91c9511fc5608a4ef5ccab928f86ad1391fc468c5fb_4056.log <== | |
| [2021-04-29 17:03:44,992 I 4056 4056] core_worker.cc:2064: Finished executing task 925f366943ecfb04d0db24eef0c4458d61de76de01000000, status=OK | |
| [2021-04-29 17:04:47,652 I 4056 4056] core_worker.cc:2064: Finished executing task cd1fea4e81eb5573d0db24eef0c4458d61de76de01000000, status=OK | |
| [2021-04-29 17:04:49,634 I 4056 4056] core_worker.cc:2064: Finished executing task 19fbd491462bad2cd0db24eef0c4458d61de76de01000000, status=OK | |
| [2021-04-29 17:04:51,105 I 4056 4056] core_worker.cc:2064: Finished executing task 63a73de6e93f6e2ad0db24eef0c4458d61de76de01000000, status=OK | |
| [2021-04-29 17:04:52,226 I 4056 4056] core_worker.cc:2064: Finished executing task 1e173051dd9faf46d0db24eef0c4458d61de76de01000000, status=OK | |
| [2021-04-29 17:04:53,147 I 4056 4056] core_worker.cc:2064: Finished executing task 284670b1beaa81c0d0db24eef0c4458d61de76de01000000, status=OK | |
| [2021-04-29 17:04:54,612 I 4056 4056] core_worker.cc:2064: Finished executing task 62714aefa9057a09d0db24eef0c4458d61de76de01000000, status=OK | |
| [2021-04-29 17:04:55,999 I 4056 4056] core_worker.cc:2064: Finished executing task 54f8aaede6d7562fd0db24eef0c4458d61de76de01000000, status=OK | |
| [2021-04-29 17:04:58,112 I 4056 4056] core_worker.cc:2064: Finished executing task 606c2de203c3ab81d0db24eef0c4458d61de76de01000000, status=OK | |
| [2021-04-29 17:04:58,219 I 4056 4056] core_worker.cc:2064: Finished executing task 1d3e7da3d16c58d8d0db24eef0c4458d61de76de01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-7072facd0065583eddf9d41c97588d77ff9613d42994b23c8e190885_9689.log <== | |
| [2021-04-29 17:04:01,192 I 9689 10407] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,219 E 9689 9689] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,229 I 9689 10407] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,776 E 9689 9689] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:01,842 I 9689 10407] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: c0511b25ade7168a5f3ba4f001000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,843 I 9689 9689] direct_task_transport.cc:47: Creating actor via GCS actor id = : c0511b25ade7168a5f3ba4f001000000 | |
| [2021-04-29 17:04:02,489 I 9689 10407] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: c0511b25ade7168a5f3ba4f001000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,391 I 9689 10407] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: c0511b25ade7168a5f3ba4f001000000, ip address: 10.134.60.23, port: 9120, worker_id: c154214f6d303e94f5f0bea40ca9e697b995953e81686158b17207bd, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,074 E 9689 9689] actor_manager.cc:93: Actor handle already exists c0511b25ade7168a5f3ba4f001000000 | |
| [2021-04-29 17:04:29,462 I 9689 9689] core_worker.cc:2064: Finished executing task ff39152fe0874778ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-708dae4bcf7f63360c933303eb99c5ff46cc61b831200b2aecb9e9c3_13030.log <== | |
| [2021-04-29 17:04:05,278 I 13030 14808] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,280 E 13030 13030] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,281 I 13030 14808] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:07,670 E 13030 13030] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:07,854 I 13030 13030] direct_task_transport.cc:47: Creating actor via GCS actor id = : b0c49a6e28359bec67532e6e01000000 | |
| [2021-04-29 17:04:07,910 I 13030 14808] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: b0c49a6e28359bec67532e6e01000000, ip address: 10.134.112.222, port: 21101, worker_id: NIL_ID, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:17,266 I 13030 14808] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: b0c49a6e28359bec67532e6e01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,399 I 13030 14808] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: b0c49a6e28359bec67532e6e01000000, ip address: 10.134.112.222, port: 9159, worker_id: 8fe182f8996c0216a580ac2c798e064a942f8edbf91f24396fd4722c, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,442 E 13030 13030] actor_manager.cc:93: Actor handle already exists b0c49a6e28359bec67532e6e01000000 | |
| [2021-04-29 17:04:29,530 I 13030 13030] core_worker.cc:2064: Finished executing task 7339db7c624aac14ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-71814fbbd2d54630f3748829591c4e6cb5aa90d27e921d0065d120ed_7511.log <== | |
| [2021-04-29 17:03:59,155 I 7511 8830] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,170 E 7511 7511] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,177 I 7511 8830] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,643 E 7511 7511] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,703 I 7511 7511] direct_task_transport.cc:47: Creating actor via GCS actor id = : d6279772071acfcfc81ac17701000000 | |
| [2021-04-29 17:03:59,715 I 7511 8830] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: d6279772071acfcfc81ac17701000000, ip address: 10.133.95.40, port: 9128, worker_id: c2ef5c72f9a4b377b97b8b3c8b04b387293da168e690185023eeac17, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,723 I 7511 8830] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: d6279772071acfcfc81ac17701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:19,263 I 7511 8830] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: d6279772071acfcfc81ac17701000000, ip address: 10.133.95.40, port: 9128, worker_id: c2ef5c72f9a4b377b97b8b3c8b04b387293da168e690185023eeac17, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,044 E 7511 7511] actor_manager.cc:93: Actor handle already exists d6279772071acfcfc81ac17701000000 | |
| [2021-04-29 17:04:28,908 I 7511 7511] core_worker.cc:2064: Finished executing task 0ac06ac0215ed0c8ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-71b4b26f7db9564522e255195d5d639cfd86291347a74fafd5677be1_10229.log <== | |
| [2021-04-29 17:04:01,827 I 10229 11829] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,862 E 10229 10229] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,872 I 10229 11829] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,145 E 10229 10229] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:04,242 I 10229 10229] direct_task_transport.cc:47: Creating actor via GCS actor id = : 33eb827ceeec421d5619998c01000000 | |
| [2021-04-29 17:04:04,269 I 10229 11829] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 33eb827ceeec421d5619998c01000000, ip address: 10.134.61.25, port: 9111, worker_id: 6609df79a25d5bf20569303e5e1cf73522dc36c9ac52db5bca2c51a8, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,549 I 10229 11829] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 33eb827ceeec421d5619998c01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,392 I 10229 11829] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 33eb827ceeec421d5619998c01000000, ip address: 10.134.61.25, port: 9111, worker_id: 6609df79a25d5bf20569303e5e1cf73522dc36c9ac52db5bca2c51a8, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,729 E 10229 10229] actor_manager.cc:93: Actor handle already exists 33eb827ceeec421d5619998c01000000 | |
| [2021-04-29 17:04:29,546 I 10229 10229] core_worker.cc:2064: Finished executing task d27edad7fdd66ee5ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-71fd32640d0626fb6223cf56e38b2a0ff0995c14b660267dc7b95302_9691.log <== | |
| [2021-04-29 17:04:01,175 I 9691 10347] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,207 E 9691 9691] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,216 I 9691 10347] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,774 E 9691 9691] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:01,843 I 9691 9691] direct_task_transport.cc:47: Creating actor via GCS actor id = : e2510c588673ad7837fc937301000000 | |
| [2021-04-29 17:04:01,862 I 9691 10347] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: e2510c588673ad7837fc937301000000, ip address: 10.134.142.16, port: 9120, worker_id: ab8ca3151f3108355bdd92bc4545c70e107c96f1d08d0f459aacd5c5, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,561 I 9691 10347] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: e2510c588673ad7837fc937301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,390 I 9691 10347] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: e2510c588673ad7837fc937301000000, ip address: 10.134.142.16, port: 9120, worker_id: ab8ca3151f3108355bdd92bc4545c70e107c96f1d08d0f459aacd5c5, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,081 E 9691 9691] actor_manager.cc:93: Actor handle already exists e2510c588673ad7837fc937301000000 | |
| [2021-04-29 17:04:29,453 I 9691 9691] core_worker.cc:2064: Finished executing task 7d567e4ee8b61b26ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-72e177bd4d80d0f51c64caca9d6b2710d1419e39311f09a15e94d331_12928.log <== | |
| [2021-04-29 17:04:05,151 I 12928 14452] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,161 E 12928 12928] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,167 I 12928 14452] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:07,309 E 12928 12928] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:07,383 I 12928 12928] direct_task_transport.cc:47: Creating actor via GCS actor id = : 95c6e5ead41dcfe1e38e5a6701000000 | |
| [2021-04-29 17:04:07,401 I 12928 14452] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 95c6e5ead41dcfe1e38e5a6701000000, ip address: 10.134.70.227, port: 9229, worker_id: 961ee76a8e73ca62ac12869121926a64149f7a6ce3a837c1978a972c, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:16,848 I 12928 14452] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 95c6e5ead41dcfe1e38e5a6701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,401 I 12928 14452] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 95c6e5ead41dcfe1e38e5a6701000000, ip address: 10.134.70.227, port: 9229, worker_id: 961ee76a8e73ca62ac12869121926a64149f7a6ce3a837c1978a972c, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,514 E 12928 12928] actor_manager.cc:93: Actor handle already exists 95c6e5ead41dcfe1e38e5a6701000000 | |
| [2021-04-29 17:04:29,538 I 12928 12928] core_worker.cc:2064: Finished executing task c11a00b9eccf1699ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-72ec37c7ece8445fd8fd888f3f1e4ca5c7c14bb446f07b212c057b20_17297.log <== | |
| [2021-04-29 17:04:08,966 I 17297 18884] service_based_accessor.cc:579: Received notification for node id = ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, IsAlive = 1 | |
| [2021-04-29 17:04:08,966 I 17297 18884] service_based_accessor.cc:579: Received notification for node id = d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, IsAlive = 1 | |
| [2021-04-29 17:04:08,966 I 17297 18884] service_based_accessor.cc:579: Received notification for node id = b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, IsAlive = 1 | |
| [2021-04-29 17:04:08,966 I 17297 18884] service_based_accessor.cc:579: Received notification for node id = 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, IsAlive = 1 | |
| [2021-04-29 17:04:08,966 I 17297 18884] service_based_accessor.cc:579: Received notification for node id = 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, IsAlive = 1 | |
| [2021-04-29 17:04:10,128 I 17297 18884] core_worker.cc:717: Exit signal received, this process will exit after all outstanding tasks have finished, exit_type=INTENDED_EXIT | |
| [2021-04-29 17:04:10,128 I 17297 17297] raylet_client.cc:129: RayletClient::Disconnect, exit_type=INTENDED_EXIT, has creation_task_exception_pb_bytes=0 | |
| [2021-04-29 17:04:10,129 I 17297 17297] core_worker.cc:308: Removed worker 72ec37c7ece8445fd8fd888f3f1e4ca5c7c14bb446f07b212c057b20 | |
| [2021-04-29 17:04:10,133 I 17297 17297] core_worker.cc:181: Destructing CoreWorkerProcess. pid: 17297 | |
| [2021-04-29 17:04:10,134 I 17297 17297] io_service_pool.cc:47: IOServicePool is stopped. | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-735aeb362760096f7bf2d2749d6ed25bbbe5e982236694d17edd613d_9731.log <== | |
| [2021-04-29 17:04:01,277 I 9731 10672] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,301 E 9731 9731] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,310 I 9731 10672] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,923 E 9731 9731] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:01,985 I 9731 9731] direct_task_transport.cc:47: Creating actor via GCS actor id = : aafb9af6e75f33c9bd2e611f01000000 | |
| [2021-04-29 17:04:01,994 I 9731 10672] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: aafb9af6e75f33c9bd2e611f01000000, ip address: 10.134.60.23, port: 9117, worker_id: 1c0c48a9bb8eda5b12b0d6cda9114377e750e8a5ea231b2edd752b13, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,531 I 9731 10672] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: aafb9af6e75f33c9bd2e611f01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,391 I 9731 10672] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: aafb9af6e75f33c9bd2e611f01000000, ip address: 10.134.60.23, port: 9117, worker_id: 1c0c48a9bb8eda5b12b0d6cda9114377e750e8a5ea231b2edd752b13, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,123 E 9731 9731] actor_manager.cc:93: Actor handle already exists aafb9af6e75f33c9bd2e611f01000000 | |
| [2021-04-29 17:04:29,469 I 9731 9731] core_worker.cc:2064: Finished executing task aee4d1a697202fe5ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-73bbc7f254ee67d68e3bd12e246039fe0e633e55c88f864580b55c8f_4044.log <== | |
| [2021-04-29 17:04:02,396 E 4044 4044] actor_manager.cc:93: Actor handle already exists e14268f338c7bb5bd08ecab401000000 | |
| [2021-04-29 17:04:10,171 I 4044 4044] core_worker.cc:2064: Finished executing task c1b983d6b874ecaaffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:10,176 E 4044 4044] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,176 E 4044 4044] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,284 I 4044 4044] direct_task_transport.cc:47: Creating actor via GCS actor id = : d407933dfb7a10e105ece27e01000000 | |
| [2021-04-29 17:04:10,313 I 4044 4732] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: d407933dfb7a10e105ece27e01000000, ip address: 10.133.95.40, port: 9204, worker_id: 87c9b862de49a286353dc31955e3b5ae5826961c5e28d2c27166f315, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,364 I 4044 4732] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: d407933dfb7a10e105ece27e01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:22,279 I 4044 4732] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: d407933dfb7a10e105ece27e01000000, ip address: 10.133.95.40, port: 9204, worker_id: 87c9b862de49a286353dc31955e3b5ae5826961c5e28d2c27166f315, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:25,130 E 4044 4044] actor_manager.cc:93: Actor handle already exists d407933dfb7a10e105ece27e01000000 | |
| [2021-04-29 17:04:29,689 I 4044 4044] core_worker.cc:2064: Finished executing task 4886df8d23f6b258ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-73c7abf278279e6ee5300a22f3023d2e097a8693b094e35469cec3da_15956.log <== | |
| [2021-04-29 17:04:08,333 I 15956 17388] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,448 E 15956 15956] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,475 I 15956 17388] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,034 E 15956 15956] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,133 I 15956 15956] direct_task_transport.cc:47: Creating actor via GCS actor id = : 548f6077f0cf42c6bf657f0301000000 | |
| [2021-04-29 17:04:11,171 I 15956 17388] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 548f6077f0cf42c6bf657f0301000000, ip address: 10.134.61.25, port: 9196, worker_id: 72c19f1d6940e61438900f509bdc2134f998b160f1594c53d3a3dd4c, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,373 I 15956 17388] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 548f6077f0cf42c6bf657f0301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,581 I 15956 17388] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 548f6077f0cf42c6bf657f0301000000, ip address: 10.134.61.25, port: 9196, worker_id: 72c19f1d6940e61438900f509bdc2134f998b160f1594c53d3a3dd4c, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:27,404 E 15956 15956] actor_manager.cc:93: Actor handle already exists 548f6077f0cf42c6bf657f0301000000 | |
| [2021-04-29 17:04:29,792 I 15956 15956] core_worker.cc:2064: Finished executing task f52686884fba286fffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-74000afa7f4db1c73a60d3a351e39eb6d5addd9a9e66ce7e9e90cb2b_7516.log <== | |
| [2021-04-29 17:03:59,216 I 7516 8803] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,239 E 7516 7516] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,246 I 7516 8803] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,685 E 7516 7516] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,746 I 7516 7516] direct_task_transport.cc:47: Creating actor via GCS actor id = : c740847a6341267ac09de8ab01000000 | |
| [2021-04-29 17:03:59,759 I 7516 8803] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: c740847a6341267ac09de8ab01000000, ip address: 10.134.142.16, port: 9129, worker_id: a6a7517133eca75fb4683e12ff437a2e029abae87e01ecb860ee4200, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,724 I 7516 8803] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: c740847a6341267ac09de8ab01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:19,263 I 7516 8803] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: c740847a6341267ac09de8ab01000000, ip address: 10.134.142.16, port: 9129, worker_id: a6a7517133eca75fb4683e12ff437a2e029abae87e01ecb860ee4200, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,055 E 7516 7516] actor_manager.cc:93: Actor handle already exists c740847a6341267ac09de8ab01000000 | |
| [2021-04-29 17:04:28,913 I 7516 7516] core_worker.cc:2064: Finished executing task 8447947178269aaeffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-740fdfc2920b8640e2688c51a4fea35dd537856415ab4bbff85b4c62_15872.log <== | |
| [2021-04-29 17:04:07,728 I 15872 16505] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:07,789 E 15872 15872] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:07,849 I 15872 16505] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,496 E 15872 15872] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:10,567 I 15872 15872] direct_task_transport.cc:47: Creating actor via GCS actor id = : 393be01fdd98e30cbaa8121201000000 | |
| [2021-04-29 17:04:10,592 I 15872 16505] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 393be01fdd98e30cbaa8121201000000, ip address: 10.134.60.23, port: 9200, worker_id: 1f952b0da3b8c10e9ffbfd03d790b7e7a8fe783f4abb8e5c97bb3b96, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,367 I 15872 16505] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 393be01fdd98e30cbaa8121201000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:22,959 I 15872 16505] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 393be01fdd98e30cbaa8121201000000, ip address: 10.134.60.23, port: 9200, worker_id: 1f952b0da3b8c10e9ffbfd03d790b7e7a8fe783f4abb8e5c97bb3b96, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:25,747 E 15872 15872] actor_manager.cc:93: Actor handle already exists 393be01fdd98e30cbaa8121201000000 | |
| [2021-04-29 17:04:29,767 I 15872 15872] core_worker.cc:2064: Finished executing task 41d3c22a663ca2b5ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-7430008cb442bce7a5ff03f883572ad0696c3a7fd741960a918e3af0_7972.log <== | |
| [2021-04-29 17:03:58,564 I 7972 9415] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,565 E 7972 7972] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,566 I 7972 9415] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,958 E 7972 7972] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,019 I 7972 7972] direct_task_transport.cc:47: Creating actor via GCS actor id = : b7907ef41fd8c3596ac1e07101000000 | |
| [2021-04-29 17:03:59,030 I 7972 9415] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: b7907ef41fd8c3596ac1e07101000000, ip address: 10.134.61.130, port: 9123, worker_id: dbbec3cb70ff275312612ce0bda1c5c966cf66d6073d0312e5d5e5f8, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:00,713 I 7972 9415] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: b7907ef41fd8c3596ac1e07101000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,208 I 7972 9415] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: b7907ef41fd8c3596ac1e07101000000, ip address: 10.134.61.130, port: 9123, worker_id: dbbec3cb70ff275312612ce0bda1c5c966cf66d6073d0312e5d5e5f8, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:07,504 E 7972 7972] actor_manager.cc:93: Actor handle already exists b7907ef41fd8c3596ac1e07101000000 | |
| [2021-04-29 17:04:12,741 I 7972 7972] core_worker.cc:2064: Finished executing task 8c69fc44d9eb6fd8ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-748fb3a5b9934a75e7d31a280588855275869e7e464e2b51ce14aed2_17968.log <== | |
| [2021-04-29 17:04:09,869 I 17968 19655] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,945 E 17968 17968] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,967 I 17968 19655] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,583 E 17968 17968] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,704 I 17968 17968] direct_task_transport.cc:47: Creating actor via GCS actor id = : 173c7eb0ed73c8fa5a97659501000000 | |
| [2021-04-29 17:04:11,756 I 17968 19655] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 173c7eb0ed73c8fa5a97659501000000, ip address: 10.134.60.23, port: 9186, worker_id: 1717dec51a719d12a576834a1c4b04da4a45c3ab013fb9411acea16e, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,386 I 17968 19655] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 173c7eb0ed73c8fa5a97659501000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:27,155 I 17968 19655] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 173c7eb0ed73c8fa5a97659501000000, ip address: 10.134.60.23, port: 9186, worker_id: 1717dec51a719d12a576834a1c4b04da4a45c3ab013fb9411acea16e, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,812 E 17968 17968] actor_manager.cc:93: Actor handle already exists 173c7eb0ed73c8fa5a97659501000000 | |
| [2021-04-29 17:04:29,846 I 17968 17968] core_worker.cc:2064: Finished executing task 6bf346e5ad249a48ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-74b2a3f7b54c37626452230892826bd3b29dcb1be0151f7ba3f6285b_15925.log <== | |
| [2021-04-29 17:04:07,996 I 15925 16975] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,203 E 15925 15925] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,249 I 15925 16975] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,963 E 15925 15925] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,058 I 15925 15925] direct_task_transport.cc:47: Creating actor via GCS actor id = : db6ffeed16439703decb3fca01000000 | |
| [2021-04-29 17:04:11,103 I 15925 16975] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: db6ffeed16439703decb3fca01000000, ip address: 10.134.60.23, port: 9197, worker_id: 69a2d65b5e557c77962c9835090bae52bf8fd71fffc50639e4cc22b9, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,369 I 15925 16975] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: db6ffeed16439703decb3fca01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,024 I 15925 16975] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: db6ffeed16439703decb3fca01000000, ip address: 10.134.60.23, port: 9197, worker_id: 69a2d65b5e557c77962c9835090bae52bf8fd71fffc50639e4cc22b9, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:26,957 E 15925 15925] actor_manager.cc:93: Actor handle already exists db6ffeed16439703decb3fca01000000 | |
| [2021-04-29 17:04:29,773 I 15925 15925] core_worker.cc:2064: Finished executing task aaf2fa6869a6ab6affffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-74bc38585eb2cabfc53a75e8220de6c0f4293801b2af4513ea62031e_9772.log <== | |
| [2021-04-29 17:04:02,349 I 9772 11218] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,369 E 9772 9772] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:02,376 I 9772 11218] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,409 E 9772 9772] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,490 I 9772 9772] direct_task_transport.cc:47: Creating actor via GCS actor id = : a8780bd3bb1ba0204f55d51001000000 | |
| [2021-04-29 17:04:05,517 I 9772 11218] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: a8780bd3bb1ba0204f55d51001000000, ip address: 10.134.112.222, port: 9108, worker_id: d7377b29d833cdde55a830e496f6d6c31b854507fb203e3d46fc611a, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,459 I 9772 11218] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: a8780bd3bb1ba0204f55d51001000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,394 I 9772 11218] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: a8780bd3bb1ba0204f55d51001000000, ip address: 10.134.112.222, port: 9108, worker_id: d7377b29d833cdde55a830e496f6d6c31b854507fb203e3d46fc611a, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,216 E 9772 9772] actor_manager.cc:93: Actor handle already exists a8780bd3bb1ba0204f55d51001000000 | |
| [2021-04-29 17:04:29,481 I 9772 9772] core_worker.cc:2064: Finished executing task 426ef4009cf5798dffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-754cc491b79d2ecf5687c266422b09d79319de6a3e607809516f505f_7582.log <== | |
| [2021-04-29 17:03:58,416 I 7582 9237] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,417 E 7582 7582] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,418 I 7582 9237] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,756 E 7582 7582] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:58,845 I 7582 7582] direct_task_transport.cc:47: Creating actor via GCS actor id = : 416aa8b4f74ef7688e7ef21e01000000 | |
| [2021-04-29 17:03:58,856 I 7582 9237] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 416aa8b4f74ef7688e7ef21e01000000, ip address: 10.134.112.222, port: 9136, worker_id: dbb0af11b4fe8ea1982db61cae144a78bf6388defb41760034a01a68, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:00,138 I 7582 9237] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 416aa8b4f74ef7688e7ef21e01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,172 I 7582 9237] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 416aa8b4f74ef7688e7ef21e01000000, ip address: 10.134.112.222, port: 9136, worker_id: dbb0af11b4fe8ea1982db61cae144a78bf6388defb41760034a01a68, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:07,256 E 7582 7582] actor_manager.cc:93: Actor handle already exists 416aa8b4f74ef7688e7ef21e01000000 | |
| [2021-04-29 17:04:12,581 I 7582 7582] core_worker.cc:2064: Finished executing task 3971d2f80ecb9d99ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-7558626766f9012883acba4e253877d1dfdb6ee8264330a25999f52b_17948.log <== | |
| [2021-04-29 17:04:09,380 I 17948 19453] service_based_accessor.cc:579: Received notification for node id = ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, IsAlive = 1 | |
| [2021-04-29 17:04:09,380 I 17948 19453] service_based_accessor.cc:579: Received notification for node id = d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, IsAlive = 1 | |
| [2021-04-29 17:04:09,380 I 17948 19453] service_based_accessor.cc:579: Received notification for node id = b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, IsAlive = 1 | |
| [2021-04-29 17:04:09,380 I 17948 19453] service_based_accessor.cc:579: Received notification for node id = 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, IsAlive = 1 | |
| [2021-04-29 17:04:09,380 I 17948 19453] service_based_accessor.cc:579: Received notification for node id = 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, IsAlive = 1 | |
| [2021-04-29 17:04:10,532 I 17948 19453] core_worker.cc:717: Exit signal received, this process will exit after all outstanding tasks have finished, exit_type=INTENDED_EXIT | |
| [2021-04-29 17:04:10,532 I 17948 17948] raylet_client.cc:129: RayletClient::Disconnect, exit_type=INTENDED_EXIT, has creation_task_exception_pb_bytes=0 | |
| [2021-04-29 17:04:10,533 I 17948 17948] core_worker.cc:308: Removed worker 7558626766f9012883acba4e253877d1dfdb6ee8264330a25999f52b | |
| [2021-04-29 17:04:10,535 I 17948 17948] core_worker.cc:181: Destructing CoreWorkerProcess. pid: 17948 | |
| [2021-04-29 17:04:10,535 I 17948 17948] io_service_pool.cc:47: IOServicePool is stopped. | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-755b86fbf55e07783e55601fb5fd007886c9a3cdd3dedcf8f80fbd78_13242.log <== | |
| [2021-04-29 17:04:06,149 I 13242 15018] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,185 E 13242 13242] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:06,199 I 13242 15018] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,830 E 13242 13242] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:08,982 I 13242 13242] direct_task_transport.cc:47: Creating actor via GCS actor id = : 60a4818707d1fbd55cf72e4a01000000 | |
| [2021-04-29 17:04:08,997 I 13242 15018] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 60a4818707d1fbd55cf72e4a01000000, ip address: 10.134.61.25, port: 9186, worker_id: 40d274b77c09655d4ba1634adbc81325b222d1c1edfe94f32433e08b, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:20,882 I 13242 15018] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 60a4818707d1fbd55cf72e4a01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,401 I 13242 15018] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 60a4818707d1fbd55cf72e4a01000000, ip address: 10.134.61.25, port: 9186, worker_id: 40d274b77c09655d4ba1634adbc81325b222d1c1edfe94f32433e08b, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,516 E 13242 13242] actor_manager.cc:93: Actor handle already exists 60a4818707d1fbd55cf72e4a01000000 | |
| [2021-04-29 17:04:29,530 I 13242 13242] core_worker.cc:2064: Finished executing task fd0e95cef7af4fbdffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-75665701eb5f210cdc45bb005d17fbd735b78dac12cfa0b2cf03c91e_17053.log <== | |
| [2021-04-29 17:04:09,293 I 17053 18694] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,435 E 17053 17053] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,460 I 17053 18694] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,488 E 17053 17053] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,588 I 17053 17053] direct_task_transport.cc:47: Creating actor via GCS actor id = : ef22a69ad8af9d329ba30d6601000000 | |
| [2021-04-29 17:04:11,625 I 17053 18694] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: ef22a69ad8af9d329ba30d6601000000, ip address: 10.134.60.23, port: 9188, worker_id: 5491b9799f1c81289a68b8b07a97ef9f1a8096de040cd3380e7863b3, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,383 I 17053 18694] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: ef22a69ad8af9d329ba30d6601000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:25,489 I 17053 18694] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: ef22a69ad8af9d329ba30d6601000000, ip address: 10.134.60.23, port: 9188, worker_id: 5491b9799f1c81289a68b8b07a97ef9f1a8096de040cd3380e7863b3, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,431 E 17053 17053] actor_manager.cc:93: Actor handle already exists ef22a69ad8af9d329ba30d6601000000 | |
| [2021-04-29 17:04:29,836 I 17053 17053] core_worker.cc:2064: Finished executing task 55bd5e572e95abacffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-75a184f936101916bf29c20693bd3e56ff69d18672133dda6dff4fcd_3994.log <== | |
| [2021-04-29 17:03:46,428 I 3994 3994] core_worker.cc:2064: Finished executing task 884c5d078d7d29ed4c81bec4835146d698f0319101000000, status=OK | |
| [2021-04-29 17:04:48,754 I 3994 3994] core_worker.cc:2064: Finished executing task f8b0232db65342784c81bec4835146d698f0319101000000, status=OK | |
| [2021-04-29 17:04:49,191 I 3994 3994] core_worker.cc:2064: Finished executing task 32ed0f1a293a35364c81bec4835146d698f0319101000000, status=OK | |
| [2021-04-29 17:04:51,156 I 3994 3994] core_worker.cc:2064: Finished executing task 5410abe3617c4b174c81bec4835146d698f0319101000000, status=OK | |
| [2021-04-29 17:04:52,378 I 3994 3994] core_worker.cc:2064: Finished executing task ab35ac58a2fd60c04c81bec4835146d698f0319101000000, status=OK | |
| [2021-04-29 17:04:52,810 I 3994 3994] core_worker.cc:2064: Finished executing task 9ad47627d74d20624c81bec4835146d698f0319101000000, status=OK | |
| [2021-04-29 17:04:55,414 I 3994 3994] core_worker.cc:2064: Finished executing task 368702add0fb1ff44c81bec4835146d698f0319101000000, status=OK | |
| [2021-04-29 17:04:56,303 I 3994 3994] core_worker.cc:2064: Finished executing task 9087e065b737fa364c81bec4835146d698f0319101000000, status=OK | |
| [2021-04-29 17:04:56,602 I 3994 3994] core_worker.cc:2064: Finished executing task 6227bb844bb779404c81bec4835146d698f0319101000000, status=OK | |
| [2021-04-29 17:04:57,724 I 3994 3994] core_worker.cc:2064: Finished executing task c4c6080e4c07f6834c81bec4835146d698f0319101000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-75dd66ffa6ada6305d3f32841cca4aafefe98d51ec487fb81edafce2_13368.log <== | |
| [2021-04-29 17:04:06,008 I 13368 15200] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,027 E 13368 13368] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:06,041 I 13368 15200] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,787 E 13368 13368] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:08,919 I 13368 13368] direct_task_transport.cc:47: Creating actor via GCS actor id = : c13f785510fb01cbc091d04301000000 | |
| [2021-04-29 17:04:08,967 I 13368 15200] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: c13f785510fb01cbc091d04301000000, ip address: 10.134.60.23, port: 9173, worker_id: 035b1676fdd764e4a31c22382dcafc479342d86e889485628f4efd67, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:20,155 I 13368 15200] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: c13f785510fb01cbc091d04301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,400 I 13368 15200] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: c13f785510fb01cbc091d04301000000, ip address: 10.134.60.23, port: 9173, worker_id: 035b1676fdd764e4a31c22382dcafc479342d86e889485628f4efd67, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,502 E 13368 13368] actor_manager.cc:93: Actor handle already exists c13f785510fb01cbc091d04301000000 | |
| [2021-04-29 17:04:29,535 I 13368 13368] core_worker.cc:2064: Finished executing task 115002606fa12715ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-75f9c51582669c9588e065894608b5ff524b085bf4afbba727986350_10282.log <== | |
| [2021-04-29 17:04:01,739 I 10282 11873] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,740 E 10282 10282] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,741 I 10282 11873] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,630 E 10282 10282] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:02,710 I 10282 10282] direct_task_transport.cc:47: Creating actor via GCS actor id = : 528eb1edf6226b2fd0541f9801000000 | |
| [2021-04-29 17:04:02,716 I 10282 11873] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 528eb1edf6226b2fd0541f9801000000, ip address: 10.134.70.227, port: 9101, worker_id: 3ea413e82184b9f0fb17289d14787a2db8074d8b6bf672447a3d7d69, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,548 I 10282 11873] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 528eb1edf6226b2fd0541f9801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,392 I 10282 11873] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 528eb1edf6226b2fd0541f9801000000, ip address: 10.134.70.227, port: 9101, worker_id: 3ea413e82184b9f0fb17289d14787a2db8074d8b6bf672447a3d7d69, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,160 E 10282 10282] actor_manager.cc:93: Actor handle already exists 528eb1edf6226b2fd0541f9801000000 | |
| [2021-04-29 17:04:29,474 I 10282 10282] core_worker.cc:2064: Finished executing task 867a672cf756cd00ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-760e1147398017f28ca0f93f479876838931aadc030c7e1125b4cc24_7532.log <== | |
| [2021-04-29 17:03:59,122 I 7532 8856] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,140 E 7532 7532] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,155 I 7532 8856] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,641 E 7532 7532] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,700 I 7532 7532] direct_task_transport.cc:47: Creating actor via GCS actor id = : 3854243503aa5aef24f2bba701000000 | |
| [2021-04-29 17:03:59,715 I 7532 8856] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 3854243503aa5aef24f2bba701000000, ip address: 10.134.70.227, port: 9118, worker_id: 41a39bc7f14a3599e629f8456277c2d97bf7b5451cab4808dc6e7042, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,722 I 7532 8856] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 3854243503aa5aef24f2bba701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:19,263 I 7532 8856] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 3854243503aa5aef24f2bba701000000, ip address: 10.134.70.227, port: 9118, worker_id: 41a39bc7f14a3599e629f8456277c2d97bf7b5451cab4808dc6e7042, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,062 E 7532 7532] actor_manager.cc:93: Actor handle already exists 3854243503aa5aef24f2bba701000000 | |
| [2021-04-29 17:04:28,922 I 7532 7532] core_worker.cc:2064: Finished executing task ebb2a5a3aea07a38ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-763628a89491f70faae3eaedaef352dc6225b209b065a8ab46d08f66_5269.log <== | |
| [2021-04-29 17:03:59,602 E 5269 5269] actor_manager.cc:93: Actor handle already exists 91d44eb4358aeb4cdad21a4801000000 | |
| [2021-04-29 17:04:01,781 I 5269 5269] core_worker.cc:2064: Finished executing task a21c4e2c6b7639cbffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:01,793 E 5269 5269] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,793 E 5269 5269] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,863 I 5269 5269] direct_task_transport.cc:47: Creating actor via GCS actor id = : 5b8a76f45c22356e82734ee401000000 | |
| [2021-04-29 17:04:01,873 I 5269 6118] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 5b8a76f45c22356e82734ee401000000, ip address: 10.134.142.44, port: 9107, worker_id: 48b745b81e4517b615d28e7424cde651997531ab1ba62d42152452ef, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,142 I 5269 6118] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 5b8a76f45c22356e82734ee401000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,390 I 5269 6118] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 5b8a76f45c22356e82734ee401000000, ip address: 10.134.142.44, port: 9107, worker_id: 48b745b81e4517b615d28e7424cde651997531ab1ba62d42152452ef, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,063 E 5269 5269] actor_manager.cc:93: Actor handle already exists 5b8a76f45c22356e82734ee401000000 | |
| [2021-04-29 17:04:29,454 I 5269 5269] core_worker.cc:2064: Finished executing task 5a3c9fdbc9a4cda8ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-76c8f74d45633567b081815af58efe25f7153658ffb032882119493f_12933.log <== | |
| [2021-04-29 17:04:05,071 I 12933 14475] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,105 E 12933 12933] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,127 I 12933 14475] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,957 E 12933 12933] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:07,138 I 12933 12933] direct_task_transport.cc:47: Creating actor via GCS actor id = : f5170f945f5f89913b8d8ead01000000 | |
| [2021-04-29 17:04:07,158 I 12933 14475] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: f5170f945f5f89913b8d8ead01000000, ip address: 10.134.112.222, port: 21101, worker_id: NIL_ID, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:16,223 I 12933 14475] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: f5170f945f5f89913b8d8ead01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,398 I 12933 14475] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: f5170f945f5f89913b8d8ead01000000, ip address: 10.134.112.222, port: 9156, worker_id: 35892d976e4a516858332f78a0dd072f88925c2a3b19f306cde03544, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,412 E 12933 12933] actor_manager.cc:93: Actor handle already exists f5170f945f5f89913b8d8ead01000000 | |
| [2021-04-29 17:04:29,520 I 12933 12933] core_worker.cc:2064: Finished executing task 0a500241cd56bf5fffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-77045ae47d427902e5ae93c2264462e7f09bd6b3babe57d73f7a4064_4038.log <== | |
| [2021-04-29 17:04:02,334 E 4038 4038] actor_manager.cc:93: Actor handle already exists 253f749a9e5822b342bc4a2601000000 | |
| [2021-04-29 17:04:10,070 I 4038 4038] core_worker.cc:2064: Finished executing task d6e396d0614da0c3ffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:10,082 E 4038 4038] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,082 E 4038 4038] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,188 I 4038 4038] direct_task_transport.cc:47: Creating actor via GCS actor id = : 0771c3cfc696bf4cd21dfe0b01000000 | |
| [2021-04-29 17:04:10,204 I 4038 4699] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 0771c3cfc696bf4cd21dfe0b01000000, ip address: 10.134.112.222, port: 9206, worker_id: 788e4bdbed6b44662d183d859841b14a7fe74ad13a6821d2b33e3541, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,363 I 4038 4699] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 0771c3cfc696bf4cd21dfe0b01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,756 I 4038 4699] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 0771c3cfc696bf4cd21dfe0b01000000, ip address: 10.134.112.222, port: 9206, worker_id: 788e4bdbed6b44662d183d859841b14a7fe74ad13a6821d2b33e3541, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:24,580 E 4038 4038] actor_manager.cc:93: Actor handle already exists 0771c3cfc696bf4cd21dfe0b01000000 | |
| [2021-04-29 17:04:29,612 I 4038 4038] core_worker.cc:2064: Finished executing task 2e131f2a7423303effffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-771b885f1500868c542e9cc1702b97ca5c8d3625abead6db654ef644_9811.log <== | |
| [2021-04-29 17:04:02,241 I 9811 11362] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,269 E 9811 9811] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:02,277 I 9811 11362] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,363 E 9811 9811] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,447 I 9811 9811] direct_task_transport.cc:47: Creating actor via GCS actor id = : 0148e65126e063ed262baf6701000000 | |
| [2021-04-29 17:04:05,470 I 9811 11362] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 0148e65126e063ed262baf6701000000, ip address: 10.134.70.227, port: 36993, worker_id: NIL_ID, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,815 I 9811 11362] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 0148e65126e063ed262baf6701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,395 I 9811 11362] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 0148e65126e063ed262baf6701000000, ip address: 10.134.70.227, port: 9143, worker_id: e95c645f977679ce6d37019180147f549c083b36f79be7eb4ac5ff5c, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,227 E 9811 9811] actor_manager.cc:93: Actor handle already exists 0148e65126e063ed262baf6701000000 | |
| [2021-04-29 17:04:29,485 I 9811 9811] core_worker.cc:2064: Finished executing task bc3012a2a97ad1d5ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-774dc042ba138743531bdb72892fdcc8834702106e43bb3decc20999_10068.log <== | |
| [2021-04-29 17:04:01,918 I 10068 11785] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,947 E 10068 10068] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,957 I 10068 11785] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,477 E 10068 10068] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:04,577 I 10068 10068] direct_task_transport.cc:47: Creating actor via GCS actor id = : ef5084fe85c2d139fb9507d301000000 | |
| [2021-04-29 17:04:04,593 I 10068 11785] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: ef5084fe85c2d139fb9507d301000000, ip address: 10.134.61.25, port: 9110, worker_id: 01d891ae0f1ff25f24658b5fa863a49159f7ffe195284fce7eb8beab, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,117 I 10068 11785] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: ef5084fe85c2d139fb9507d301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,392 I 10068 11785] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: ef5084fe85c2d139fb9507d301000000, ip address: 10.134.61.25, port: 9110, worker_id: 01d891ae0f1ff25f24658b5fa863a49159f7ffe195284fce7eb8beab, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,724 E 10068 10068] actor_manager.cc:93: Actor handle already exists ef5084fe85c2d139fb9507d301000000 | |
| [2021-04-29 17:04:29,540 I 10068 10068] core_worker.cc:2064: Finished executing task 2e44b1c5332acb16ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-77fa6898b67d94ecd09d7e179d37e721a7bc8feecd50dcdfaa4f01cf_17892.log <== | |
| [2021-04-29 17:04:09,765 I 17892 19461] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,870 E 17892 17892] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,892 I 17892 19461] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,578 E 17892 17892] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,717 I 17892 17892] direct_task_transport.cc:47: Creating actor via GCS actor id = : 4367e6650f05178c232f59bc01000000 | |
| [2021-04-29 17:04:11,754 I 17892 19461] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 4367e6650f05178c232f59bc01000000, ip address: 10.134.70.227, port: 9197, worker_id: f982698d3cf8fafc89dc23c5515182e69051ef4870cb105d4b091d3a, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,387 I 17892 19461] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 4367e6650f05178c232f59bc01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:28,215 I 17892 19461] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 4367e6650f05178c232f59bc01000000, ip address: 10.134.70.227, port: 9197, worker_id: f982698d3cf8fafc89dc23c5515182e69051ef4870cb105d4b091d3a, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:29,020 E 17892 17892] actor_manager.cc:93: Actor handle already exists 4367e6650f05178c232f59bc01000000 | |
| [2021-04-29 17:04:29,858 I 17892 17892] core_worker.cc:2064: Finished executing task ea76fb022a987be9ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-787690366e78d20f1930f7682c7cf1a43eb5b029acb46a58c10368ce_7491.log <== | |
| [2021-04-29 17:03:59,342 I 7491 8533] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,358 E 7491 7491] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,363 I 7491 8533] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,734 E 7491 7491] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,787 I 7491 7491] direct_task_transport.cc:47: Creating actor via GCS actor id = : 87f22b2ef9e1a0ef8a7132a201000000 | |
| [2021-04-29 17:03:59,813 I 7491 8533] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 87f22b2ef9e1a0ef8a7132a201000000, ip address: 10.134.70.227, port: 9116, worker_id: e6c406bc5e9b2d47b81e8679d0e43cd736ca2eba42f985677e11c082, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,725 I 7491 8533] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 87f22b2ef9e1a0ef8a7132a201000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,167 I 7491 8533] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 87f22b2ef9e1a0ef8a7132a201000000, ip address: 10.134.70.227, port: 9116, worker_id: e6c406bc5e9b2d47b81e8679d0e43cd736ca2eba42f985677e11c082, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,861 E 7491 7491] actor_manager.cc:93: Actor handle already exists 87f22b2ef9e1a0ef8a7132a201000000 | |
| [2021-04-29 17:04:29,392 I 7491 7491] core_worker.cc:2064: Finished executing task dc43564890144617ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-79ec2a0c54a5a27d310ed95949d54e2f06729cab44a7cb9b504acc2f_13366.log <== | |
| [2021-04-29 17:04:06,074 I 13366 15067] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,105 E 13366 13366] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:06,117 I 13366 15067] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,804 E 13366 13366] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:08,960 I 13366 13366] direct_task_transport.cc:47: Creating actor via GCS actor id = : 1e29fe7d608602ed8c02847201000000 | |
| [2021-04-29 17:04:08,967 I 13366 15067] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 1e29fe7d608602ed8c02847201000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:20,211 I 13366 15067] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 1e29fe7d608602ed8c02847201000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,401 I 13366 15067] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1e29fe7d608602ed8c02847201000000, ip address: 10.134.112.222, port: 9184, worker_id: 812cd7b998bf8858ce3d00bb61f63bd99a0c68c9a2412127b635de80, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,495 E 13366 13366] actor_manager.cc:93: Actor handle already exists 1e29fe7d608602ed8c02847201000000 | |
| [2021-04-29 17:04:29,537 I 13366 13366] core_worker.cc:2064: Finished executing task d8a4407bb8899828ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-7a8ee0fcdfd289a671eb68c429b70e9bf60bcbafa0c9d91545a6305c_15957.log <== | |
| [2021-04-29 17:04:08,252 I 15957 17324] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,418 E 15957 15957] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,444 I 15957 17324] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,030 E 15957 15957] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,133 I 15957 15957] direct_task_transport.cc:47: Creating actor via GCS actor id = : f5fa1c19ff4198b48cd8685b01000000 | |
| [2021-04-29 17:04:11,171 I 15957 17324] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: f5fa1c19ff4198b48cd8685b01000000, ip address: 10.134.142.44, port: 9208, worker_id: 9240ecb8a5979981567075de19053afe087d3eef2b16d9096e7890bb, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,373 I 15957 17324] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: f5fa1c19ff4198b48cd8685b01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:25,550 I 15957 17324] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: f5fa1c19ff4198b48cd8685b01000000, ip address: 10.134.142.44, port: 9208, worker_id: 9240ecb8a5979981567075de19053afe087d3eef2b16d9096e7890bb, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,444 E 15957 15957] actor_manager.cc:93: Actor handle already exists f5fa1c19ff4198b48cd8685b01000000 | |
| [2021-04-29 17:04:29,846 I 15957 15957] core_worker.cc:2064: Finished executing task 6290035395daaea0ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-7a9742e34bb20013747b6cb9b4d8a26296278f472f60644694c194af_9923.log <== | |
| [2021-04-29 17:04:02,065 I 9923 11610] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,100 E 9923 9923] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:02,106 I 9923 11610] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,967 E 9923 9923] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,043 I 9923 9923] direct_task_transport.cc:47: Creating actor via GCS actor id = : 5e9b2f27e77cd71873cee92601000000 | |
| [2021-04-29 17:04:05,070 I 9923 11610] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 5e9b2f27e77cd71873cee92601000000, ip address: 10.134.61.25, port: 9109, worker_id: 8511d16e12a729b4e32734b96dd9d08cafa8300a54480c10629192cf, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,206 I 9923 11610] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 5e9b2f27e77cd71873cee92601000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,393 I 9923 11610] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 5e9b2f27e77cd71873cee92601000000, ip address: 10.134.61.25, port: 9109, worker_id: 8511d16e12a729b4e32734b96dd9d08cafa8300a54480c10629192cf, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,174 E 9923 9923] actor_manager.cc:93: Actor handle already exists 5e9b2f27e77cd71873cee92601000000 | |
| [2021-04-29 17:04:29,470 I 9923 9923] core_worker.cc:2064: Finished executing task d53d186424969303ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-7b26a86c420c6c8e7b46626f28c79f0041050f71d1af5516a31d84ff_12874.log <== | |
| [2021-04-29 17:04:05,000 I 12874 14214] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,062 E 12874 12874] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,070 I 12874 14214] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,881 E 12874 12874] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:07,045 I 12874 12874] direct_task_transport.cc:47: Creating actor via GCS actor id = : 6a9241a4869a6acc9c85d26501000000 | |
| [2021-04-29 17:04:07,112 I 12874 14214] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 6a9241a4869a6acc9c85d26501000000, ip address: 10.134.61.25, port: 17901, worker_id: NIL_ID, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:15,026 I 12874 14214] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 6a9241a4869a6acc9c85d26501000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,397 I 12874 14214] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 6a9241a4869a6acc9c85d26501000000, ip address: 10.134.61.25, port: 9157, worker_id: 8dc5ea9a40f19a4c9ac1dd8fa773fa50cc2b6ef59d1fb6bea88c69e2, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,368 E 12874 12874] actor_manager.cc:93: Actor handle already exists 6a9241a4869a6acc9c85d26501000000 | |
| [2021-04-29 17:04:29,516 I 12874 12874] core_worker.cc:2064: Finished executing task 72d8586bdb862ce2ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-7b7a887fff532bd6a051dc8d07e8249b88c42cb268b8dd3d6bece20e_5314.log <== | |
| [2021-04-29 17:03:57,938 E 5314 5314] actor_manager.cc:93: Actor handle already exists 70b1c00380612932fc2420ea01000000 | |
| [2021-04-29 17:03:59,629 I 5314 5314] core_worker.cc:2064: Finished executing task 6c1b551520e4e0d5ffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:03:59,634 E 5314 5314] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,634 E 5314 5314] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,696 I 5314 5314] direct_task_transport.cc:47: Creating actor via GCS actor id = : f259169cadfaf98d05c3aa0101000000 | |
| [2021-04-29 17:03:59,700 I 5314 6367] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: f259169cadfaf98d05c3aa0101000000, ip address: 10.134.70.227, port: 9119, worker_id: d98dc141633a5d9ace74e522e261dc18b2c7f23888c0327882ce4013, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,721 I 5314 6367] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: f259169cadfaf98d05c3aa0101000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:19,016 I 5314 6367] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: f259169cadfaf98d05c3aa0101000000, ip address: 10.134.70.227, port: 9119, worker_id: d98dc141633a5d9ace74e522e261dc18b2c7f23888c0327882ce4013, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:21,911 E 5314 5314] actor_manager.cc:93: Actor handle already exists f259169cadfaf98d05c3aa0101000000 | |
| [2021-04-29 17:04:28,868 I 5314 5314] core_worker.cc:2064: Finished executing task d176c7b1c88b44a2ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-7bbfa0dd2ae5f0526fa942fa76b87e31a533cd68b30d29327fe280de_16009.log <== | |
| [2021-04-29 17:04:08,477 I 16009 17686] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,582 E 16009 16009] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,624 I 16009 17686] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,124 E 16009 16009] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,222 I 16009 16009] direct_task_transport.cc:47: Creating actor via GCS actor id = : a7da9ce267b58ce72b5de7b801000000 | |
| [2021-04-29 17:04:11,273 I 16009 17686] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: a7da9ce267b58ce72b5de7b801000000, ip address: 10.134.60.23, port: 9196, worker_id: 70c288a8e3e5fb18dc2dbb3a9d739f3147b54704735c83e20c547718, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,376 I 16009 17686] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: a7da9ce267b58ce72b5de7b801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:24,865 I 16009 17686] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: a7da9ce267b58ce72b5de7b801000000, ip address: 10.134.60.23, port: 9196, worker_id: 70c288a8e3e5fb18dc2dbb3a9d739f3147b54704735c83e20c547718, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:27,596 E 16009 16009] actor_manager.cc:93: Actor handle already exists a7da9ce267b58ce72b5de7b801000000 | |
| [2021-04-29 17:04:29,817 I 16009 16009] core_worker.cc:2064: Finished executing task 5902213b518f860dffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-7c2048ee145f71688ca7eb51503897a3c5f1ca29c7076995fed1796b_16611.log <== | |
| [2021-04-29 17:04:09,293 I 16611 18699] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,434 E 16611 16611] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,460 I 16611 18699] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,490 E 16611 16611] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,586 I 16611 16611] direct_task_transport.cc:47: Creating actor via GCS actor id = : 2a17566d4fa623dea292f53b01000000 | |
| [2021-04-29 17:04:11,625 I 16611 18699] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 2a17566d4fa623dea292f53b01000000, ip address: 10.134.61.130, port: 9208, worker_id: 39de32f8b5bf722d9e30117ec837ec6d3e84e81c2716f96f809ee07d, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,382 I 16611 18699] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 2a17566d4fa623dea292f53b01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:27,754 I 16611 18699] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 2a17566d4fa623dea292f53b01000000, ip address: 10.134.61.130, port: 9208, worker_id: 39de32f8b5bf722d9e30117ec837ec6d3e84e81c2716f96f809ee07d, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,969 E 16611 16611] actor_manager.cc:93: Actor handle already exists 2a17566d4fa623dea292f53b01000000 | |
| [2021-04-29 17:04:29,859 I 16611 16611] core_worker.cc:2064: Finished executing task 85ad47e25c2e38bbffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-7c5505daf0dc3807ec457c70ae0260ffa885e0195fbdd6998a2b1eed_18052.log <== | |
| [2021-04-29 17:04:10,044 I 18052 19923] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,119 E 18052 18052] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,142 I 18052 19923] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,594 E 18052 18052] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,703 I 18052 18052] direct_task_transport.cc:47: Creating actor via GCS actor id = : ef70ab75ac88134afd63ec9201000000 | |
| [2021-04-29 17:04:11,715 I 18052 19923] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: ef70ab75ac88134afd63ec9201000000, ip address: 10.133.95.40, port: 9187, worker_id: 35e25ba9540d6edc290e19080bb29b81e3656c39b05dceecd7866ad9, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,385 I 18052 19923] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: ef70ab75ac88134afd63ec9201000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:27,383 I 18052 19923] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: ef70ab75ac88134afd63ec9201000000, ip address: 10.133.95.40, port: 9187, worker_id: 35e25ba9540d6edc290e19080bb29b81e3656c39b05dceecd7866ad9, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,852 E 18052 18052] actor_manager.cc:93: Actor handle already exists ef70ab75ac88134afd63ec9201000000 | |
| [2021-04-29 17:04:29,856 I 18052 18052] core_worker.cc:2064: Finished executing task b26ddc2b32416ae3ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-7c980b39e393d91bf274eb177ebd17c90e225aeaab82dca7d1aa55a8_18041.log <== | |
| [2021-04-29 17:04:09,968 I 18041 19765] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,022 E 18041 18041] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,043 I 18041 19765] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,587 E 18041 18041] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,702 I 18041 18041] direct_task_transport.cc:47: Creating actor via GCS actor id = : 3fb04a099965634091c41bbe01000000 | |
| [2021-04-29 17:04:11,715 I 18041 19765] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 3fb04a099965634091c41bbe01000000, ip address: 10.134.61.130, port: 9203, worker_id: 181eb2c370cef0c607b697b86d4511d8f176cbf2744b4504ac2885f8, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,385 I 18041 19765] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 3fb04a099965634091c41bbe01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:28,332 I 18041 19765] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 3fb04a099965634091c41bbe01000000, ip address: 10.134.61.130, port: 9203, worker_id: 181eb2c370cef0c607b697b86d4511d8f176cbf2744b4504ac2885f8, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:29,046 E 18041 18041] actor_manager.cc:93: Actor handle already exists 3fb04a099965634091c41bbe01000000 | |
| [2021-04-29 17:04:29,861 I 18041 18041] core_worker.cc:2064: Finished executing task f53ce05d9b36dd88ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-7cf854968d3fc97a704fe222455d2d7ac632df91e0b4797bcad42ce8_15984.log <== | |
| [2021-04-29 17:04:08,414 I 15984 17622] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,506 E 15984 15984] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:08,537 I 15984 17622] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,053 E 15984 15984] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,176 I 15984 15984] direct_task_transport.cc:47: Creating actor via GCS actor id = : 96186dd467f7c8d485f9c88301000000 | |
| [2021-04-29 17:04:11,194 I 15984 17622] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 96186dd467f7c8d485f9c88301000000, ip address: 10.134.142.44, port: 9205, worker_id: 5be70b53511c9494715aeec5b3543250c8367cf2f29fed5cda6fc34c, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,376 I 15984 17622] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 96186dd467f7c8d485f9c88301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:25,691 I 15984 17622] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 96186dd467f7c8d485f9c88301000000, ip address: 10.134.142.44, port: 9205, worker_id: 5be70b53511c9494715aeec5b3543250c8367cf2f29fed5cda6fc34c, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,467 E 15984 15984] actor_manager.cc:93: Actor handle already exists 96186dd467f7c8d485f9c88301000000 | |
| [2021-04-29 17:04:29,843 I 15984 15984] core_worker.cc:2064: Finished executing task 93d5586527d43572ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-7e0b77eb4f77b44c5fdf680d838cd8ca1f5734499be23365bbcd503e_5329.log <== | |
| [2021-04-29 17:03:46,412 I 5329 5329] core_worker.cc:2064: Finished executing task e9912f406afaa84768b1def8388b13532f93b7ab01000000, status=OK | |
| [2021-04-29 17:04:37,237 I 5329 5329] core_worker.cc:2064: Finished executing task 7b96182e50ec53d168b1def8388b13532f93b7ab01000000, status=OK | |
| [2021-04-29 17:04:49,101 I 5329 5329] core_worker.cc:2064: Finished executing task f3cd981d3fd7dc3a68b1def8388b13532f93b7ab01000000, status=OK | |
| [2021-04-29 17:04:50,262 I 5329 5329] core_worker.cc:2064: Finished executing task e4717a4612bc7ac068b1def8388b13532f93b7ab01000000, status=OK | |
| [2021-04-29 17:04:50,476 I 5329 5329] core_worker.cc:2064: Finished executing task c183bbb5264d9eff68b1def8388b13532f93b7ab01000000, status=OK | |
| [2021-04-29 17:04:52,311 I 5329 5329] core_worker.cc:2064: Finished executing task 544f80235c5124e668b1def8388b13532f93b7ab01000000, status=OK | |
| [2021-04-29 17:04:53,516 I 5329 5329] core_worker.cc:2064: Finished executing task ab63ae999869cac168b1def8388b13532f93b7ab01000000, status=OK | |
| [2021-04-29 17:04:54,522 I 5329 5329] core_worker.cc:2064: Finished executing task c3f26e2281f0766668b1def8388b13532f93b7ab01000000, status=OK | |
| [2021-04-29 17:04:56,003 I 5329 5329] core_worker.cc:2064: Finished executing task d8c87ec19ee61d8b68b1def8388b13532f93b7ab01000000, status=OK | |
| [2021-04-29 17:04:57,940 I 5329 5329] core_worker.cc:2064: Finished executing task 9720fe2b4e03f22368b1def8388b13532f93b7ab01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-7e3a2ade467e3654b53f294ed5cafa534450b1cd6d76201ce79482b4_9759.log <== | |
| [2021-04-29 17:04:02,403 I 9759 11165] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,422 E 9759 9759] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:02,429 I 9759 11165] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,425 E 9759 9759] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,507 I 9759 9759] direct_task_transport.cc:47: Creating actor via GCS actor id = : 5db98e6404dd43910bc40c2f01000000 | |
| [2021-04-29 17:04:05,516 I 9759 11165] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 5db98e6404dd43910bc40c2f01000000, ip address: 10.134.142.44, port: 12853, worker_id: NIL_ID, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,757 I 9759 11165] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 5db98e6404dd43910bc40c2f01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,396 I 9759 11165] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 5db98e6404dd43910bc40c2f01000000, ip address: 10.134.142.44, port: 9145, worker_id: 27ff7162a383182719fc46db13f1c46794cb74074183cc6ad0501908, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,303 E 9759 9759] actor_manager.cc:93: Actor handle already exists 5db98e6404dd43910bc40c2f01000000 | |
| [2021-04-29 17:04:29,493 I 9759 9759] core_worker.cc:2064: Finished executing task 6fe89f40d72b8de8ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-7e3c0b8931f573f9b3dce639e81cc5e3ba5ccba6e4a5f30c55a38e50_12930.log <== | |
| [2021-04-29 17:04:05,167 I 12930 14552] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,180 E 12930 12930] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:05,185 I 12930 14552] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:07,363 E 12930 12930] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:07,470 I 12930 12930] direct_task_transport.cc:47: Creating actor via GCS actor id = : 6bd54744e58fc8f30b238f2c01000000 | |
| [2021-04-29 17:04:07,480 I 12930 14552] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 6bd54744e58fc8f30b238f2c01000000, ip address: 10.134.142.16, port: 22629, worker_id: NIL_ID, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:17,054 I 12930 14552] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 6bd54744e58fc8f30b238f2c01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,400 I 12930 14552] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 6bd54744e58fc8f30b238f2c01000000, ip address: 10.134.142.16, port: 9166, worker_id: 0c9cf23f7a73eb59aff23558a24a1ba3ceb31d5b3aeaa2b621db8e8f, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,462 E 12930 12930] actor_manager.cc:93: Actor handle already exists 6bd54744e58fc8f30b238f2c01000000 | |
| [2021-04-29 17:04:29,525 I 12930 12930] core_worker.cc:2064: Finished executing task cc97be457ddcb240ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-7e48e3d9d60c20b6d04734ec8747efa39f446a13ef58e3bd8c4cec72_18031.log <== | |
| [2021-04-29 17:04:10,021 I 18031 19916] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,119 E 18031 18031] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,142 I 18031 19916] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,594 E 18031 18031] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,702 I 18031 18031] direct_task_transport.cc:47: Creating actor via GCS actor id = : 0139bb0623ccec36cc5be81301000000 | |
| [2021-04-29 17:04:11,716 I 18031 19916] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 0139bb0623ccec36cc5be81301000000, ip address: 10.134.61.25, port: 9184, worker_id: 801a9cbb2126f7210334733ccfd94ba25988cf7cdeba59108325524b, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,384 I 18031 19916] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 0139bb0623ccec36cc5be81301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:27,339 I 18031 19916] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 0139bb0623ccec36cc5be81301000000, ip address: 10.134.61.25, port: 9184, worker_id: 801a9cbb2126f7210334733ccfd94ba25988cf7cdeba59108325524b, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,845 E 18031 18031] actor_manager.cc:93: Actor handle already exists 0139bb0623ccec36cc5be81301000000 | |
| [2021-04-29 17:04:29,848 I 18031 18031] core_worker.cc:2064: Finished executing task 7b95dc5f24459d86ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-7e76c84f25293cf2cce22bc192b0d8f38957717d49ec143bce553257_8036.log <== | |
| [2021-04-29 17:03:58,593 I 8036 9567] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,594 E 8036 8036] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,595 I 8036 9567] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,045 E 8036 8036] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,116 I 8036 8036] direct_task_transport.cc:47: Creating actor via GCS actor id = : 3906fb8470b529a1ed9f93fe01000000 | |
| [2021-04-29 17:03:59,121 I 8036 9567] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 3906fb8470b529a1ed9f93fe01000000, ip address: 10.134.70.227, port: 9123, worker_id: 197747f741e5461dbc1c048a755c171d6b02aa0295a5e407386a36a5, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,033 I 8036 9567] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 3906fb8470b529a1ed9f93fe01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,218 I 8036 9567] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 3906fb8470b529a1ed9f93fe01000000, ip address: 10.134.70.227, port: 9123, worker_id: 197747f741e5461dbc1c048a755c171d6b02aa0295a5e407386a36a5, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:07,574 E 8036 8036] actor_manager.cc:93: Actor handle already exists 3906fb8470b529a1ed9f93fe01000000 | |
| [2021-04-29 17:04:12,760 I 8036 8036] core_worker.cc:2064: Finished executing task b1c402be9f805c40ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-7ea0e0db803477675e913222ae3f4713b1b63461991c695af2f1d61f_8049.log <== | |
| [2021-04-29 17:03:58,772 I 8049 9399] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,811 E 8049 8049] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,818 I 8049 9399] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,500 E 8049 8049] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,548 I 8049 8049] direct_task_transport.cc:47: Creating actor via GCS actor id = : ab0a1eeb5ec614b1ddebb61c01000000 | |
| [2021-04-29 17:03:59,557 I 8049 9399] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: ab0a1eeb5ec614b1ddebb61c01000000, ip address: 10.134.142.44, port: 9121, worker_id: 7f37c51416b27a76b333a0739939ced9d4a472a9de74a4e40dffba5c, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,439 I 8049 9399] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: ab0a1eeb5ec614b1ddebb61c01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:13,679 I 8049 9399] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: ab0a1eeb5ec614b1ddebb61c01000000, ip address: 10.134.142.44, port: 9121, worker_id: 7f37c51416b27a76b333a0739939ced9d4a472a9de74a4e40dffba5c, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:15,413 E 8049 8049] actor_manager.cc:93: Actor handle already exists ab0a1eeb5ec614b1ddebb61c01000000 | |
| [2021-04-29 17:04:22,310 I 8049 8049] core_worker.cc:2064: Finished executing task b0ef3c2fd910dd4bffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-7ecb5317c1b8194e06b91225f8ae6b9d143c8df64115ef7aaf993108_9781.log <== | |
| [2021-04-29 17:04:02,476 I 9781 11118] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,486 E 9781 9781] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:02,493 I 9781 11118] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,443 E 9781 9781] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,525 I 9781 9781] direct_task_transport.cc:47: Creating actor via GCS actor id = : ef2d7bd2573c2c9eb262f39701000000 | |
| [2021-04-29 17:04:05,530 I 9781 11118] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: ef2d7bd2573c2c9eb262f39701000000, ip address: 10.134.61.130, port: 13217, worker_id: NIL_ID, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:07,421 I 9781 11118] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: ef2d7bd2573c2c9eb262f39701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,395 I 9781 11118] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: ef2d7bd2573c2c9eb262f39701000000, ip address: 10.134.61.130, port: 9145, worker_id: 819066ebe914e5878a6fd9598bcda6c46d2c3f5407935715bd663356, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,266 E 9781 9781] actor_manager.cc:93: Actor handle already exists ef2d7bd2573c2c9eb262f39701000000 | |
| [2021-04-29 17:04:29,486 I 9781 9781] core_worker.cc:2064: Finished executing task 54b6046ab7ab2213ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-7ecdc31bcb00f1a189e7ffeebdec4e3e08766abcc7904c92ee2be14a_4020.log <== | |
| [2021-04-29 17:03:47,249 E 4020 4020] actor_manager.cc:93: Actor handle already exists ff125b7bd2b8fb7149bc306301000000 | |
| [2021-04-29 17:03:47,350 I 4020 4020] core_worker.cc:2064: Finished executing task ffffffffffffffff93053e1e8b952ade2d02457601000000, status=OK | |
| [2021-04-29 17:03:47,350 I 4020 4020] direct_actor_transport.cc:497: Actor creation task finished, task_id: ffffffffffffffff93053e1e8b952ade2d02457601000000, actor_id: 93053e1e8b952ade2d02457601000000 | |
| [2021-04-29 17:03:47,352 I 4020 4339] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 93053e1e8b952ade2d02457601000000, ip address: 10.133.100.138, port: 9113, worker_id: 7ecdc31bcb00f1a189e7ffeebdec4e3e08766abcc7904c92ee2be14a, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:47,352 I 4020 4020] core_worker.cc:2064: Finished executing task 5b2ae4dc536c0f7393053e1e8b952ade2d02457601000000, status=OK | |
| [2021-04-29 17:03:47,392 I 4020 4020] core_worker.cc:2064: Finished executing task 69e5ce51a165f1b993053e1e8b952ade2d02457601000000, status=OK | |
| [2021-04-29 17:03:47,528 I 4020 4020] core_worker.cc:2064: Finished executing task 7136b63f141502fc93053e1e8b952ade2d02457601000000, status=OK | |
| [2021-04-29 17:03:47,714 I 4020 4020] core_worker.cc:2064: Finished executing task f915572935384e2193053e1e8b952ade2d02457601000000, status=OK | |
| [2021-04-29 17:03:47,724 I 4020 4020] core_worker.cc:2064: Finished executing task 3f3b0bba3b58393f93053e1e8b952ade2d02457601000000, status=OK | |
| [2021-04-29 17:03:51,794 I 4020 4020] core_worker.cc:2064: Finished executing task f903125f2cd0ae9d93053e1e8b952ade2d02457601000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-7eda358a14e7aea68d575c5956e26f775e2dc3e7ec9c702ee104b22e_5408.log <== | |
| [2021-04-29 17:03:46,133 E 5408 5408] actor_manager.cc:93: Actor handle already exists ff125b7bd2b8fb7149bc306301000000 | |
| [2021-04-29 17:03:46,235 I 5408 5408] core_worker.cc:2064: Finished executing task ffffffffffffffffe36f6f96925e67164dad10d301000000, status=OK | |
| [2021-04-29 17:03:46,235 I 5408 5408] direct_actor_transport.cc:497: Actor creation task finished, task_id: ffffffffffffffffe36f6f96925e67164dad10d301000000, actor_id: e36f6f96925e67164dad10d301000000 | |
| [2021-04-29 17:03:46,237 I 5408 7049] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: e36f6f96925e67164dad10d301000000, ip address: 10.133.100.138, port: 9243, worker_id: 7eda358a14e7aea68d575c5956e26f775e2dc3e7ec9c702ee104b22e, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:46,238 I 5408 5408] core_worker.cc:2064: Finished executing task 257d70f3f8e8061ae36f6f96925e67164dad10d301000000, status=OK | |
| [2021-04-29 17:03:46,247 I 5408 5408] core_worker.cc:2064: Finished executing task 116766671183d135e36f6f96925e67164dad10d301000000, status=OK | |
| [2021-04-29 17:03:46,270 I 5408 5408] core_worker.cc:2064: Finished executing task 3a08c80f54658cffe36f6f96925e67164dad10d301000000, status=OK | |
| [2021-04-29 17:03:46,594 I 5408 5408] core_worker.cc:2064: Finished executing task 5ffe1356db76944ce36f6f96925e67164dad10d301000000, status=OK | |
| [2021-04-29 17:03:46,604 I 5408 5408] core_worker.cc:2064: Finished executing task ef6a0e10ecca4e28e36f6f96925e67164dad10d301000000, status=OK | |
| [2021-04-29 17:03:51,099 I 5408 5408] core_worker.cc:2064: Finished executing task 83e7bdfe47a8b43be36f6f96925e67164dad10d301000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-7f91506c18f12c238de2930f20f122175ea5f5625ad1351a5d24f268_17474.log <== | |
| [2021-04-29 17:04:09,113 I 17474 19089] service_based_accessor.cc:579: Received notification for node id = ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, IsAlive = 1 | |
| [2021-04-29 17:04:09,113 I 17474 19089] service_based_accessor.cc:579: Received notification for node id = d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, IsAlive = 1 | |
| [2021-04-29 17:04:09,113 I 17474 19089] service_based_accessor.cc:579: Received notification for node id = b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, IsAlive = 1 | |
| [2021-04-29 17:04:09,113 I 17474 19089] service_based_accessor.cc:579: Received notification for node id = 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, IsAlive = 1 | |
| [2021-04-29 17:04:09,113 I 17474 19089] service_based_accessor.cc:579: Received notification for node id = 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, IsAlive = 1 | |
| [2021-04-29 17:04:10,131 I 17474 19089] core_worker.cc:717: Exit signal received, this process will exit after all outstanding tasks have finished, exit_type=INTENDED_EXIT | |
| [2021-04-29 17:04:10,131 I 17474 17474] raylet_client.cc:129: RayletClient::Disconnect, exit_type=INTENDED_EXIT, has creation_task_exception_pb_bytes=0 | |
| [2021-04-29 17:04:10,131 I 17474 17474] core_worker.cc:308: Removed worker 7f91506c18f12c238de2930f20f122175ea5f5625ad1351a5d24f268 | |
| [2021-04-29 17:04:10,134 I 17474 17474] core_worker.cc:181: Destructing CoreWorkerProcess. pid: 17474 | |
| [2021-04-29 17:04:10,134 I 17474 17474] io_service_pool.cc:47: IOServicePool is stopped. | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-801b77d3488a3f90c7cd424ca272a41f720d25581a77fd89dfd33d9d_9657.log <== | |
| [2021-04-29 17:04:00,944 I 9657 10004] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:00,981 E 9657 9657] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,008 I 9657 10004] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,765 E 9657 9657] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:01,827 I 9657 9657] direct_task_transport.cc:47: Creating actor via GCS actor id = : f33aebd3089a650b3b598a6c01000000 | |
| [2021-04-29 17:04:01,843 I 9657 10004] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: f33aebd3089a650b3b598a6c01000000, ip address: 10.134.61.25, port: 9120, worker_id: 88f850d417f1e3db0cf2faa36f0e706421b43c7ea0b5e7619460d5eb, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,226 I 9657 10004] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: f33aebd3089a650b3b598a6c01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,390 I 9657 10004] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: f33aebd3089a650b3b598a6c01000000, ip address: 10.134.61.25, port: 9120, worker_id: 88f850d417f1e3db0cf2faa36f0e706421b43c7ea0b5e7619460d5eb, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,027 E 9657 9657] actor_manager.cc:93: Actor handle already exists f33aebd3089a650b3b598a6c01000000 | |
| [2021-04-29 17:04:29,447 I 9657 9657] core_worker.cc:2064: Finished executing task 2a1db3a88993e624ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-8070f7019a20ef6da52599fd341ebfde10377b4201049a45b1d4d80f_7538.log <== | |
| [2021-04-29 17:03:59,122 I 7538 8841] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,140 E 7538 7538] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,155 I 7538 8841] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,642 E 7538 7538] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,700 I 7538 7538] direct_task_transport.cc:47: Creating actor via GCS actor id = : 6791ea75608808f42b66e14a01000000 | |
| [2021-04-29 17:03:59,715 I 7538 8841] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 6791ea75608808f42b66e14a01000000, ip address: 10.134.142.44, port: 9119, worker_id: 94a8ba762aa31f7832294b94a3941fafc3a301ffb01daf9939a5cfdf, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,722 I 7538 8841] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 6791ea75608808f42b66e14a01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:19,263 I 7538 8841] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 6791ea75608808f42b66e14a01000000, ip address: 10.134.142.44, port: 9119, worker_id: 94a8ba762aa31f7832294b94a3941fafc3a301ffb01daf9939a5cfdf, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,053 E 7538 7538] actor_manager.cc:93: Actor handle already exists 6791ea75608808f42b66e14a01000000 | |
| [2021-04-29 17:04:28,912 I 7538 7538] core_worker.cc:2064: Finished executing task d787c1e0f1dd624dffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-80b00dc295065311761a5d653a89c2e02712a272002c7cfae1afc0c6_4003.log <== | |
| [2021-04-29 17:03:47,239 E 4003 4003] actor_manager.cc:93: Actor handle already exists ff125b7bd2b8fb7149bc306301000000 | |
| [2021-04-29 17:03:47,340 I 4003 4003] core_worker.cc:2064: Finished executing task ffffffffffffffffb0390cd4ec33f7fef878938501000000, status=OK | |
| [2021-04-29 17:03:47,340 I 4003 4003] direct_actor_transport.cc:497: Actor creation task finished, task_id: ffffffffffffffffb0390cd4ec33f7fef878938501000000, actor_id: b0390cd4ec33f7fef878938501000000 | |
| [2021-04-29 17:03:47,342 I 4003 4241] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: b0390cd4ec33f7fef878938501000000, ip address: 10.133.100.138, port: 9108, worker_id: 80b00dc295065311761a5d653a89c2e02712a272002c7cfae1afc0c6, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:47,343 I 4003 4003] core_worker.cc:2064: Finished executing task 555a36569eff4417b0390cd4ec33f7fef878938501000000, status=OK | |
| [2021-04-29 17:03:47,390 I 4003 4003] core_worker.cc:2064: Finished executing task 8d9e8ffb4c8873dab0390cd4ec33f7fef878938501000000, status=OK | |
| [2021-04-29 17:03:47,528 I 4003 4003] core_worker.cc:2064: Finished executing task 9d630ef9b5d8e090b0390cd4ec33f7fef878938501000000, status=OK | |
| [2021-04-29 17:03:47,714 I 4003 4003] core_worker.cc:2064: Finished executing task 0dbc559bf9908bc0b0390cd4ec33f7fef878938501000000, status=OK | |
| [2021-04-29 17:03:47,723 I 4003 4003] core_worker.cc:2064: Finished executing task 7d0becaa05545c53b0390cd4ec33f7fef878938501000000, status=OK | |
| [2021-04-29 17:03:51,691 I 4003 4003] core_worker.cc:2064: Finished executing task 37600e824393a972b0390cd4ec33f7fef878938501000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-80f705b4b2591aabae28d8caed44c7a237d3667803c785de3a4f19c4_5294.log <== | |
| [2021-04-29 17:03:58,076 E 5294 5294] actor_manager.cc:93: Actor handle already exists 984e942318971ae8001c2cd101000000 | |
| [2021-04-29 17:03:59,689 I 5294 5294] core_worker.cc:2064: Finished executing task 06885b75430125faffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:03:59,699 E 5294 5294] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,699 E 5294 5294] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,767 I 5294 5294] direct_task_transport.cc:47: Creating actor via GCS actor id = : e3872465ce5f88bed482a07301000000 | |
| [2021-04-29 17:03:59,775 I 5294 6306] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: e3872465ce5f88bed482a07301000000, ip address: 10.134.61.25, port: 9127, worker_id: 65771fb41001b5b55b21c88c486ae4fe3e2f2df5a929b1b2fe0fe2a7, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,724 I 5294 6306] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: e3872465ce5f88bed482a07301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:19,882 I 5294 6306] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: e3872465ce5f88bed482a07301000000, ip address: 10.134.61.25, port: 9127, worker_id: 65771fb41001b5b55b21c88c486ae4fe3e2f2df5a929b1b2fe0fe2a7, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,238 E 5294 5294] actor_manager.cc:93: Actor handle already exists e3872465ce5f88bed482a07301000000 | |
| [2021-04-29 17:04:28,998 I 5294 5294] core_worker.cc:2064: Finished executing task 016995209544d1dbffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-820c341a822f0b3f8bf5c1a1b201fca889cbf06d9a5dc99e633de4fe_7593.log <== | |
| [2021-04-29 17:03:58,820 I 7593 9144] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,849 E 7593 7593] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,856 I 7593 9144] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,588 E 7593 7593] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,637 I 7593 7593] direct_task_transport.cc:47: Creating actor via GCS actor id = : 325993f6948fb4cd178f432801000000 | |
| [2021-04-29 17:03:59,644 I 7593 9144] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 325993f6948fb4cd178f432801000000, ip address: 10.134.61.130, port: 9120, worker_id: 17b395057016fbb9158aeac3c4fbdd586eed84201c88e1bf504c91c2, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,665 I 7593 9144] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 325993f6948fb4cd178f432801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:17,278 I 7593 9144] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 325993f6948fb4cd178f432801000000, ip address: 10.134.61.130, port: 9120, worker_id: 17b395057016fbb9158aeac3c4fbdd586eed84201c88e1bf504c91c2, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:20,363 E 7593 7593] actor_manager.cc:93: Actor handle already exists 325993f6948fb4cd178f432801000000 | |
| [2021-04-29 17:04:28,204 I 7593 7593] core_worker.cc:2064: Finished executing task a8d9c15e3b7c7e5bffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-820eaf4caa4ad423460a92e56574a95b273aa034ed0d2b70d4f65811_12780.log <== | |
| [2021-04-29 17:04:04,548 I 12780 13288] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,597 E 12780 12780] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:04,616 I 12780 13288] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:06,512 E 12780 12780] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:06,622 I 12780 12780] direct_task_transport.cc:47: Creating actor via GCS actor id = : a96fccf6d40fdc84f0c3de8f01000000 | |
| [2021-04-29 17:04:06,678 I 12780 13288] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: a96fccf6d40fdc84f0c3de8f01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:12,942 I 12780 13288] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: a96fccf6d40fdc84f0c3de8f01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,396 I 12780 13288] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: a96fccf6d40fdc84f0c3de8f01000000, ip address: 10.134.142.44, port: 9235, worker_id: a66cd3a95c035ff8169fc6f1bdd1e91c240a2b5b913ef7318479f8be, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,333 E 12780 12780] actor_manager.cc:93: Actor handle already exists a96fccf6d40fdc84f0c3de8f01000000 | |
| [2021-04-29 17:04:29,509 I 12780 12780] core_worker.cc:2064: Finished executing task bf3c28353b264493ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-828bf56c8e2cae754ac4e9d04938ce18aad7cc197363730fba552e02_9727.log <== | |
| [2021-04-29 17:04:01,278 I 9727 10728] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,302 E 9727 9727] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,310 I 9727 10728] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,923 E 9727 9727] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:01,987 I 9727 9727] direct_task_transport.cc:47: Creating actor via GCS actor id = : dbd68f6b941307cacc7a944401000000 | |
| [2021-04-29 17:04:01,994 I 9727 10728] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: dbd68f6b941307cacc7a944401000000, ip address: 10.134.112.222, port: 9117, worker_id: 4bd545b4fb6bacf388d80e763e989195691fed3b0d9b312ce2d6f8f1, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,535 I 9727 10728] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: dbd68f6b941307cacc7a944401000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,390 I 9727 10728] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: dbd68f6b941307cacc7a944401000000, ip address: 10.134.112.222, port: 9117, worker_id: 4bd545b4fb6bacf388d80e763e989195691fed3b0d9b312ce2d6f8f1, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,033 E 9727 9727] actor_manager.cc:93: Actor handle already exists dbd68f6b941307cacc7a944401000000 | |
| [2021-04-29 17:04:29,441 I 9727 9727] core_worker.cc:2064: Finished executing task 1471f38e1e564f0cffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-82cdb4a7577e4f582213fb0716cdc099ceaeb796162ceecb2deec90f_9971.log <== | |
| [2021-04-29 17:04:01,975 I 9971 11750] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,996 E 9971 9971] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:02,003 I 9971 11750] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:04,569 E 9971 9971] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:04,671 I 9971 9971] direct_task_transport.cc:47: Creating actor via GCS actor id = : b5a9e83be3e7534ca3553bca01000000 | |
| [2021-04-29 17:04:04,694 I 9971 11750] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: b5a9e83be3e7534ca3553bca01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,170 I 9971 11750] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: b5a9e83be3e7534ca3553bca01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,392 I 9971 11750] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: b5a9e83be3e7534ca3553bca01000000, ip address: 10.134.112.222, port: 9109, worker_id: 8338127b2bcf0eb699d04f36085eff41002629842358142c17959b01, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,142 E 9971 9971] actor_manager.cc:93: Actor handle already exists b5a9e83be3e7534ca3553bca01000000 | |
| [2021-04-29 17:04:29,468 I 9971 9971] core_worker.cc:2064: Finished executing task d969447a5e1aec62ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-83a6d0cca3f084ae8bc6cf3c012a1458412c8e742bfab91b2ceb5994_17956.log <== | |
| [2021-04-29 17:04:09,917 I 17956 19722] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,970 E 17956 17956] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,991 I 17956 19722] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,584 E 17956 17956] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,716 I 17956 17956] direct_task_transport.cc:47: Creating actor via GCS actor id = : 550e5978f9af66973cf8c46f01000000 | |
| [2021-04-29 17:04:11,755 I 17956 19722] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 550e5978f9af66973cf8c46f01000000, ip address: 10.134.142.44, port: 9203, worker_id: dc1baa328ca6dd888154df769ee7a043c8afafec529acef95b9201d5, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,387 I 17956 19722] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 550e5978f9af66973cf8c46f01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:28,288 I 17956 19722] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 550e5978f9af66973cf8c46f01000000, ip address: 10.134.142.44, port: 9203, worker_id: dc1baa328ca6dd888154df769ee7a043c8afafec529acef95b9201d5, raylet_id: ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:29,042 E 17956 17956] actor_manager.cc:93: Actor handle already exists 550e5978f9af66973cf8c46f01000000 | |
| [2021-04-29 17:04:29,866 I 17956 17956] core_worker.cc:2064: Finished executing task c98f83887694bf39ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-83e2f40b2b8d69e8888f122ba55cc3776ee077329577b9c777d8f9b5_5330.log <== | |
| [2021-04-29 17:03:46,228 E 5330 5330] actor_manager.cc:93: Actor handle already exists ff125b7bd2b8fb7149bc306301000000 | |
| [2021-04-29 17:03:46,329 I 5330 5330] core_worker.cc:2064: Finished executing task ffffffffffffffffcce95cc5f0aa7575c2bf692901000000, status=OK | |
| [2021-04-29 17:03:46,329 I 5330 5330] direct_actor_transport.cc:497: Actor creation task finished, task_id: ffffffffffffffffcce95cc5f0aa7575c2bf692901000000, actor_id: cce95cc5f0aa7575c2bf692901000000 | |
| [2021-04-29 17:03:46,331 I 5330 6794] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: cce95cc5f0aa7575c2bf692901000000, ip address: 10.133.100.138, port: 9231, worker_id: 83e2f40b2b8d69e8888f122ba55cc3776ee077329577b9c777d8f9b5, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:46,332 I 5330 5330] core_worker.cc:2064: Finished executing task 081d57ca74d41ee3cce95cc5f0aa7575c2bf692901000000, status=OK | |
| [2021-04-29 17:03:46,336 I 5330 5330] core_worker.cc:2064: Finished executing task c03668af32313d8fcce95cc5f0aa7575c2bf692901000000, status=OK | |
| [2021-04-29 17:03:46,406 I 5330 5330] core_worker.cc:2064: Finished executing task d81fd1750091a45fcce95cc5f0aa7575c2bf692901000000, status=OK | |
| [2021-04-29 17:03:46,603 I 5330 5330] core_worker.cc:2064: Finished executing task eb7d0ce8fc537443cce95cc5f0aa7575c2bf692901000000, status=OK | |
| [2021-04-29 17:03:46,609 I 5330 5330] core_worker.cc:2064: Finished executing task 44604d9a95d58acdcce95cc5f0aa7575c2bf692901000000, status=OK | |
| [2021-04-29 17:03:51,820 I 5330 5330] core_worker.cc:2064: Finished executing task b4e1c6a6ba141c4ecce95cc5f0aa7575c2bf692901000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-842b6fc61bb4be8a57b9cc66462acd2019222381b60690adc48b7e47_4015.log <== | |
| [2021-04-29 18:18:48,641 I 4015 4015] core_worker.cc:2064: Finished executing task d6458f9de015834d9bd23964a7171c972eb385e001000000, status=OK | |
| [2021-04-29 18:18:48,720 I 4015 4015] core_worker.cc:2064: Finished executing task 96338f1ab681566d9bd23964a7171c972eb385e001000000, status=OK | |
| [2021-04-29 18:18:48,721 I 4015 4015] core_worker.cc:2064: Finished executing task c97c7d4c5b8a64849bd23964a7171c972eb385e001000000, status=OK | |
| [2021-04-29 18:18:48,726 I 4015 4015] core_worker.cc:2064: Finished executing task 8ac524e116f1b4449bd23964a7171c972eb385e001000000, status=OK | |
| [2021-04-29 18:18:48,728 I 4015 4015] core_worker.cc:2064: Finished executing task e00e3c438f942d429bd23964a7171c972eb385e001000000, status=OK | |
| [2021-04-29 18:18:48,766 I 4015 4015] core_worker.cc:2064: Finished executing task 9a90cf3f3fc45d6c9bd23964a7171c972eb385e001000000, status=OK | |
| [2021-04-29 18:18:49,196 I 4015 4015] core_worker.cc:2064: Finished executing task 0ceebfa6803a73fe9bd23964a7171c972eb385e001000000, status=OK | |
| [2021-04-29 18:18:49,343 I 4015 4015] core_worker.cc:2064: Finished executing task 39194fbff77ade6c9bd23964a7171c972eb385e001000000, status=OK | |
| [2021-04-29 18:18:50,290 I 4015 4015] core_worker.cc:2064: Finished executing task a6dfe4f960312db59bd23964a7171c972eb385e001000000, status=OK | |
| [2021-04-29 18:18:50,372 I 4015 4015] core_worker.cc:2064: Finished executing task 45096e2bf5b4c8149bd23964a7171c972eb385e001000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-8440d5701e1ea209025219c1ab7565420aeb3a259c014e98632954ac_15875.log <== | |
| [2021-04-29 17:04:07,787 I 15875 16615] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:07,933 E 15875 15875] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:07,993 I 15875 16615] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,743 E 15875 15875] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:10,832 I 15875 15875] direct_task_transport.cc:47: Creating actor via GCS actor id = : b54f3d7b733947e99d051c2401000000 | |
| [2021-04-29 17:04:10,855 I 15875 16615] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: b54f3d7b733947e99d051c2401000000, ip address: 10.134.112.222, port: 9197, worker_id: fdc06987674f2eb1d7f6f365840f9593823aaa5d0fc1e005a53e8ec9, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,368 I 15875 16615] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: b54f3d7b733947e99d051c2401000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:23,063 I 15875 16615] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: b54f3d7b733947e99d051c2401000000, ip address: 10.134.112.222, port: 9197, worker_id: fdc06987674f2eb1d7f6f365840f9593823aaa5d0fc1e005a53e8ec9, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:26,053 E 15875 15875] actor_manager.cc:93: Actor handle already exists b54f3d7b733947e99d051c2401000000 | |
| [2021-04-29 17:04:29,774 I 15875 15875] core_worker.cc:2064: Finished executing task b0929a8b2cd1cde1ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-8494784273dc67ce9f74f6dc9d4bedad5723c5cb45afdb59cfa8eb05_7539.log <== | |
| [2021-04-29 17:03:59,225 I 7539 8809] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,239 E 7539 7539] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,246 I 7539 8809] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,685 E 7539 7539] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,763 I 7539 7539] direct_task_transport.cc:47: Creating actor via GCS actor id = : 0d783347815465ef5513b45301000000 | |
| [2021-04-29 17:03:59,775 I 7539 8809] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 0d783347815465ef5513b45301000000, ip address: 10.134.112.222, port: 9126, worker_id: 5cc05e187f449ab41c752aa0a2b71a4866479ea56bbc49cbc539fe85, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,724 I 7539 8809] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 0d783347815465ef5513b45301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:19,263 I 7539 8809] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 0d783347815465ef5513b45301000000, ip address: 10.134.112.222, port: 9126, worker_id: 5cc05e187f449ab41c752aa0a2b71a4866479ea56bbc49cbc539fe85, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,049 E 7539 7539] actor_manager.cc:93: Actor handle already exists 0d783347815465ef5513b45301000000 | |
| [2021-04-29 17:04:28,909 I 7539 7539] core_worker.cc:2064: Finished executing task e3fb3fa9127139c4ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-84aff3672616a1aeaaf5e36f5aec83556a7b93e5598249c9bb976c39_17830.log <== | |
| [2021-04-29 17:04:09,731 I 17830 19377] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,845 E 17830 17830] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,868 I 17830 19377] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,578 E 17830 17830] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,702 I 17830 17830] direct_task_transport.cc:47: Creating actor via GCS actor id = : 5f1fa449904765f5e1c6fffe01000000 | |
| [2021-04-29 17:04:11,715 I 17830 19377] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 5f1fa449904765f5e1c6fffe01000000, ip address: 10.134.61.130, port: 9209, worker_id: 741e31aa23b0f1dd198ecc71221802ddf5bbe24d07ee8906bd18a6d6, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,385 I 17830 19377] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 5f1fa449904765f5e1c6fffe01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:28,231 I 17830 19377] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 5f1fa449904765f5e1c6fffe01000000, ip address: 10.134.61.130, port: 9209, worker_id: 741e31aa23b0f1dd198ecc71221802ddf5bbe24d07ee8906bd18a6d6, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:29,025 E 17830 17830] actor_manager.cc:93: Actor handle already exists 5f1fa449904765f5e1c6fffe01000000 | |
| [2021-04-29 17:04:29,864 I 17830 17830] core_worker.cc:2064: Finished executing task bc76fb6c711028e3ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-851864023629f057cceaa763a82a2969c083264659d123a7111730ae_12096.log <== | |
| [2021-04-29 17:04:03,465 I 12096 12631] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,494 E 12096 12096] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:03,507 I 12096 12631] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,875 E 12096 12096] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,943 I 12096 12096] direct_task_transport.cc:47: Creating actor via GCS actor id = : 5fae7c1c0ad48bbb6b1d566101000000 | |
| [2021-04-29 17:04:05,968 I 12096 12631] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 5fae7c1c0ad48bbb6b1d566101000000, ip address: 10.133.95.40, port: 9105, worker_id: eafd1454ea7b891a2a948145e6a7bbd59082f5bc370b89d0102a0993, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,787 I 12096 12631] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 5fae7c1c0ad48bbb6b1d566101000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,395 I 12096 12631] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 5fae7c1c0ad48bbb6b1d566101000000, ip address: 10.133.95.40, port: 9105, worker_id: eafd1454ea7b891a2a948145e6a7bbd59082f5bc370b89d0102a0993, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,233 E 12096 12096] actor_manager.cc:93: Actor handle already exists 5fae7c1c0ad48bbb6b1d566101000000 | |
| [2021-04-29 17:04:29,480 I 12096 12096] core_worker.cc:2064: Finished executing task 25101ad438e64abdffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-8520be78dda1c04d94589679f3a444ceb9a06806af5feded67a4e3f9_4122.log <== | |
| [2021-04-29 18:18:50,575 I 4122 4122] core_worker.cc:2064: Finished executing task 6065fbd5e8d8722a4977e7b4fb33e628bd4ea7a601000000, status=OK | |
| [2021-04-29 18:18:50,577 I 4122 4122] core_worker.cc:2064: Finished executing task edad2688b753339b4977e7b4fb33e628bd4ea7a601000000, status=OK | |
| [2021-04-29 18:18:50,679 I 4122 4122] core_worker.cc:2064: Finished executing task 44872b00e4947b584977e7b4fb33e628bd4ea7a601000000, status=OK | |
| [2021-04-29 18:18:50,687 I 4122 4122] core_worker.cc:2064: Finished executing task ab621a90115ac5e04977e7b4fb33e628bd4ea7a601000000, status=OK | |
| [2021-04-29 18:18:50,765 I 4122 4122] core_worker.cc:2064: Finished executing task d0c2e90da5a3d79f4977e7b4fb33e628bd4ea7a601000000, status=OK | |
| [2021-04-29 18:18:50,771 I 4122 4122] core_worker.cc:2064: Finished executing task e4112094b552601e4977e7b4fb33e628bd4ea7a601000000, status=OK | |
| [2021-04-29 18:18:50,795 I 4122 4122] core_worker.cc:2064: Finished executing task 35c17814ee7eb2384977e7b4fb33e628bd4ea7a601000000, status=OK | |
| [2021-04-29 18:18:50,800 I 4122 4122] core_worker.cc:2064: Finished executing task 9ed152861bd81d1e4977e7b4fb33e628bd4ea7a601000000, status=OK | |
| [2021-04-29 18:18:50,802 I 4122 4122] core_worker.cc:2064: Finished executing task 0bcd507604106cfd4977e7b4fb33e628bd4ea7a601000000, status=OK | |
| [2021-04-29 18:18:50,807 I 4122 4122] core_worker.cc:2064: Finished executing task 3bb9e55f920417be4977e7b4fb33e628bd4ea7a601000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-852301d11511f6067444b0beb19d92d289c92fc2a20917d2ba2cbb78_12936.log <== | |
| [2021-04-29 17:04:04,999 I 12936 14749] service_based_accessor.cc:579: Received notification for node id = ccd14cc86f69727ef5a2309d41d50d7ebde4d31afffdef51de620010, IsAlive = 1 | |
| [2021-04-29 17:04:04,999 I 12936 14749] service_based_accessor.cc:579: Received notification for node id = d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, IsAlive = 1 | |
| [2021-04-29 17:04:05,000 I 12936 14749] service_based_accessor.cc:579: Received notification for node id = b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, IsAlive = 1 | |
| [2021-04-29 17:04:05,000 I 12936 14749] service_based_accessor.cc:579: Received notification for node id = 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, IsAlive = 1 | |
| [2021-04-29 17:04:05,000 I 12936 14749] service_based_accessor.cc:579: Received notification for node id = 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, IsAlive = 1 | |
| [2021-04-29 17:04:06,182 I 12936 14749] core_worker.cc:717: Exit signal received, this process will exit after all outstanding tasks have finished, exit_type=INTENDED_EXIT | |
| [2021-04-29 17:04:06,182 I 12936 12936] raylet_client.cc:129: RayletClient::Disconnect, exit_type=INTENDED_EXIT, has creation_task_exception_pb_bytes=0 | |
| [2021-04-29 17:04:06,182 I 12936 12936] core_worker.cc:308: Removed worker 852301d11511f6067444b0beb19d92d289c92fc2a20917d2ba2cbb78 | |
| [2021-04-29 17:04:06,185 I 12936 12936] core_worker.cc:181: Destructing CoreWorkerProcess. pid: 12936 | |
| [2021-04-29 17:04:06,185 I 12936 12936] io_service_pool.cc:47: IOServicePool is stopped. | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-85301f245301c1017fe71075c25a75e850d1f77534da1e3b7fbca685_12126.log <== | |
| [2021-04-29 17:04:03,477 I 12126 12665] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,511 E 12126 12126] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:03,519 I 12126 12665] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,876 E 12126 12126] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,970 I 12126 12126] direct_task_transport.cc:47: Creating actor via GCS actor id = : 5c4f6ec99122e70ba8e7f1b901000000 | |
| [2021-04-29 17:04:05,983 I 12126 12665] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 5c4f6ec99122e70ba8e7f1b901000000, ip address: 10.134.70.227, port: 9165, worker_id: 10c64d2487d02c895f527d86f74b7296c939fe1859c50fc2ff6e8fe5, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,424 I 12126 12665] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 5c4f6ec99122e70ba8e7f1b901000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,396 I 12126 12665] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 5c4f6ec99122e70ba8e7f1b901000000, ip address: 10.134.70.227, port: 9165, worker_id: 10c64d2487d02c895f527d86f74b7296c939fe1859c50fc2ff6e8fe5, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,298 E 12126 12126] actor_manager.cc:93: Actor handle already exists 5c4f6ec99122e70ba8e7f1b901000000 | |
| [2021-04-29 17:04:29,489 I 12126 12126] core_worker.cc:2064: Finished executing task f174b79248a6a5c2ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-853467f61f9a640537386b1a82103aa368e9d25316e74f5074c4d080_4041.log <== | |
| [2021-04-29 17:03:44,989 I 4041 4041] core_worker.cc:2064: Finished executing task ef45c811ae2ca12f4ffcd6da7cb77a50d7d68b1401000000, status=OK | |
| [2021-04-29 17:04:49,001 I 4041 4041] core_worker.cc:2064: Finished executing task d8db6d429e12dc774ffcd6da7cb77a50d7d68b1401000000, status=OK | |
| [2021-04-29 17:04:50,312 I 4041 4041] core_worker.cc:2064: Finished executing task 2b51f79eb7d94eba4ffcd6da7cb77a50d7d68b1401000000, status=OK | |
| [2021-04-29 17:04:50,664 I 4041 4041] core_worker.cc:2064: Finished executing task 077fb35a3bc33e874ffcd6da7cb77a50d7d68b1401000000, status=OK | |
| [2021-04-29 17:04:51,944 I 4041 4041] core_worker.cc:2064: Finished executing task b9a09abb6a4640264ffcd6da7cb77a50d7d68b1401000000, status=OK | |
| [2021-04-29 17:04:53,644 I 4041 4041] core_worker.cc:2064: Finished executing task c41a47b4addbb6474ffcd6da7cb77a50d7d68b1401000000, status=OK | |
| [2021-04-29 17:04:54,522 I 4041 4041] core_worker.cc:2064: Finished executing task 370632bb4fa3629e4ffcd6da7cb77a50d7d68b1401000000, status=OK | |
| [2021-04-29 17:04:55,746 I 4041 4041] core_worker.cc:2064: Finished executing task e1199a93293fdb524ffcd6da7cb77a50d7d68b1401000000, status=OK | |
| [2021-04-29 17:04:57,550 I 4041 4041] core_worker.cc:2064: Finished executing task 9e6cf301f2e0fe644ffcd6da7cb77a50d7d68b1401000000, status=OK | |
| [2021-04-29 17:04:57,783 I 4041 4041] core_worker.cc:2064: Finished executing task 7c566fab510c7e184ffcd6da7cb77a50d7d68b1401000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-853a5df0f5ed1a8c2ca3c56a8a73aa7e8e712cd1c125e84cc576321e_9761.log <== | |
| [2021-04-29 17:04:02,512 I 9761 11096] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,531 E 9761 9761] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:02,538 I 9761 11096] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,585 E 9761 9761] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,644 I 9761 9761] direct_task_transport.cc:47: Creating actor via GCS actor id = : ab0adc65b9a0bde509c8aa6d01000000 | |
| [2021-04-29 17:04:05,664 I 9761 11096] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: ab0adc65b9a0bde509c8aa6d01000000, ip address: 10.134.70.227, port: 9145, worker_id: 50cd93d10381b4262205929e5b93055d2be3c0b7b30fb5665ac6a534, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:07,604 I 9761 11096] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: ab0adc65b9a0bde509c8aa6d01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,395 I 9761 11096] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: ab0adc65b9a0bde509c8aa6d01000000, ip address: 10.134.70.227, port: 9145, worker_id: 50cd93d10381b4262205929e5b93055d2be3c0b7b30fb5665ac6a534, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,242 E 9761 9761] actor_manager.cc:93: Actor handle already exists ab0adc65b9a0bde509c8aa6d01000000 | |
| [2021-04-29 17:04:29,484 I 9761 9761] core_worker.cc:2064: Finished executing task 388b15d1e2020179ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-855bc01948f2b510f973dc3ab930cdaf1e77fe0fd89771165364e2de_9680.log <== | |
| [2021-04-29 17:04:01,175 I 9680 10339] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,207 E 9680 9680] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,216 I 9680 10339] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,775 E 9680 9680] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:01,869 I 9680 9680] direct_task_transport.cc:47: Creating actor via GCS actor id = : 66d429edd5817241ca8e923e01000000 | |
| [2021-04-29 17:04:01,873 I 9680 10339] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 66d429edd5817241ca8e923e01000000, ip address: 10.134.70.227, port: 36993, worker_id: NIL_ID, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,263 I 9680 10339] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 66d429edd5817241ca8e923e01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,390 I 9680 10339] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 66d429edd5817241ca8e923e01000000, ip address: 10.134.70.227, port: 9108, worker_id: 663cfef800ce766fa24df06dbe2fa9c46c03fc12dc5c10018cb49397, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,067 E 9680 9680] actor_manager.cc:93: Actor handle already exists 66d429edd5817241ca8e923e01000000 | |
| [2021-04-29 17:04:29,456 I 9680 9680] core_worker.cc:2064: Finished executing task 883bb57a51de153dffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-85b1677fbcefb173d57705b46f007cb8831538470bc8d9baff6a3dce_5295.log <== | |
| [2021-04-29 17:03:58,602 E 5295 5295] actor_manager.cc:93: Actor handle already exists cc4044121f22ebd36202986601000000 | |
| [2021-04-29 17:04:00,060 I 5295 5295] core_worker.cc:2064: Finished executing task 5515007e6144a2ebffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:04:00,064 E 5295 5295] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:00,064 E 5295 5295] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:00,133 I 5295 5295] direct_task_transport.cc:47: Creating actor via GCS actor id = : 6cf8d74ba497c5137ba20fc601000000 | |
| [2021-04-29 17:04:00,143 I 5295 6228] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 6cf8d74ba497c5137ba20fc601000000, ip address: 10.134.61.130, port: 9113, worker_id: d971bb6b3bb6a994a71afd28b967ecb5f0c2a24ae63b49df36082e91, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,730 I 5295 6228] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 6cf8d74ba497c5137ba20fc601000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,367 I 5295 6228] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 6cf8d74ba497c5137ba20fc601000000, ip address: 10.134.61.130, port: 9113, worker_id: d971bb6b3bb6a994a71afd28b967ecb5f0c2a24ae63b49df36082e91, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:22,937 E 5295 5295] actor_manager.cc:93: Actor handle already exists 6cf8d74ba497c5137ba20fc601000000 | |
| [2021-04-29 17:04:29,421 I 5295 5295] core_worker.cc:2064: Finished executing task 2939239fbe95deacffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-85c1a6b298328690a1147d19da1bad114cd3ef0a5c9a1c8a6ea68903_5364.log <== | |
| [2021-04-29 17:03:46,213 E 5364 5364] actor_manager.cc:93: Actor handle already exists ff125b7bd2b8fb7149bc306301000000 | |
| [2021-04-29 17:03:46,314 I 5364 5364] core_worker.cc:2064: Finished executing task ffffffffffffffffaa337de69d993972c6c7b40301000000, status=OK | |
| [2021-04-29 17:03:46,314 I 5364 5364] direct_actor_transport.cc:497: Actor creation task finished, task_id: ffffffffffffffffaa337de69d993972c6c7b40301000000, actor_id: aa337de69d993972c6c7b40301000000 | |
| [2021-04-29 17:03:46,316 I 5364 6625] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: aa337de69d993972c6c7b40301000000, ip address: 10.133.100.138, port: 9222, worker_id: 85c1a6b298328690a1147d19da1bad114cd3ef0a5c9a1c8a6ea68903, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:46,317 I 5364 5364] core_worker.cc:2064: Finished executing task 7956d6b844a70ac1aa337de69d993972c6c7b40301000000, status=OK | |
| [2021-04-29 17:03:46,320 I 5364 5364] core_worker.cc:2064: Finished executing task b4a2c0f6c205a3eeaa337de69d993972c6c7b40301000000, status=OK | |
| [2021-04-29 17:03:46,337 I 5364 5364] core_worker.cc:2064: Finished executing task b0cc08efc1e2947caa337de69d993972c6c7b40301000000, status=OK | |
| [2021-04-29 17:03:46,600 I 5364 5364] core_worker.cc:2064: Finished executing task e957137c8656d4fbaa337de69d993972c6c7b40301000000, status=OK | |
| [2021-04-29 17:03:46,609 I 5364 5364] core_worker.cc:2064: Finished executing task 6a3e2113697c81c9aa337de69d993972c6c7b40301000000, status=OK | |
| [2021-04-29 17:03:51,716 I 5364 5364] core_worker.cc:2064: Finished executing task 1ed7336be85712ecaa337de69d993972c6c7b40301000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-85ce5ae144260ba56ab6227165f61f1f337dc71ea5a86c15c6cc42ec_5396.log <== | |
| [2021-04-29 17:03:47,118 E 5396 5396] actor_manager.cc:93: Actor handle already exists ff125b7bd2b8fb7149bc306301000000 | |
| [2021-04-29 17:03:47,219 I 5396 5396] core_worker.cc:2064: Finished executing task ffffffffffffffff21eabe8d266ee49ccbb3945101000000, status=OK | |
| [2021-04-29 17:03:47,219 I 5396 5396] direct_actor_transport.cc:497: Actor creation task finished, task_id: ffffffffffffffff21eabe8d266ee49ccbb3945101000000, actor_id: 21eabe8d266ee49ccbb3945101000000 | |
| [2021-04-29 17:03:47,223 I 5396 5396] core_worker.cc:2064: Finished executing task 501702044d4bd0c921eabe8d266ee49ccbb3945101000000, status=OK | |
| [2021-04-29 17:03:47,229 I 5396 6786] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 21eabe8d266ee49ccbb3945101000000, ip address: 10.133.100.138, port: 9230, worker_id: 85ce5ae144260ba56ab6227165f61f1f337dc71ea5a86c15c6cc42ec, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:47,276 I 5396 5396] core_worker.cc:2064: Finished executing task fc24b7763979a05b21eabe8d266ee49ccbb3945101000000, status=OK | |
| [2021-04-29 17:03:47,404 I 5396 5396] core_worker.cc:2064: Finished executing task 751f3a02f700903221eabe8d266ee49ccbb3945101000000, status=OK | |
| [2021-04-29 17:03:47,694 I 5396 5396] core_worker.cc:2064: Finished executing task 8ce5f912ed3c21fe21eabe8d266ee49ccbb3945101000000, status=OK | |
| [2021-04-29 17:03:47,721 I 5396 5396] core_worker.cc:2064: Finished executing task 9f74342accf289b721eabe8d266ee49ccbb3945101000000, status=OK | |
| [2021-04-29 17:03:51,383 I 5396 5396] core_worker.cc:2064: Finished executing task cdd5d239a1de0ab121eabe8d266ee49ccbb3945101000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-86a536f930f089394309b13fe03badc6e426e1d5274abac822a0fe73_5311.log <== | |
| [2021-04-29 17:03:57,918 E 5311 5311] actor_manager.cc:93: Actor handle already exists bc874a966e47480c875ba35301000000 | |
| [2021-04-29 17:03:59,549 I 5311 5311] core_worker.cc:2064: Finished executing task a3695459c1e2f605ffffffffffffffffffffffff01000000, status=OK | |
| [2021-04-29 17:03:59,556 E 5311 5311] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,556 E 5311 5311] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:59,599 I 5311 5311] direct_task_transport.cc:47: Creating actor via GCS actor id = : dd55b772317d1793079117c801000000 | |
| [2021-04-29 17:03:59,608 I 5311 6394] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: dd55b772317d1793079117c801000000, ip address: 10.133.95.40, port: 9130, worker_id: aa53e326bbbcb698f86e98ba325af28eb91681346c67f1173c672791, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,518 I 5311 6394] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: dd55b772317d1793079117c801000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:17,209 I 5311 6394] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: dd55b772317d1793079117c801000000, ip address: 10.133.95.40, port: 9130, worker_id: aa53e326bbbcb698f86e98ba325af28eb91681346c67f1173c672791, raylet_id: 5cff32e96f30979763fbe8807fc6a6a4311a4dc64927a1130775698f, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:19,715 E 5311 5311] actor_manager.cc:93: Actor handle already exists dd55b772317d1793079117c801000000 | |
| [2021-04-29 17:04:27,710 I 5311 5311] core_worker.cc:2064: Finished executing task 319f3a10d851d258ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-86ab190ca7f8e353145f09ed3a1e3b853348634b4fe7e33a48ccaa39_9654.log <== | |
| [2021-04-29 17:04:00,909 I 9654 9965] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:00,962 E 9654 9654] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:00,975 I 9654 9965] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,761 E 9654 9654] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:01,810 I 9654 9654] direct_task_transport.cc:47: Creating actor via GCS actor id = : 2e9e83cb173c3585fe72cbae01000000 | |
| [2021-04-29 17:04:01,826 I 9654 9965] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 2e9e83cb173c3585fe72cbae01000000, ip address: 10.134.60.23, port: 9122, worker_id: 3b3fc4f854e0be9921783f63cf496caccec6defc999392897a1f70e4, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,910 I 9654 9965] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 2e9e83cb173c3585fe72cbae01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,389 I 9654 9965] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 2e9e83cb173c3585fe72cbae01000000, ip address: 10.134.60.23, port: 9122, worker_id: 3b3fc4f854e0be9921783f63cf496caccec6defc999392897a1f70e4, raylet_id: 765014361ebf26df47dc3385f953dbd6c0e458fb91b8afaffecc66ca, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,006 E 9654 9654] actor_manager.cc:93: Actor handle already exists 2e9e83cb173c3585fe72cbae01000000 | |
| [2021-04-29 17:04:29,436 I 9654 9654] core_worker.cc:2064: Finished executing task 9325b12c1b4d5de3ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-87481f20f0d9405a728defdc258aa18e7903e4fecf6bfeef09ce6653_7575.log <== | |
| [2021-04-29 17:03:58,438 I 7575 9260] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,439 E 7575 7575] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,440 I 7575 9260] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,822 E 7575 7575] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:58,897 I 7575 7575] direct_task_transport.cc:47: Creating actor via GCS actor id = : d4dc42ab9a08cdf158aad1ad01000000 | |
| [2021-04-29 17:03:58,908 I 7575 9260] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: d4dc42ab9a08cdf158aad1ad01000000, ip address: 10.134.70.227, port: 9125, worker_id: 81a540b5031060a06a95ceb5440de0e55e918f8a9a33ea5767d2b121, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:00,317 I 7575 9260] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: d4dc42ab9a08cdf158aad1ad01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,172 I 7575 9260] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: d4dc42ab9a08cdf158aad1ad01000000, ip address: 10.134.70.227, port: 9125, worker_id: 81a540b5031060a06a95ceb5440de0e55e918f8a9a33ea5767d2b121, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:07,285 E 7575 7575] actor_manager.cc:93: Actor handle already exists d4dc42ab9a08cdf158aad1ad01000000 | |
| [2021-04-29 17:04:12,628 I 7575 7575] core_worker.cc:2064: Finished executing task ac1ca37b46b8229affffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-879761a5a7bb04791ce5602b5dee43f13a7c154328482621650bdb9f_18042.log <== | |
| [2021-04-29 17:04:10,205 I 18042 19902] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:10,294 E 18042 18042] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:10,313 I 18042 19902] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,623 E 18042 18042] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,756 I 18042 18042] direct_task_transport.cc:47: Creating actor via GCS actor id = : af1dc36f57b06f087017697301000000 | |
| [2021-04-29 17:04:11,778 I 18042 19902] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: af1dc36f57b06f087017697301000000, ip address: 10.134.142.16, port: 9185, worker_id: 3297eb44dcc5a2ec09e6bb93134a7ee7d1dd744de7d302a506e3a1c0, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,389 I 18042 19902] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: af1dc36f57b06f087017697301000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:27,570 I 18042 19902] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: af1dc36f57b06f087017697301000000, ip address: 10.134.142.16, port: 9185, worker_id: 3297eb44dcc5a2ec09e6bb93134a7ee7d1dd744de7d302a506e3a1c0, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,898 E 18042 18042] actor_manager.cc:93: Actor handle already exists af1dc36f57b06f087017697301000000 | |
| [2021-04-29 17:04:29,856 I 18042 18042] core_worker.cc:2064: Finished executing task cee1802fe00e02baffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-88940c2823546cc12b2817f9f824f0bbc01a3e5acf5b4ec9b22908c6_16043.log <== | |
| [2021-04-29 17:04:08,924 I 16043 18045] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:09,057 E 16043 16043] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:09,084 I 16043 18045] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:11,205 E 16043 16043] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:11,301 I 16043 16043] direct_task_transport.cc:47: Creating actor via GCS actor id = : a8bd0ba919e4c8824b930eb101000000 | |
| [2021-04-29 17:04:11,327 I 16043 18045] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: a8bd0ba919e4c8824b930eb101000000, ip address: 10.134.70.227, port: 9203, worker_id: e88fe190e44a2b45891677d00b19142c0b82acc3fadb8e7f4e69476a, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,378 I 16043 18045] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: a8bd0ba919e4c8824b930eb101000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:27,431 I 16043 18045] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: a8bd0ba919e4c8824b930eb101000000, ip address: 10.134.70.227, port: 9203, worker_id: e88fe190e44a2b45891677d00b19142c0b82acc3fadb8e7f4e69476a, raylet_id: 84ab516af7a29a090ef1282517317d20ae78a7eeb4ef6e8578f14981, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:28,865 E 16043 16043] actor_manager.cc:93: Actor handle already exists a8bd0ba919e4c8824b930eb101000000 | |
| [2021-04-29 17:04:29,854 I 16043 16043] core_worker.cc:2064: Finished executing task 1d6004af1dd212c0ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-891acaa7e6197346a43efe739f7fd5c676e0aee46d84c4d104a2d57a_9857.log <== | |
| [2021-04-29 17:04:02,081 I 9857 11581] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:02,109 E 9857 9857] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:02,116 I 9857 11581] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,004 E 9857 9857] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:05,085 I 9857 9857] direct_task_transport.cc:47: Creating actor via GCS actor id = : ad24da3ec8599e6a8163ba9f01000000 | |
| [2021-04-29 17:04:05,101 I 9857 11581] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: ad24da3ec8599e6a8163ba9f01000000, ip address: 10.134.61.130, port: 13217, worker_id: NIL_ID, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:05,211 I 9857 11581] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: ad24da3ec8599e6a8163ba9f01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,396 I 9857 11581] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: ad24da3ec8599e6a8163ba9f01000000, ip address: 10.134.61.130, port: 9142, worker_id: 76f36cbd16ac1acc3b33b5484f974ca006634db66533ef1a9624803a, raylet_id: 214a9942ce114b35fed4ae8533076a088c48dd7782db303dad66ea0e, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,329 E 9857 9857] actor_manager.cc:93: Actor handle already exists ad24da3ec8599e6a8163ba9f01000000 | |
| [2021-04-29 17:04:29,502 I 9857 9857] core_worker.cc:2064: Finished executing task 64c1b3467a4b6b3cffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-895b983244402f2369c85fd1d0afc9e38e472afa4c88aeb2c5e5f158_7592.log <== | |
| [2021-04-29 17:03:58,804 I 7592 9150] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,840 E 7592 7592] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,847 I 7592 9150] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,556 E 7592 7592] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,604 I 7592 7592] direct_task_transport.cc:47: Creating actor via GCS actor id = : e719b8e24d9c896e07b51f2d01000000 | |
| [2021-04-29 17:03:59,608 I 7592 9150] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: e719b8e24d9c896e07b51f2d01000000, ip address: 10.134.61.25, port: 9132, worker_id: de29432fac26d2bccffbc0440b3eb776d591cdbed467f7b49d23c0cb, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,593 I 7592 9150] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: e719b8e24d9c896e07b51f2d01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:17,209 I 7592 9150] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: e719b8e24d9c896e07b51f2d01000000, ip address: 10.134.61.25, port: 9132, worker_id: de29432fac26d2bccffbc0440b3eb776d591cdbed467f7b49d23c0cb, raylet_id: 954d6b29fc033c02d54cbdb6b86bf5f980c892faf57c0c2d9654f28b, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:19,699 E 7592 7592] actor_manager.cc:93: Actor handle already exists e719b8e24d9c896e07b51f2d01000000 | |
| [2021-04-29 17:04:27,707 I 7592 7592] core_worker.cc:2064: Finished executing task cd2112dade07dd09ffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-89ab0dbee4e214f4abe51c42d69de9eae575caa5e636b87104793b7d_7845.log <== | |
| [2021-04-29 17:03:58,670 I 7845 9433] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:58,706 E 7845 7845] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:03:58,713 I 7845 9433] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:03:59,382 E 7845 7845] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:03:59,446 I 7845 7845] direct_task_transport.cc:47: Creating actor via GCS actor id = : 6510d78f4175facd5de49ae701000000 | |
| [2021-04-29 17:03:59,452 I 7845 9433] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: 6510d78f4175facd5de49ae701000000, ip address: 10.134.112.222, port: 9132, worker_id: a33a0f854a4e1c3444ddc7ef33f4d21b8fa42f8c08d43df26b631232, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,272 I 7845 9433] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: 6510d78f4175facd5de49ae701000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:08,430 I 7845 9433] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 6510d78f4175facd5de49ae701000000, ip address: 10.134.112.222, port: 9132, worker_id: a33a0f854a4e1c3444ddc7ef33f4d21b8fa42f8c08d43df26b631232, raylet_id: d60884b2115a0b3a9ac2dbe69a6ebde919a9ab44b868dd33bb129118, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:11,028 E 7845 7845] actor_manager.cc:93: Actor handle already exists 6510d78f4175facd5de49ae701000000 | |
| [2021-04-29 17:04:16,167 I 7845 7845] core_worker.cc:2064: Finished executing task 7d8cf92f4de1a20cffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-89c8dbe631a371356c97d2a78e8351203c76b775b4abc09237468aa9_9718.log <== | |
| [2021-04-29 17:04:01,255 I 9718 10637] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 1ab9d0c5077c26e201078b4d01000000, ip address: 10.133.100.138, port: 9143, worker_id: fc53e42bcadcd5da12766a8956141a65ee4ecdac0d78a9b63b85b0e8, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,279 E 9718 9718] actor_manager.cc:93: Actor handle already exists 1ab9d0c5077c26e201078b4d01000000 | |
| [2021-04-29 17:04:01,287 I 9718 10637] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: 8458f0d3cd94cfc58ef7456001000000, ip address: 10.133.100.138, port: 9101, worker_id: 2f2a2ad37bad57d5eca3db5bd49d0385a1d2fb43ee2dfb629bbb4e3d, raylet_id: b0b02fa3383a45bf8121552a7e3981176665649125a8c23552736ce7, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:01,921 E 9718 9718] actor_manager.cc:93: Actor handle already exists 8458f0d3cd94cfc58ef7456001000000 | |
| [2021-04-29 17:04:01,992 I 9718 9718] direct_task_transport.cc:47: Creating actor via GCS actor id = : e4a42cc9106875d78eac6d5f01000000 | |
| [2021-04-29 17:04:02,004 I 9718 10637] actor_manager.cc:127: received notification on actor, state: PENDING_CREATION, actor_id: e4a42cc9106875d78eac6d5f01000000, ip address: 10.134.142.16, port: 9117, worker_id: 72d1702f5a731a927f06eeefe8694d5e35e15f83d0fb03ef91268e00, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:03,538 I 9718 10637] actor_manager.cc:127: received notification on actor, state: DEPENDENCIES_UNREADY, actor_id: e4a42cc9106875d78eac6d5f01000000, ip address: , port: 0, worker_id: NIL_ID, raylet_id: NIL_ID, num_restarts: 0, has creation_task_exception=0 | |
| [2021-04-29 17:04:21,390 I 9718 10637] actor_manager.cc:127: received notification on actor, state: ALIVE, actor_id: e4a42cc9106875d78eac6d5f01000000, ip address: 10.134.142.16, port: 9117, worker_id: 72d1702f5a731a927f06eeefe8694d5e35e15f83d0fb03ef91268e00, raylet_id: 4c50ee8801cabacd89edaaa2f4286ccfd7c0004982620a88a5aa8186, num_restarts: 0, has creation_task_exception=1 | |
| [2021-04-29 17:04:23,044 E 9718 9718] actor_manager.cc:93: Actor handle already exists e4a42cc9106875d78eac6d5f01000000 | |
| [2021-04-29 17:04:29,447 I 9718 9718] core_worker.cc:2064: Finished executing task f8c1336bf04c5f8cffffffffffffffffffffffff01000000, status=OK | |
| ==> /tmp/ray/session_latest/logs/python-core-worker-8a52416bef41e73d4198328ab419934bb94adf256e764d561468174f_5350.log <== | |
| [2021-04-29 17:03:46,418 I 5350 5350] core_worker.cc:2064: Finished executing task cd71e5ea625b4241b882581db969d787fcfc73cd01000000, status=OK | |
| [2021-04-29 17:04 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment