`ray.wait` node failure example

logs

WARNING: 3 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 3 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 3 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 3 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 3 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 3 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 3 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 3 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 3 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 4 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 4 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 4 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 4 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 3 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 4 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 4 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 4 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 4 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 4 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 5 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 5 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 5 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 5 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 5 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 5 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 5 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 4 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 6 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 5 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 5 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 6 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 6 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 6 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 6 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 6 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 5 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 6 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 7 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 6 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 6 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 7 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 7 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 7 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 7 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 7 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 7 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 8 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 7 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 7 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 6 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 8 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 8 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 8 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 8 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 8 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 9 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 8 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 8 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 9 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 9 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 9 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 10 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 10 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 10 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 10 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 11 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 11 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
WARNING: 12 workers have been started. This could be a result of using a large number of actors, or it could be a consequence of using nested tasks (see https://github.com/ray-project/ray/issues/3644) for some a discussion of workarounds.
Killing all monitor(s)
Killing all log_monitor(s)
Killing all worker(s)
Killing all raylet(s)
Killing all plasma_store(s)
Killing all redis_server(s)
Killing all web_ui(s)
Waiting all monitor(s)
Waiting all log_monitor(s)
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0114 21:12:53.645241 24920 raylet_extension.cc:66]  Check failed: _s.ok() [RayletClient] Failed to get a task from raylet.: IOError: [RayletClient] Raylet connection closed.
*** Check failure stack trace: ***
Fatal Python error: Aborted

Stack (most recent call first):
  File "/home/stephanie/ray-fork/python/ray/worker.py", line 971 in _get_next_task_from_local_scheduler
  File "/home/stephanie/ray-fork/python/ray/worker.py", line 988 in main_loop
  File "/home/stephanie/ray-fork/python/ray/workers/default_worker.py", line 101 in <module>
Waiting all worker(s)
Waiting all raylet(s)
Waiting all plasma_store(s)
Waiting all redis_server(s)
Waiting all web_ui(s)
WARNING: Logging before InitGoogleLogging() is written to STDERR
E0114 21:12:53.846320 24922 raylet_client.cc:345] IOError: [RayletClient] Connection closed unexpectedly. [RayletClient] Failed to push profile events.
Pushing error to dirver, type: worker_crash, message: Traceback (most recent call last):
  File "/home/stephanie/ray-fork/python/ray/worker.py", line 864, in _process_task
    self._store_outputs_in_object_store(return_object_ids, outputs)
  File "/home/stephanie/ray-fork/python/ray/worker.py", line 773, in _store_outputs_in_object_store
    self.put_object(object_ids[i], outputs[i])
  File "/home/stephanie/ray-fork/python/ray/worker.py", line 364, in put_object
    self.store_and_register(object_id, value)
  File "/home/stephanie/ray-fork/python/ray/worker.py", line 299, in store_and_register
    self.task_driver_id))
  File "/home/stephanie/ray-fork/python/ray/utils.py", line 441, in _wrapper
    return orig_attr(*args, **kwargs)
  File "pyarrow/_plasma.pyx", line 491, in pyarrow._plasma.PlasmaClient.put
  File "pyarrow/_plasma.pyx", line 322, in pyarrow._plasma.PlasmaClient.create
  File "pyarrow/error.pxi", line 83, in pyarrow.lib.check_status
pyarrow.lib.ArrowIOError: Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/stephanie/ray-fork/python/ray/workers/default_worker.py", line 101, in <module>
    ray.worker.global_worker.main_loop()
  File "/home/stephanie/ray-fork/python/ray/worker.py", line 989, in main_loop
    self._wait_for_and_process_task(task)
  File "/home/stephanie/ray-fork/python/ray/worker.py", line 943, in _wait_for_and_process_task
    self._process_task(task, execution_info)
  File "/home/stephanie/ray-fork/python/ray/worker.py", line 868, in _process_task
    ray.utils.format_error_message(traceback.format_exc()))
  File "/home/stephanie/ray-fork/python/ray/worker.py", line 878, in _handle_process_task_failure
    self._store_outputs_in_object_store(return_object_ids, failure_objects)
  File "/home/stephanie/ray-fork/python/ray/worker.py", line 773, in _store_outputs_in_object_store
    self.put_object(object_ids[i], outputs[i])
  File "/home/stephanie/ray-fork/python/ray/worker.py", line 364, in put_object
    self.store_and_register(object_id, value)
  File "/home/stephanie/ray-fork/python/ray/worker.py", line 299, in store_and_register
    self.task_driver_id))
  File "/home/stephanie/ray-fork/python/ray/utils.py", line 441, in _wrapper
    return orig_attr(*args, **kwargs)
  File "pyarrow/_plasma.pyx", line 491, in pyarrow._plasma.PlasmaClient.put
  File "pyarrow/_plasma.pyx", line 322, in pyarrow._plasma.PlasmaClient.create
  File "pyarrow/error.pxi", line 83, in pyarrow.lib.check_status
pyarrow.lib.ArrowIOError: Broken pipe

  This error is unexpected and should not have happened. Somehow a worker
  crashed in an unanticipated way causing the main_loop to throw an exception,
  which is being caught in "python/ray/workers/default_worker.py".
  .
F0114 21:12:54.648187 24761 raylet_extension.cc:222]  Check failed: _s.ok() [RayletClient] Failed to push errors to raylet.: IOError: [RayletClient] Connection closed unexpectedly.
*** Check failure stack trace: ***
Fatal Python error: Aborted

Stack (most recent call first):
  File "/home/stephanie/ray-fork/python/ray/utils.py", line 78 in push_error_to_driver
  File "/home/stephanie/ray-fork/python/ray/workers/default_worker.py", line 108 in <module>
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0114 21:12:54.789479 24662 raylet_extension.cc:66]  Check failed: _s.ok() [RayletClient] Failed to get a task from raylet.: IOError: [RayletClient] Raylet connection closed.
*** Check failure stack trace: ***
Fatal Python error: Aborted

Stack (most recent call first):
  File "/home/stephanie/ray-fork/python/ray/worker.py", line 971 in _get_next_task_from_local_scheduler
  File "/home/stephanie/ray-fork/python/ray/worker.py", line 988 in main_loop
  File "/home/stephanie/ray-fork/python/ray/workers/default_worker.py", line 101 in <module>
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0114 21:12:54.921722 24531 raylet_extension.cc:66]  Check failed: _s.ok() [RayletClient] Failed to get a task from raylet.: IOError: [RayletClient] Raylet connection closed.
*** Check failure stack trace: ***
Fatal Python error: Aborted

Stack (most recent call first):
  File "/home/stephanie/ray-fork/python/ray/worker.py", line 971 in _get_next_task_from_local_scheduler
  File "/home/stephanie/ray-fork/python/ray/worker.py", line 988 in main_loop
  File "/home/stephanie/ray-fork/python/ray/workers/default_worker.py", line 101 in <module>
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0114 21:12:55.063496 24411 raylet_extension.cc:66]  Check failed: _s.ok() [RayletClient] Failed to get a task from raylet.: IOError: [RayletClient] Raylet connection closed.
*** Check failure stack trace: ***
Fatal Python error: Aborted

Stack (most recent call first):
  File "/home/stephanie/ray-fork/python/ray/worker.py", line 971 in _get_next_task_from_local_scheduler
  File "/home/stephanie/ray-fork/python/ray/worker.py", line 988 in main_loop
  File "/home/stephanie/ray-fork/python/ray/workers/default_worker.py", line 101 in <module>
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0114 21:12:55.179350 24293 raylet_extension.cc:66]  Check failed: _s.ok() [RayletClient] Failed to get a task from raylet.: IOError: [RayletClient] Raylet connection closed.
*** Check failure stack trace: ***
Fatal Python error: Aborted

Stack (most recent call first):
  File "/home/stephanie/ray-fork/python/ray/worker.py", line 971 in _get_next_task_from_local_scheduler
  File "/home/stephanie/ray-fork/python/ray/worker.py", line 988 in main_loop
  File "/home/stephanie/ray-fork/python/ray/workers/default_worker.py", line 101 in <module>
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0114 21:12:55.858193 24218 node_manager.cc:1791] Resubmitting task 00000000776fda355d409df1da82dfa77ebed1df on client 4f1a8371c9f4d7a56e2ba43b43bbd6cd8e776fef
I0114 21:12:55.858343 24218 node_manager.cc:1791] Resubmitting task 00000000847b2dd8b493ec1c92ecd332b0a4cafa on client 4f1a8371c9f4d7a56e2ba43b43bbd6cd8e776fef
I0114 21:12:55.858425 24218 node_manager.cc:1856] Failed to forward task 00000000847b2dd8b493ec1c92ecd332b0a4cafa to node manager 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4
I0114 21:12:56.058207 24218 node_manager.cc:1791] Resubmitting task 00000000776fda355d409df1da82dfa77ebed1df on client 4f1a8371c9f4d7a56e2ba43b43bbd6cd8e776fef
I0114 21:12:56.058305 24218 node_manager.cc:1856] Failed to forward task 00000000776fda355d409df1da82dfa77ebed1df to node manager 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4
I0114 21:12:56.858549 24218 node_manager.cc:1880] Resubmitting task 00000000847b2dd8b493ec1c92ecd332b0a4cafa because ForwardTask failed.
I0114 21:12:56.858649 24218 node_manager.cc:1856] Failed to forward task 00000000847b2dd8b493ec1c92ecd332b0a4cafa to node manager 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4
I0114 21:12:56.861583 24218 node_manager.cc:1791] Resubmitting task 00000000cfce33ca7a65392caa9b9e871293b93d on client 4f1a8371c9f4d7a56e2ba43b43bbd6cd8e776fef
I0114 21:12:57.058441 24218 node_manager.cc:1880] Resubmitting task 00000000776fda355d409df1da82dfa77ebed1df because ForwardTask failed.
I0114 21:12:57.061815 24218 node_manager.cc:1791] Resubmitting task 00000000cfce33ca7a65392caa9b9e871293b93d on client 4f1a8371c9f4d7a56e2ba43b43bbd6cd8e776fef
W0114 21:12:57.061836 24218 node_manager.cc:1231] Task 00000000cfce33ca7a65392caa9b9e871293b93d already in lineage cache. This is most likely due to reconstruction.
I0114 21:12:57.262033 24218 node_manager.cc:1791] Resubmitting task 00000000cfce33ca7a65392caa9b9e871293b93d on client 4f1a8371c9f4d7a56e2ba43b43bbd6cd8e776fef
W0114 21:12:57.262053 24218 node_manager.cc:1231] Task 00000000cfce33ca7a65392caa9b9e871293b93d already in lineage cache. This is most likely due to reconstruction.
I0114 21:12:57.462201 24218 node_manager.cc:1791] Resubmitting task 00000000cfce33ca7a65392caa9b9e871293b93d on client 4f1a8371c9f4d7a56e2ba43b43bbd6cd8e776fef
W0114 21:12:57.462237 24218 node_manager.cc:1231] Task 00000000cfce33ca7a65392caa9b9e871293b93d already in lineage cache. This is most likely due to reconstruction.
I0114 21:12:57.858714 24218 node_manager.cc:1880] Resubmitting task 00000000847b2dd8b493ec1c92ecd332b0a4cafa because ForwardTask failed.
I0114 21:12:58.867885 24218 node_manager.cc:1791] Resubmitting task 0000000015ada82c652c2418726d361f194ee08d on client 4f1a8371c9f4d7a56e2ba43b43bbd6cd8e776fef
I0114 21:12:59.068208 24218 node_manager.cc:1791] Resubmitting task 0000000015ada82c652c2418726d361f194ee08d on client 4f1a8371c9f4d7a56e2ba43b43bbd6cd8e776fef
W0114 21:12:59.068244 24218 node_manager.cc:1231] Task 0000000015ada82c652c2418726d361f194ee08d already in lineage cache. This is most likely due to reconstruction.
I0114 21:12:59.268471 24218 node_manager.cc:1791] Resubmitting task 0000000015ada82c652c2418726d361f194ee08d on client 4f1a8371c9f4d7a56e2ba43b43bbd6cd8e776fef
W0114 21:12:59.268522 24218 node_manager.cc:1231] Task 0000000015ada82c652c2418726d361f194ee08d already in lineage cache. This is most likely due to reconstruction.
I0114 21:12:59.468752 24218 node_manager.cc:1791] Resubmitting task 0000000015ada82c652c2418726d361f194ee08d on client 4f1a8371c9f4d7a56e2ba43b43bbd6cd8e776fef
W0114 21:12:59.468786 24218 node_manager.cc:1231] Task 0000000015ada82c652c2418726d361f194ee08d already in lineage cache. This is most likely due to reconstruction.
I0114 21:12:59.870695 24218 node_manager.cc:1791] Resubmitting task 00000000f364a2a176241a745f461c0960c3b544 on client 4f1a8371c9f4d7a56e2ba43b43bbd6cd8e776fef
I0114 21:13:00.070829 24218 node_manager.cc:1791] Resubmitting task 00000000f364a2a176241a745f461c0960c3b544 on client 4f1a8371c9f4d7a56e2ba43b43bbd6cd8e776fef
W0114 21:13:00.070853 24218 node_manager.cc:1231] Task 00000000f364a2a176241a745f461c0960c3b544 already in lineage cache. This is most likely due to reconstruction.
I0114 21:13:00.271056 24218 node_manager.cc:1791] Resubmitting task 00000000f364a2a176241a745f461c0960c3b544 on client 4f1a8371c9f4d7a56e2ba43b43bbd6cd8e776fef
W0114 21:13:00.271096 24218 node_manager.cc:1231] Task 00000000f364a2a176241a745f461c0960c3b544 already in lineage cache. This is most likely due to reconstruction.
I0114 21:13:00.471858 24218 node_manager.cc:1791] Resubmitting task 00000000f364a2a176241a745f461c0960c3b544 on client 4f1a8371c9f4d7a56e2ba43b43bbd6cd8e776fef
W0114 21:13:00.471925 24218 node_manager.cc:1231] Task 00000000f364a2a176241a745f461c0960c3b544 already in lineage cache. This is most likely due to reconstruction.
round 0 dict_values([1]) ready: 1
round 1 dict_values([1, 1]) ready: 1
round 2 dict_values([1, 1, 1]) ready: 1
round 3 dict_values([1, 1, 1, 1]) ready: 1
round 4 dict_values([1, 1, 1, 1, 1]) ready: 1
round 5 dict_values([1, 1, 1, 1, 1, 1]) ready: 1
round 6 dict_values([1, 1, 1, 1, 1, 1, 1]) ready: 1
round 7 dict_values([1, 1, 1, 1, 1, 1, 1, 1]) ready: 1
round 8 dict_values([1, 1, 1, 1, 1, 1, 1, 1, 1]) ready: 1
round 9 dict_values([1, 1, 1, 1, 1, 1, 1, 1, 1, 1]) ready: 1
round 10 dict_values([1, 1, 1, 1, 2, 1, 1, 1, 1, 1]) ready: 1
round 11 dict_values([1, 1, 1, 2, 2, 1, 1, 1, 1, 1]) ready: 1
round 12 dict_values([1, 1, 1, 2, 2, 1, 1, 1, 1, 2]) ready: 1
round 13 dict_values([1, 1, 2, 2, 2, 1, 1, 1, 1, 2]) ready: 1
round 14 dict_values([1, 1, 2, 2, 2, 1, 2, 1, 1, 2]) ready: 1
round 15 dict_values([1, 2, 2, 2, 2, 1, 2, 1, 1, 2]) ready: 1
round 16 dict_values([1, 2, 2, 2, 2, 1, 2, 1, 2, 2]) ready: 1
round 17 dict_values([1, 2, 2, 2, 2, 2, 2, 1, 2, 2]) ready: 1
round 18 dict_values([1, 2, 2, 2, 2, 2, 2, 2, 2, 2]) ready: 1
round 19 dict_values([2, 2, 2, 2, 2, 2, 2, 2, 2, 2]) ready: 1
round 20 dict_values([2, 2, 2, 2, 2, 2, 3, 2, 2, 2]) ready: 1
round 21 dict_values([2, 2, 2, 2, 3, 2, 3, 2, 2, 2]) ready: 1
round 22 dict_values([2, 2, 2, 2, 3, 2, 3, 2, 2, 3]) ready: 1
round 23 dict_values([2, 2, 3, 2, 3, 2, 3, 2, 2, 3]) ready: 1
round 24 dict_values([2, 3, 3, 2, 3, 2, 3, 2, 2, 3]) ready: 1
round 25 dict_values([2, 3, 3, 2, 3, 2, 3, 2, 3, 3]) ready: 1
round 26 dict_values([2, 3, 3, 3, 3, 2, 3, 2, 3, 3]) ready: 1
round 27 dict_values([2, 3, 3, 3, 3, 2, 3, 3, 3, 3]) ready: 1
round 28 dict_values([2, 3, 3, 3, 3, 3, 3, 3, 3, 3]) ready: 1
round 29 dict_values([3, 3, 3, 3, 3, 3, 3, 3, 3, 3]) ready: 1
round 30 dict_values([3, 3, 3, 3, 3, 3, 3, 3, 4, 3]) ready: 1
round 31 dict_values([3, 3, 3, 3, 4, 3, 3, 3, 4, 3]) ready: 1
round 32 dict_values([3, 3, 4, 3, 4, 3, 3, 3, 4, 3]) ready: 1
round 33 dict_values([3, 3, 4, 4, 4, 3, 3, 3, 4, 3]) ready: 1
round 34 dict_values([3, 4, 4, 4, 4, 3, 3, 3, 4, 3]) ready: 1
round 35 dict_values([3, 4, 4, 4, 4, 3, 3, 3, 4, 4]) ready: 1
round 36 dict_values([3, 4, 4, 4, 4, 3, 3, 4, 4, 4]) ready: 1
round 37 dict_values([3, 4, 4, 4, 4, 4, 3, 4, 4, 4]) ready: 1
round 38 dict_values([3, 4, 4, 4, 4, 4, 4, 4, 4, 4]) ready: 1
round 39 dict_values([4, 4, 4, 4, 4, 4, 4, 4, 4, 4]) ready: 1
round 40 dict_values([4, 4, 4, 4, 4, 4, 5, 4, 4, 4]) ready: 1
round 41 dict_values([4, 4, 5, 4, 4, 4, 5, 4, 4, 4]) ready: 1
round 42 dict_values([4, 4, 5, 4, 5, 4, 5, 4, 4, 4]) ready: 1
round 43 dict_values([4, 5, 5, 4, 5, 4, 5, 4, 4, 4]) ready: 1
round 44 dict_values([4, 5, 5, 4, 5, 4, 5, 4, 4, 5]) ready: 1
round 45 dict_values([4, 5, 5, 5, 5, 4, 5, 4, 4, 5]) ready: 1
round 46 dict_values([4, 5, 5, 5, 5, 5, 5, 4, 4, 5]) ready: 1
round 47 dict_values([4, 5, 5, 5, 5, 5, 5, 4, 5, 5]) ready: 1
round 48 dict_values([4, 5, 5, 5, 5, 5, 5, 5, 5, 5]) ready: 1
round 49 dict_values([5, 5, 5, 5, 5, 5, 5, 5, 5, 5]) ready: 1
round 50 dict_values([5, 5, 5, 5, 5, 5, 6, 5, 5, 5]) ready: 1
round 51 dict_values([5, 5, 5, 5, 5, 5, 6, 5, 5, 6]) ready: 1
round 52 dict_values([5, 5, 6, 5, 5, 5, 6, 5, 5, 6]) ready: 1
round 53 dict_values([5, 5, 6, 5, 6, 5, 6, 5, 5, 6]) ready: 1
round 54 dict_values([5, 6, 6, 5, 6, 5, 6, 5, 5, 6]) ready: 1
round 55 dict_values([5, 6, 6, 6, 6, 5, 6, 5, 5, 6]) ready: 1
round 56 dict_values([5, 6, 6, 6, 6, 6, 6, 5, 5, 6]) ready: 1
round 57 dict_values([5, 6, 6, 6, 6, 6, 6, 6, 5, 6]) ready: 1
round 58 dict_values([5, 6, 6, 6, 6, 6, 6, 6, 6, 6]) ready: 1
round 59 dict_values([5, 6, 6, 6, 6, 6, 7, 6, 6, 6]) ready: 1
round 60 dict_values([5, 6, 6, 6, 6, 6, 7, 6, 7, 6]) ready: 1
round 61 dict_values([5, 6, 6, 6, 7, 6, 7, 6, 7, 6]) ready: 1
round 62 dict_values([5, 6, 7, 6, 7, 6, 7, 6, 7, 6]) ready: 1
round 63 dict_values([5, 7, 7, 6, 7, 6, 7, 6, 7, 6]) ready: 1
round 64 dict_values([5, 7, 7, 6, 7, 6, 7, 6, 7, 7]) ready: 1
round 65 dict_values([5, 7, 7, 7, 7, 6, 7, 6, 7, 7]) ready: 1
round 66 dict_values([5, 7, 7, 7, 7, 7, 7, 6, 7, 7]) ready: 1
round 67 dict_values([5, 7, 7, 7, 7, 7, 7, 7, 7, 7]) ready: 1
round 68 dict_values([5, 7, 7, 7, 7, 7, 8, 7, 7, 7]) ready: 1
round 69 dict_values([5, 7, 7, 7, 7, 7, 8, 7, 8, 7]) ready: 1
round 70 dict_values([5, 7, 8, 7, 7, 7, 8, 7, 8, 7]) ready: 1
round 71 dict_values([5, 7, 8, 7, 8, 7, 8, 7, 8, 7]) ready: 1
round 72 dict_values([5, 7, 8, 7, 8, 7, 8, 8, 8, 7]) ready: 1
round 73 dict_values([5, 7, 8, 7, 8, 8, 8, 8, 8, 7]) ready: 1
round 74 dict_values([5, 8, 8, 7, 8, 8, 8, 8, 8, 7]) ready: 1
round 75 dict_values([5, 8, 8, 7, 8, 8, 8, 8, 8, 8]) ready: 1
round 76 dict_values([5, 8, 8, 8, 8, 8, 8, 8, 8, 8]) ready: 1
round 77 dict_values([5, 8, 8, 8, 9, 8, 8, 8, 8, 8]) ready: 1
round 78 dict_values([5, 8, 8, 8, 9, 8, 9, 8, 8, 8]) ready: 1
round 79 dict_values([5, 8, 8, 8, 9, 8, 9, 8, 9, 8]) ready: 1
round 80 dict_values([5, 8, 8, 8, 9, 8, 9, 9, 9, 8]) ready: 1
round 81 dict_values([5, 8, 8, 8, 9, 9, 9, 9, 9, 8]) ready: 1
round 82 dict_values([5, 8, 9, 8, 9, 9, 9, 9, 9, 8]) ready: 1
round 83 dict_values([5, 9, 9, 8, 9, 9, 9, 9, 9, 8]) ready: 1
round 84 dict_values([5, 9, 9, 9, 9, 9, 9, 9, 9, 8]) ready: 1
round 85 dict_values([5, 9, 9, 9, 9, 9, 9, 9, 9, 9]) ready: 1
round 86 dict_values([5, 9, 9, 9, 9, 9, 9, 9, 10, 9]) ready: 1
round 87 dict_values([5, 9, 9, 10, 9, 9, 9, 9, 10, 9]) ready: 1
round 88 dict_values([5, 9, 9, 10, 10, 9, 9, 9, 10, 9]) ready: 1
round 89 dict_values([5, 9, 9, 10, 10, 9, 9, 10, 10, 9]) ready: 1
round 90 dict_values([5, 9, 9, 10, 10, 9, 9, 10, 10, 10]) ready: 1
round 91 dict_values([5, 9, 9, 10, 10, 10, 9, 10, 10, 10]) ready: 1
round 92 dict_values([5, 9, 9, 10, 10, 10, 10, 10, 10, 10]) ready: 1
round 93 dict_values([5, 10, 9, 10, 10, 10, 10, 10, 10, 10]) ready: 1
round 94 dict_values([5, 10, 10, 10, 10, 10, 10, 10, 10, 10]) ready: 1
round 95 dict_values([5, 10, 10, 10, 10, 10, 11, 10, 10, 10]) ready: 1
round 96 dict_values([5, 10, 11, 10, 10, 10, 11, 10, 10, 10]) ready: 1
round 97 dict_values([5, 11, 11, 10, 10, 10, 11, 10, 10, 10]) ready: 1
round 98 dict_values([5, 11, 11, 11, 10, 10, 11, 10, 10, 10]) ready: 1
round 99 dict_values([5, 11, 11, 11, 11, 10, 11, 10, 10, 10]) ready: 1
round 100 dict_values([5, 11, 11, 11, 11, 10, 11, 10, 10, 11]) ready: 1
round 101 dict_values([5, 11, 11, 11, 11, 10, 11, 11, 10, 11]) ready: 1
round 102 dict_values([5, 11, 11, 11, 11, 11, 11, 11, 10, 11]) ready: 1
round 103 dict_values([5, 11, 11, 11, 11, 11, 11, 11, 11, 11]) ready: 1
round 104 dict_values([5, 11, 11, 11, 11, 11, 11, 11, 12, 11]) ready: 1
round 105 dict_values([5, 11, 11, 11, 12, 11, 11, 11, 12, 11]) ready: 1
round 106 dict_values([5, 11, 12, 11, 12, 11, 11, 11, 12, 11]) ready: 1
round 107 dict_values([5, 12, 12, 11, 12, 11, 11, 11, 12, 11]) ready: 1
round 108 dict_values([5, 12, 12, 12, 12, 11, 11, 11, 12, 11]) ready: 1
round 109 dict_values([5, 12, 12, 12, 12, 12, 11, 11, 12, 11]) ready: 1
round 110 dict_values([5, 12, 12, 12, 12, 12, 11, 11, 12, 12]) ready: 1
round 111 dict_values([5, 12, 12, 12, 12, 12, 12, 11, 12, 12]) ready: 1
round 112 dict_values([5, 12, 12, 12, 12, 12, 12, 12, 12, 12]) ready: 1
round 113 dict_values([5, 12, 12, 12, 12, 12, 13, 12, 12, 12]) ready: 1
round 114 dict_values([5, 12, 12, 12, 12, 12, 13, 12, 13, 12]) ready: 1
round 115 dict_values([5, 12, 13, 12, 12, 12, 13, 12, 13, 12]) ready: 1
round 116 dict_values([5, 12, 13, 12, 12, 12, 13, 12, 13, 13]) ready: 1
round 117 dict_values([5, 13, 13, 12, 12, 12, 13, 12, 13, 13]) ready: 1
round 118 dict_values([5, 13, 13, 13, 12, 12, 13, 12, 13, 13]) ready: 1
round 119 dict_values([5, 13, 13, 13, 13, 12, 13, 12, 13, 13]) ready: 1
round 120 dict_values([5, 13, 13, 13, 13, 13, 13, 12, 13, 13]) ready: 1
round 121 dict_values([5, 13, 13, 13, 13, 13, 13, 13, 13, 13]) ready: 1
round 122 dict_values([5, 13, 13, 13, 14, 13, 13, 13, 13, 13]) ready: 1
round 123 dict_values([5, 13, 14, 13, 14, 13, 13, 13, 13, 13]) ready: 1
round 124 dict_values([5, 13, 14, 13, 14, 13, 13, 13, 14, 13]) ready: 1
round 125 dict_values([5, 14, 14, 13, 14, 13, 13, 13, 14, 13]) ready: 1
round 126 dict_values([5, 14, 14, 14, 14, 13, 13, 13, 14, 13]) ready: 1
round 127 dict_values([5, 14, 14, 14, 14, 13, 13, 13, 14, 14]) ready: 1
round 128 dict_values([5, 14, 14, 14, 14, 13, 14, 13, 14, 14]) ready: 1
round 129 dict_values([5, 14, 14, 14, 14, 14, 14, 13, 14, 14]) ready: 1
round 130 dict_values([5, 14, 14, 14, 14, 14, 14, 14, 14, 14]) ready: 1
round 131 dict_values([5, 14, 14, 14, 14, 14, 15, 14, 14, 14]) ready: 1
round 132 dict_values([5, 14, 14, 14, 14, 15, 15, 14, 14, 14]) ready: 1
round 133 dict_values([5, 14, 14, 14, 14, 15, 15, 14, 15, 14]) ready: 1
round 134 dict_values([5, 14, 15, 14, 14, 15, 15, 14, 15, 14]) ready: 1
round 135 dict_values([5, 14, 15, 14, 15, 15, 15, 14, 15, 14]) ready: 1
round 136 dict_values([5, 15, 15, 14, 15, 15, 15, 14, 15, 14]) ready: 1
round 137 dict_values([5, 15, 15, 14, 15, 15, 15, 14, 15, 15]) ready: 1
round 138 dict_values([5, 15, 15, 15, 15, 15, 15, 14, 15, 15]) ready: 1
round 139 dict_values([5, 15, 15, 15, 15, 15, 15, 15, 15, 15]) ready: 1
round 140 dict_values([5, 15, 15, 15, 15, 15, 15, 15, 16, 15]) ready: 1
round 141 dict_values([5, 15, 15, 15, 15, 15, 16, 15, 16, 15]) ready: 1
round 142 dict_values([5, 15, 15, 15, 15, 16, 16, 15, 16, 15]) ready: 1
round 143 dict_values([5, 15, 15, 15, 15, 16, 16, 16, 16, 15]) ready: 1
round 144 dict_values([5, 15, 16, 15, 15, 16, 16, 16, 16, 15]) ready: 1
round 145 dict_values([5, 16, 16, 15, 15, 16, 16, 16, 16, 15]) ready: 1
round 146 dict_values([5, 16, 16, 15, 15, 16, 16, 16, 16, 16]) ready: 1
round 147 dict_values([5, 16, 16, 15, 16, 16, 16, 16, 16, 16]) ready: 1
round 148 dict_values([5, 16, 16, 16, 16, 16, 16, 16, 16, 16]) ready: 1
round 149 dict_values([5, 16, 16, 16, 16, 16, 17, 16, 16, 16]) ready: 1
round 150 dict_values([5, 16, 16, 16, 17, 16, 17, 16, 16, 16]) ready: 1
round 151 dict_values([5, 16, 16, 16, 17, 16, 17, 17, 16, 16]) ready: 1
round 152 dict_values([5, 16, 16, 16, 17, 17, 17, 17, 16, 16]) ready: 1
round 153 dict_values([5, 16, 17, 16, 17, 17, 17, 17, 16, 16]) ready: 1
round 154 dict_values([5, 16, 17, 16, 17, 17, 17, 17, 16, 17]) ready: 1
round 155 dict_values([5, 17, 17, 16, 17, 17, 17, 17, 16, 17]) ready: 1
round 156 dict_values([5, 17, 17, 16, 17, 17, 17, 17, 17, 17]) ready: 1
round 157 dict_values([5, 17, 17, 17, 17, 17, 17, 17, 17, 17]) ready: 1
round 158 dict_values([5, 17, 17, 17, 17, 17, 17, 17, 18, 17]) ready: 1
round 159 dict_values([5, 17, 17, 17, 18, 17, 17, 17, 18, 17]) ready: 1
round 160 dict_values([5, 17, 17, 17, 18, 17, 17, 17, 18, 18]) ready: 1
round 161 dict_values([5, 17, 17, 17, 18, 17, 17, 18, 18, 18]) ready: 1
round 162 dict_values([5, 17, 18, 17, 18, 17, 17, 18, 18, 18]) ready: 1
round 163 dict_values([5, 18, 18, 17, 18, 17, 17, 18, 18, 18]) ready: 1
round 164 dict_values([5, 18, 18, 17, 18, 18, 17, 18, 18, 18]) ready: 1
round 165 dict_values([5, 18, 18, 17, 18, 18, 18, 18, 18, 18]) ready: 1
round 166 dict_values([5, 18, 18, 18, 18, 18, 18, 18, 18, 18]) ready: 1
round 167 dict_values([5, 18, 18, 18, 18, 18, 19, 18, 18, 18]) ready: 1
round 168 dict_values([5, 18, 18, 18, 18, 18, 19, 18, 19, 18]) ready: 1
round 169 dict_values([5, 18, 18, 19, 18, 18, 19, 18, 19, 18]) ready: 1
round 170 dict_values([5, 18, 18, 19, 18, 18, 19, 18, 19, 19]) ready: 1
round 171 dict_values([5, 18, 18, 19, 19, 18, 19, 18, 19, 19]) ready: 1
round 172 dict_values([5, 18, 18, 19, 19, 19, 19, 18, 19, 19]) ready: 1
round 173 dict_values([5, 18, 19, 19, 19, 19, 19, 18, 19, 19]) ready: 1
round 174 dict_values([5, 19, 19, 19, 19, 19, 19, 18, 19, 19]) ready: 1
round 175 dict_values([5, 19, 19, 19, 19, 19, 19, 19, 19, 19]) ready: 1
round 176 dict_values([5, 19, 19, 19, 19, 19, 20, 19, 19, 19]) ready: 1
round 177 dict_values([5, 19, 19, 20, 19, 19, 20, 19, 19, 19]) ready: 1
round 178 dict_values([5, 19, 19, 20, 19, 19, 20, 19, 20, 19]) ready: 1
round 179 dict_values([5, 19, 19, 20, 20, 19, 20, 19, 20, 19]) ready: 1
round 180 dict_values([5, 20, 19, 20, 20, 19, 20, 19, 20, 19]) ready: 1
round 181 dict_values([5, 20, 19, 20, 20, 20, 20, 19, 20, 19]) ready: 1
round 182 dict_values([5, 20, 20, 20, 20, 20, 20, 19, 20, 19]) ready: 1
round 183 dict_values([5, 20, 20, 20, 20, 20, 20, 19, 20, 20]) ready: 1
round 184 dict_values([5, 20, 20, 20, 20, 20, 20, 20, 20, 20]) ready: 1
round 185 dict_values([5, 20, 20, 20, 20, 20, 21, 20, 20, 20]) ready: 1
round 186 dict_values([5, 20, 20, 20, 21, 20, 21, 20, 20, 20]) ready: 1
round 187 dict_values([5, 20, 21, 20, 21, 20, 21, 20, 20, 20]) ready: 1
round 188 dict_values([5, 21, 21, 20, 21, 20, 21, 20, 20, 20]) ready: 1
round 189 dict_values([5, 21, 21, 20, 21, 20, 21, 20, 21, 20]) ready: 1
round 190 dict_values([5, 21, 21, 21, 21, 20, 21, 20, 21, 20]) ready: 1
round 191 dict_values([5, 21, 21, 21, 21, 21, 21, 20, 21, 20]) ready: 1
round 192 dict_values([5, 21, 21, 21, 21, 21, 21, 20, 21, 21]) ready: 1
round 193 dict_values([5, 21, 21, 21, 21, 21, 21, 21, 21, 21]) ready: 1
round 194 dict_values([5, 21, 21, 21, 21, 21, 22, 21, 21, 21]) ready: 1
round 195 dict_values([5, 21, 22, 21, 21, 21, 22, 21, 21, 21]) ready: 1
round 196 dict_values([5, 22, 22, 21, 21, 21, 22, 21, 21, 21]) ready: 1
round 197 dict_values([5, 22, 22, 22, 21, 21, 22, 21, 21, 21]) ready: 1
round 198 dict_values([5, 22, 22, 22, 22, 21, 22, 21, 21, 21]) ready: 1
round 199 dict_values([5, 22, 22, 22, 22, 21, 22, 21, 22, 21]) ready: 1
round 200 dict_values([5, 22, 22, 22, 22, 21, 22, 21, 22, 22]) ready: 1
round 201 dict_values([5, 22, 22, 22, 22, 21, 22, 22, 22, 22]) ready: 1
round 202 dict_values([5, 22, 22, 22, 22, 22, 22, 22, 22, 22]) ready: 1
round 203 dict_values([5, 22, 22, 22, 22, 22, 23, 22, 22, 22]) ready: 1
round 204 dict_values([5, 22, 23, 22, 22, 22, 23, 22, 22, 22]) ready: 1
round 205 dict_values([5, 22, 23, 22, 22, 22, 23, 22, 22, 23]) ready: 1
round 206 dict_values([5, 23, 23, 22, 22, 22, 23, 22, 22, 23]) ready: 1
round 207 dict_values([5, 23, 23, 22, 22, 23, 23, 22, 22, 23]) ready: 1
round 208 dict_values([5, 23, 23, 22, 22, 23, 23, 23, 22, 23]) ready: 1
round 209 dict_values([5, 23, 23, 22, 23, 23, 23, 23, 22, 23]) ready: 1
round 210 dict_values([5, 23, 23, 23, 23, 23, 23, 23, 22, 23]) ready: 1
round 211 dict_values([5, 23, 23, 23, 23, 23, 23, 23, 23, 23]) ready: 1
round 212 dict_values([5, 23, 23, 23, 23, 23, 24, 23, 23, 23]) ready: 1
round 213 dict_values([5, 23, 24, 23, 23, 23, 24, 23, 23, 23]) ready: 1
round 214 dict_values([5, 23, 24, 23, 23, 23, 24, 23, 23, 24]) ready: 1
round 215 dict_values([5, 23, 24, 23, 23, 24, 24, 23, 23, 24]) ready: 1
round 216 dict_values([5, 23, 24, 23, 23, 24, 24, 24, 23, 24]) ready: 1
round 217 dict_values([5, 24, 24, 23, 23, 24, 24, 24, 23, 24]) ready: 1
round 218 dict_values([5, 24, 24, 24, 23, 24, 24, 24, 23, 24]) ready: 1
round 219 dict_values([5, 24, 24, 24, 24, 24, 24, 24, 23, 24]) ready: 1
round 220 dict_values([5, 24, 24, 24, 24, 24, 24, 24, 24, 24]) ready: 1
round 221 dict_values([5, 24, 24, 24, 24, 24, 25, 24, 24, 24]) ready: 1
round 222 dict_values([5, 24, 24, 24, 25, 24, 25, 24, 24, 24]) ready: 1
round 223 dict_values([5, 24, 24, 24, 25, 24, 25, 24, 25, 24]) ready: 1
round 224 dict_values([5, 24, 25, 24, 25, 24, 25, 24, 25, 24]) ready: 1
round 225 dict_values([5, 25, 25, 24, 25, 24, 25, 24, 25, 24]) ready: 1
round 226 dict_values([5, 25, 25, 24, 25, 24, 25, 24, 25, 25]) ready: 1
round 227 dict_values([5, 25, 25, 24, 25, 24, 25, 25, 25, 25]) ready: 1
round 228 dict_values([5, 25, 25, 24, 25, 25, 25, 25, 25, 25]) ready: 1
round 229 dict_values([5, 25, 25, 25, 25, 25, 25, 25, 25, 25]) ready: 1
round 230 dict_values([5, 25, 25, 25, 26, 25, 25, 25, 25, 25]) ready: 1
round 231 dict_values([5, 26, 25, 25, 26, 25, 25, 25, 25, 25]) ready: 1
round 232 dict_values([5, 26, 25, 25, 26, 25, 25, 25, 26, 25]) ready: 1
round 233 dict_values([5, 26, 25, 25, 26, 25, 26, 25, 26, 25]) ready: 1
round 234 dict_values([5, 26, 26, 25, 26, 25, 26, 25, 26, 25]) ready: 1
round 235 dict_values([5, 26, 26, 25, 26, 25, 26, 25, 26, 26]) ready: 1
round 236 dict_values([5, 26, 26, 25, 26, 26, 26, 25, 26, 26]) ready: 1
round 237 dict_values([5, 26, 26, 26, 26, 26, 26, 25, 26, 26]) ready: 1
round 238 dict_values([5, 26, 26, 26, 26, 26, 26, 26, 26, 26]) ready: 1
round 239 dict_values([5, 26, 26, 26, 27, 26, 26, 26, 26, 26]) ready: 1
round 240 dict_values([5, 27, 26, 26, 27, 26, 26, 26, 26, 26]) ready: 1
round 241 dict_values([5, 27, 26, 26, 27, 26, 26, 26, 27, 26]) ready: 1
round 242 dict_values([5, 27, 27, 26, 27, 26, 26, 26, 27, 26])WARNING: Logging before InitGoogleLogging() is written to STDERR
WARNING: Logging before InitGoogleLogging() is written to STDERR
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0114 21:13:23.578975 24242 node_manager.cc:416] Actor 1043cb2eed39d8ad7ded2ffefa5547f74b354a5f is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.578949 24265 node_manager.cc:416] Actor 1043cb2eed39d8ad7ded2ffefa5547f74b354a5f is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0114 21:13:23.579005 24097 node_manager.cc:416] Actor 1043cb2eed39d8ad7ded2ffefa5547f74b354a5f is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.579111 24242 node_manager.cc:416] Actor 1d6495aa7e94c862f365953fa846b7f99a9c30ee is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.579113 24265 node_manager.cc:416] Actor 9175992b4bd11b03d85a01c0bf844a4b7d4f0b1d is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.579151 24097 node_manager.cc:416] Actor 9175992b4bd11b03d85a01c0bf844a4b7d4f0b1d is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.579155 24242 node_manager.cc:416] Actor 9175992b4bd11b03d85a01c0bf844a4b7d4f0b1d is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.579160 24265 node_manager.cc:416] Actor 1d6495aa7e94c862f365953fa846b7f99a9c30ee is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.579195 24097 node_manager.cc:416] Actor 1d6495aa7e94c862f365953fa846b7f99a9c30ee is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.579201 24242 node_manager.cc:416] Actor b22335c675464940c3d3b534fe6554808dc15734 is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.579207 24265 node_manager.cc:416] Actor b22335c675464940c3d3b534fe6554808dc15734 is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.579237 24097 node_manager.cc:416] Actor b22335c675464940c3d3b534fe6554808dc15734 is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.579237 24242 node_manager.cc:416] Actor 57f5548715fbc3ee865c8f80845521faef018f7f is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.579247 24265 node_manager.cc:416] Actor 57f5548715fbc3ee865c8f80845521faef018f7f is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.579274 24097 node_manager.cc:416] Actor 57f5548715fbc3ee865c8f80845521faef018f7f is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0114 21:13:23.579505 24122 node_manager.cc:416] Actor 1043cb2eed39d8ad7ded2ffefa5547f74b354a5f is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.579582 24122 node_manager.cc:416] Actor 9175992b4bd11b03d85a01c0bf844a4b7d4f0b1d is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.579617 24122 node_manager.cc:416] Actor 1d6495aa7e94c862f365953fa846b7f99a9c30ee is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.579636 24122 node_manager.cc:416] Actor b22335c675464940c3d3b534fe6554808dc15734 is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.579654 24122 node_manager.cc:416] Actor 57f5548715fbc3ee865c8f80845521faef018f7f is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0114 21:13:23.579767 24145 node_manager.cc:416] Actor 1043cb2eed39d8ad7ded2ffefa5547f74b354a5f is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.579787 24218 node_manager.cc:416] Actor 1043cb2eed39d8ad7ded2ffefa5547f74b354a5f is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.579829 24145 node_manager.cc:416] Actor 9175992b4bd11b03d85a01c0bf844a4b7d4f0b1d is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.579854 24145 node_manager.cc:416] Actor 1d6495aa7e94c862f365953fa846b7f99a9c30ee is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.579849 24218 node_manager.cc:416] Actor 9175992b4bd11b03d85a01c0bf844a4b7d4f0b1d is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.579874 24145 node_manager.cc:416] Actor b22335c675464940c3d3b534fe6554808dc15734 is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.579885 24218 node_manager.cc:416] Actor 1d6495aa7e94c862f365953fa846b7f99a9c30ee is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.579890 24145 node_manager.cc:416] Actor 57f5548715fbc3ee865c8f80845521faef018f7f is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.579921 24218 node_manager.cc:416] Actor b22335c675464940c3d3b534fe6554808dc15734 is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.579951 24218 node_manager.cc:416] Actor 57f5548715fbc3ee865c8f80845521faef018f7f is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.580090 24218 node_manager.cc:1856] Failed to forward task 00000000cfce33ca7a65392caa9b9e871293b93d to node manager 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4
I0114 21:13:23.580247 24218 node_manager.cc:1856] Failed to forward task 00000000776fda355d409df1da82dfa77ebed1df to node manager 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4
I0114 21:13:23.580327 24218 node_manager.cc:1856] Failed to forward task 00000000cfce33ca7a65392caa9b9e871293b93d to node manager 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4
I0114 21:13:23.580359 24218 node_manager.cc:1856] Failed to forward task 00000000cfce33ca7a65392caa9b9e871293b93d to node manager 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4
I0114 21:13:23.580389 24218 node_manager.cc:1856] Failed to forward task 00000000cfce33ca7a65392caa9b9e871293b93d to node manager 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4
I0114 21:13:23.580415 24218 node_manager.cc:1856] Failed to forward task 00000000847b2dd8b493ec1c92ecd332b0a4cafa to node manager 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4
I0114 21:13:23.580456 24218 node_manager.cc:1856] Failed to forward task 0000000015ada82c652c2418726d361f194ee08d to node manager 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4
I0114 21:13:23.580566 24218 node_manager.cc:1856] Failed to forward task 0000000015ada82c652c2418726d361f194ee08d to node manager 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4
I0114 21:13:23.580662 24218 node_manager.cc:1856] Failed to forward task 0000000015ada82c652c2418726d361f194ee08d to node manager 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4
I0114 21:13:23.580694 24218 node_manager.cc:1856] Failed to forward task 0000000015ada82c652c2418726d361f194ee08d to node manager 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4
I0114 21:13:23.580723 24218 node_manager.cc:1856] Failed to forward task 00000000f364a2a176241a745f461c0960c3b544 to node manager 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4
I0114 21:13:23.580809 24218 node_manager.cc:1856] Failed to forward task 00000000f364a2a176241a745f461c0960c3b544 to node manager 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4
I0114 21:13:23.578949 24194 node_manager.cc:416] Actor 1043cb2eed39d8ad7ded2ffefa5547f74b354a5f is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.580902 24218 node_manager.cc:1856] Failed to forward task 00000000f364a2a176241a745f461c0960c3b544 to node manager 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4
I0114 21:13:23.580946 24218 node_manager.cc:1856] Failed to forward task 00000000f364a2a176241a745f461c0960c3b544 to node manager 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4
I0114 21:13:23.581043 24194 node_manager.cc:416] Actor 1d6495aa7e94c862f365953fa846b7f99a9c30ee is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.581092 24194 node_manager.cc:416] Actor 9175992b4bd11b03d85a01c0bf844a4b7d4f0b1d is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0114 21:13:23.581064 24076 node_manager.cc:416] Actor 1043cb2eed39d8ad7ded2ffefa5547f74b354a5f is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0114 21:13:23.581105 24170 node_manager.cc:416] Actor 1043cb2eed39d8ad7ded2ffefa5547f74b354a5f is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.581164 24194 node_manager.cc:416] Actor b22335c675464940c3d3b534fe6554808dc15734 is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.581178 24076 node_manager.cc:416] Actor 1d6495aa7e94c862f365953fa846b7f99a9c30ee is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.581197 24194 node_manager.cc:416] Actor 57f5548715fbc3ee865c8f80845521faef018f7f is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.581198 24170 node_manager.cc:416] Actor 9175992b4bd11b03d85a01c0bf844a4b7d4f0b1d is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.581223 24076 node_manager.cc:416] Actor 9175992b4bd11b03d85a01c0bf844a4b7d4f0b1d is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.581234 24170 node_manager.cc:416] Actor 1d6495aa7e94c862f365953fa846b7f99a9c30ee is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.581267 24076 node_manager.cc:416] Actor b22335c675464940c3d3b534fe6554808dc15734 is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.581274 24170 node_manager.cc:416] Actor b22335c675464940c3d3b534fe6554808dc15734 is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.581302 24076 node_manager.cc:416] Actor 57f5548715fbc3ee865c8f80845521faef018f7f is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
I0114 21:13:23.581305 24170 node_manager.cc:416] Actor 57f5548715fbc3ee865c8f80845521faef018f7f is disconnected, because its node 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 is removed from cluster. It may be reconstructed.
The node with client ID 0e1a3eeec23c93eca1ee05eadd274ce625abf5d4 has been marked dead because the monitor has missed too many heartbeats from it.
I0114 21:13:24.580459 24218 node_manager.cc:1880] Resubmitting task 00000000cfce33ca7a65392caa9b9e871293b93d because ForwardTask failed.
I0114 21:13:24.580934 24218 node_manager.cc:1880] Resubmitting task 00000000776fda355d409df1da82dfa77ebed1df because ForwardTask failed.
I0114 21:13:24.581341 24218 node_manager.cc:1880] Resubmitting task 00000000cfce33ca7a65392caa9b9e871293b93d because ForwardTask failed.
W0114 21:13:24.581400 24218 node_manager.cc:1231] Task 00000000cfce33ca7a65392caa9b9e871293b93d already in lineage cache. This is most likely due to reconstruction.
I0114 21:13:24.581562 24218 node_manager.cc:1880] Resubmitting task 00000000cfce33ca7a65392caa9b9e871293b93d because ForwardTask failed.
W0114 21:13:24.581591 24218 node_manager.cc:1231] Task 00000000cfce33ca7a65392caa9b9e871293b93d already in lineage cache. This is most likely due to reconstruction.
I0114 21:13:24.581924 24218 node_manager.cc:1880] Resubmitting task 00000000cfce33ca7a65392caa9b9e871293b93d because ForwardTask failed.
W0114 21:13:24.581959 24218 node_manager.cc:1231] Task 00000000cfce33ca7a65392caa9b9e871293b93d already in lineage cache. This is most likely due to reconstruction.
I0114 21:13:24.582406 24218 node_manager.cc:1880] Resubmitting task 00000000847b2dd8b493ec1c92ecd332b0a4cafa because ForwardTask failed.
I0114 21:13:24.582707 24218 node_manager.cc:1880] Resubmitting task 0000000015ada82c652c2418726d361f194ee08d because ForwardTask failed.
I0114 21:13:24.583016 24218 node_manager.cc:1880] Resubmitting task 0000000015ada82c652c2418726d361f194ee08d because ForwardTask failed.
W0114 21:13:24.583047 24218 node_manager.cc:1231] Task 0000000015ada82c652c2418726d361f194ee08d already in lineage cache. This is most likely due to reconstruction.
I0114 21:13:24.583181 24218 node_manager.cc:1880] Resubmitting task 0000000015ada82c652c2418726d361f194ee08d because ForwardTask failed.
W0114 21:13:24.583200 24218 node_manager.cc:1231] Task 0000000015ada82c652c2418726d361f194ee08d already in lineage cache. This is most likely due to reconstruction.
I0114 21:13:24.583367 24218 node_manager.cc:1880] Resubmitting task 0000000015ada82c652c2418726d361f194ee08d because ForwardTask failed.
W0114 21:13:24.583386 24218 node_manager.cc:1231] Task 0000000015ada82c652c2418726d361f194ee08d already in lineage cache. This is most likely due to reconstruction.
I0114 21:13:24.583559 24218 node_manager.cc:1880] Resubmitting task 00000000f364a2a176241a745f461c0960c3b544 because ForwardTask failed.
I0114 21:13:24.583972 24218 node_manager.cc:1880] Resubmitting task 00000000f364a2a176241a745f461c0960c3b544 because ForwardTask failed.
W0114 21:13:24.584017 24218 node_manager.cc:1231] Task 00000000f364a2a176241a745f461c0960c3b544 already in lineage cache. This is most likely due to reconstruction.
I0114 21:13:24.584226 24218 node_manager.cc:1880] Resubmitting task 00000000f364a2a176241a745f461c0960c3b544 because ForwardTask failed.
W0114 21:13:24.584271 24218 node_manager.cc:1231] Task 00000000f364a2a176241a745f461c0960c3b544 already in lineage cache. This is most likely due to reconstruction.
I0114 21:13:24.584492 24218 node_manager.cc:1880] Resubmitting task 00000000f364a2a176241a745f461c0960c3b544 because ForwardTask failed.
W0114 21:13:24.584535 24218 node_manager.cc:1231] Task 00000000f364a2a176241a745f461c0960c3b544 already in lineage cache. This is most likely due to reconstruction.
 ready: 1
round 243 dict_values([5, 27, 27, 26, 27, 27, 26, 26, 27, 26]) ready: 1
round 244 dict_values([5, 27, 27, 26, 27, 27, 26, 26, 27, 27]) ready: 1
round 245 dict_values([5, 27, 27, 27, 27, 27, 26, 26, 27, 27]) ready: 1
round 246 dict_values([5, 27, 27, 27, 27, 27, 27, 26, 27, 27]) ready: 1
round 247 dict_values([5, 27, 27, 27, 27, 27, 27, 27, 27, 27]) ready: 1
round 248 dict_values([5, 27, 27, 27, 28, 27, 27, 27, 27, 27]) ready: 1
round 249 dict_values([5, 27, 27, 28, 28, 27, 27, 27, 27, 27]) ready: 1
round 250 dict_values([5, 27, 27, 28, 28, 27, 27, 27, 27, 28]) ready: 1
round 251 dict_values([5, 27, 27, 28, 28, 27, 27, 27, 28, 28]) ready: 1
round 252 dict_values([5, 27, 28, 28, 28, 27, 27, 27, 28, 28]) ready: 1
round 253 dict_values([5, 28, 28, 28, 28, 27, 27, 27, 28, 28]) ready: 1
round 254 dict_values([5, 28, 28, 28, 28, 27, 28, 27, 28, 28]) ready: 1
round 255 dict_values([5, 28, 28, 28, 28, 28, 28, 27, 28, 28]) ready: 1
round 256 dict_values([5, 28, 28, 28, 28, 28, 28, 28, 28, 28]) ready: 1
round 257 dict_values([5, 28, 28, 28, 28, 28, 29, 28, 28, 28]) ready: 1
round 258 dict_values([5, 28, 28, 28, 28, 28, 29, 29, 28, 28]) ready: 1
round 259 dict_values([5, 28, 28, 29, 28, 28, 29, 29, 28, 28]) ready: 1
round 260 dict_values([5, 28, 28, 29, 28, 29, 29, 29, 28, 28]) ready: 1
round 261 dict_values([5, 28, 28, 29, 29, 29, 29, 29, 28, 28]) ready: 1
round 262 dict_values([5, 28, 28, 29, 29, 29, 29, 29, 28, 29]) ready: 1
round 263 dict_values([5, 28, 28, 29, 29, 29, 29, 29, 29, 29]) ready: 1
round 264 dict_values([5, 28, 29, 29, 29, 29, 29, 29, 29, 29]) ready: 1
round 265 dict_values([5, 29, 29, 29, 29, 29, 29, 29, 29, 29]) ready: 1
round 266 dict_values([5, 30, 29, 29, 29, 29, 29, 29, 29, 29]) ready: 1
round 267 dict_values([5, 30, 29, 29, 29, 29, 30, 29, 29, 29]) ready: 1
round 268 dict_values([5, 30, 29, 29, 29, 30, 30, 29, 29, 29]) ready: 1
round 269 dict_values([5, 30, 29, 29, 29, 30, 30, 30, 29, 29]) ready: 1
round 270 dict_values([5, 30, 29, 29, 29, 30, 30, 30, 29, 30]) ready: 1
round 271 dict_values([5, 30, 29, 30, 29, 30, 30, 30, 29, 30]) ready: 1
round 272 dict_values([5, 30, 29, 30, 30, 30, 30, 30, 29, 30]) ready: 1
round 273 dict_values([5, 30, 30, 30, 30, 30, 30, 30, 29, 30]) ready: 1
round 274 dict_values([5, 30, 30, 30, 30, 30, 30, 30, 30, 30]) ready: 1
round 275 dict_values([5, 30, 30, 30, 30, 30, 30, 30, 31, 30]) ready: 1
round 276 dict_values([5, 30, 30, 30, 30, 30, 30, 30, 31, 31]) ready: 1
round 277 dict_values([5, 30, 31, 30, 30, 30, 30, 30, 31, 31]) ready: 1
round 278 dict_values([5, 31, 31, 30, 30, 30, 30, 30, 31, 31]) ready: 1
round 279 dict_values([5, 31, 31, 31, 30, 30, 30, 30, 31, 31]) ready: 1
round 280 dict_values([5, 31, 31, 31, 30, 30, 30, 31, 31, 31]) ready: 1
round 281 dict_values([5, 31, 31, 31, 31, 30, 30, 31, 31, 31]) ready: 1
round 282 dict_values([5, 31, 31, 31, 31, 31, 30, 31, 31, 31]) ready: 1
round 283 dict_values([5, 31, 31, 31, 31, 31, 31, 31, 31, 31]) ready: 1
round 284 dict_values([5, 31, 31, 31, 31, 31, 31, 31, 32, 31]) ready: 1
round 285 dict_values([5, 31, 31, 31, 31, 31, 31, 31, 32, 32]) ready: 1
round 286 dict_values([5, 31, 32, 31, 31, 31, 31, 31, 32, 32]) ready: 1
round 287 dict_values([5, 32, 32, 31, 31, 31, 31, 31, 32, 32]) ready: 1
round 288 dict_values([5, 32, 32, 31, 31, 31, 31, 32, 32, 32]) ready: 1
round 289 dict_values([5, 32, 32, 32, 31, 31, 31, 32, 32, 32]) ready: 1
round 290 dict_values([5, 32, 32, 32, 32, 31, 31, 32, 32, 32]) ready: 1
round 291 dict_values([5, 32, 32, 32, 32, 32, 31, 32, 32, 32]) ready: 1
round 292 dict_values([5, 32, 32, 32, 32, 32, 32, 32, 32, 32]) ready: 1
round 293 dict_values([5, 32, 32, 32, 32, 32, 32, 33, 32, 32]) ready: 1
round 294 dict_values([5, 32, 32, 32, 32, 32, 32, 33, 32, 33]) ready: 1
round 295 dict_values([5, 32, 32, 32, 33, 32, 32, 33, 32, 33]) ready: 1
round 296 dict_values([5, 32, 33, 32, 33, 32, 32, 33, 32, 33]) ready: 1
round 297 dict_values([5, 33, 33, 32, 33, 32, 32, 33, 32, 33]) ready: 1
round 298 dict_values([5, 33, 33, 32, 33, 32, 32, 33, 33, 33]) ready: 1
round 299 dict_values([5, 33, 33, 33, 33, 32, 32, 33, 33, 33]) ready: 1
round 300 dict_values([5, 33, 33, 33, 33, 33, 32, 33, 33, 33]) ready: 1
round 301 dict_values([5, 33, 33, 33, 33, 33, 33, 33, 33, 33]) ready: 1
round 302 dict_values([5, 33, 33, 33, 33, 33, 33, 33, 34, 33]) ready: 1
round 303 dict_values([5, 33, 33, 33, 34, 33, 33, 33, 34, 33]) ready: 1
round 304 dict_values([5, 33, 33, 33, 34, 33, 33, 33, 34, 34]) ready: 1
round 305 dict_values([5, 33, 34, 33, 34, 33, 33, 33, 34, 34]) ready: 1
round 306 dict_values([5, 34, 34, 33, 34, 33, 33, 33, 34, 34]) ready: 1
round 307 dict_values([5, 34, 34, 34, 34, 33, 33, 33, 34, 34]) ready: 1
round 308 dict_values([5, 34, 34, 34, 34, 34, 33, 33, 34, 34]) ready: 1
round 309 dict_values([5, 34, 34, 34, 34, 34, 34, 33, 34, 34]) ready: 1
round 310 dict_values([5, 34, 34, 34, 34, 34, 34, 34, 34, 34]) ready: 1
round 311 dict_values([5, 34, 34, 34, 34, 34, 34, 34, 35, 34]) ready: 1
round 312 dict_values([5, 34, 35, 34, 34, 34, 34, 34, 35, 34]) ready: 1
round 313 dict_values([5, 34, 35, 34, 34, 34, 34, 34, 35, 35]) ready: 1
round 314 dict_values([5, 34, 35, 34, 35, 34, 34, 34, 35, 35]) ready: 1
round 315 dict_values([5, 35, 35, 34, 35, 34, 34, 34, 35, 35]) ready: 1
round 316 dict_values([5, 35, 35, 35, 35, 34, 34, 34, 35, 35]) ready: 1
round 317 dict_values([5, 35, 35, 35, 35, 34, 35, 34, 35, 35]) ready: 1
round 318 dict_values([5, 35, 35, 35, 35, 35, 35, 34, 35, 35]) ready: 1
round 319 dict_values([5, 35, 35, 35, 35, 35, 35, 35, 35, 35]) ready: 1
round 320 dict_values([5, 36, 35, 35, 35, 35, 35, 35, 35, 35]) ready: 1
round 321 dict_values([5, 36, 35, 35, 35, 35, 36, 35, 35, 35]) ready: 1
round 322 dict_values([5, 36, 35, 35, 35, 35, 36, 35, 35, 36]) ready: 1
round 323 dict_values([5, 36, 35, 35, 35, 35, 36, 36, 35, 36]) ready: 1
round 324 dict_values([5, 36, 35, 35, 35, 36, 36, 36, 35, 36]) ready: 1
round 325 dict_values([5, 36, 35, 35, 36, 36, 36, 36, 35, 36]) ready: 1
round 326 dict_values([5, 36, 36, 35, 36, 36, 36, 36, 35, 36]) ready: 1
round 327 dict_values([5, 36, 36, 36, 36, 36, 36, 36, 35, 36]) ready: 1
round 328 dict_values([5, 36, 36, 36, 36, 36, 36, 36, 36, 36]) ready: 1
round 329 dict_values([6, 36, 36, 36, 36, 36, 36, 36, 36, 36]) ready: 1
round 330 dict_values([7, 36, 36, 36, 36, 36, 36, 36, 36, 36]) ready: 1
round 331 dict_values([8, 36, 36, 36, 36, 36, 36, 36, 36, 36]) ready: 1
round 332 dict_values([9, 36, 36, 36, 36, 36, 36, 36, 36, 36]) ready: 1
round 333 dict_values([10, 36, 36, 36, 36, 36, 36, 36, 36, 36]) ready: 1
round 334 dict_values([11, 36, 36, 36, 36, 36, 36, 36, 36, 36]) ready: 1
round 335 dict_values([12, 36, 36, 36, 36, 36, 36, 36, 36, 36]) ready: 1
round 336 dict_values([13, 36, 36, 36, 36, 36, 36, 36, 36, 36]) ready: 1
round 337 dict_values([14, 36, 36, 36, 36, 36, 36, 36, 36, 36]) ready: 1
round 338 dict_values([15, 36, 36, 36, 36, 36, 36, 36, 36, 36]) ready: 1
round 339 dict_values([16, 36, 36, 36, 36, 36, 36, 36, 36, 36]) ready: 1
round 340 dict_values([17, 36, 36, 36, 36, 36, 36, 36, 36, 36]) ready: 1
round 341 dict_values([18, 36, 36, 36, 36, 36, 36, 36, 36, 36]) ready: 1
round 342 dict_values([19, 36, 36, 36, 36, 36, 36, 36, 36, 36]) ready: 1
round 343 dict_values([20, 36, 36, 36, 36, 36, 36, 36, 36, 36]) ready: 1
round 344 dict_values([21, 36, 36, 36, 36, 36, 36, 36, 36, 36]) ready: 1
round 345 dict_values([22, 36, 36, 36, 36, 36, 36, 36, 36, 36]) ready: 1
round 346 dict_values([23, 36, 36, 36, 36, 36, 36, 36, 36, 36]) ready: 1
round 347 dict_values([24, 36, 36, 36, 36, 36, 36, 36, 36, 36]) ready: 1
round 348 dict_values([25, 36, 36, 36, 36, 36, 36, 36, 36, 36]) ready: 1
round 349 dict_values([26, 36, 36, 36, 36, 36, 36, 36, 36, 36]) ready: 1
round 350 dict_values([27, 36, 36, 36, 36, 36, 36, 36, 36, 36]) ready: 1
round 351 dict_values([28, 36, 36, 36, 36, 36, 36, 36, 36, 36]) ready: 1
round 352 dict_values([29, 36, 36, 36, 36, 36, 36, 36, 36, 36]) ready: 1
round 353 dict_values([30, 36, 36, 36, 36, 36, 36, 36, 36, 36]) ready: 1
...

wait_test.py

import ray
import time
from ray.test.cluster_utils import Cluster
from collections import Counter
import simplejson as json

cluster = Cluster(
    initialize_head=True,
    head_node_args={
            "_internal_config": json.dumps({
                "initial_reconstruction_timeout_milliseconds": 200
            })
    })
num_nodes = 10
for i in range(num_nodes - 1):
    node = cluster.add_node(
            _internal_config=json.dumps({
                "initial_reconstruction_timeout_milliseconds": 200
            })
            )
ray.init(redis_address=cluster.redis_address)


@ray.remote
class Actor(object):
    def __init__(self):
        return
    def ping(self):
        time.sleep(1)
    def node(self):
        return ray.worker.global_worker.plasma_client.store_socket_name


actors = [Actor.remote() for _ in range(num_nodes * 8)]
nodes = ray.get([actor.node.remote() for actor in actors])
node_count = Counter()
pings = [actor.ping.remote() for actor in actors]
r = 0
while True:
    indices = {
            ping: i for i, ping in enumerate(pings)
            }
    ready, unready = ray.wait(pings)
    for obj in ready:
        i = indices[obj]
        pings[i] = actors[i].ping.remote()
        node_count[nodes[i]] += 1
    print("round", r, node_count.values(), "ready:", len(ready))

    if r == 50:
        cluster.remove_node(node)
    r += 1

The script starts 10 nodes and several actors on each node. It then enters a loop which calls ray.wait on the current tasks for each actor and submits new tasks to any actors that are ready. After a few rounds, the script kills one of the nodes.

The attached logs show the number of actor tasks that have become ready so far, per node. During normal execution, the number of ready tasks should be about the same across all of the nodes and should increase steadily as tasks complete.

After the node has been killed, the number of ready tasks on the killed node remains the same for many rounds (line). This is because any actors on that node are still considered to be alive, but their tasks' return values will never appear. Once the node has been marked as dead, the backend marks the actors' return values as failed, and the number of ready tasks on the killed node begins to increase again (line).

However, if the initial_reconstruction_timeout_milliseconds set in this line is too high, then the tasks on the killed node will never become ready.