Skip to content

Instantly share code, notes, and snippets.

@jjerphan

jjerphan/server

Last active May 27, 2019 12:31
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save jjerphan/018908beb8422d7c81fb00198f5fd9f7 to your computer and use it in GitHub Desktop.
Save jjerphan/018908beb8422d7c81fb00198f5fd9f7 to your computer and use it in GitHub Desktop.
Download ZIP
jjerphan/joblib_dask_deadlock: hanging on `Client._update_scheduler_info` — commit: 3f3fb9b708b5ec74607c3edb5fb32f7fac9eeea1
Raw
server
Showing logs of joblib-dask-deadlock-test-server-5796bf7779-4dkw2
[2019-05-27 12:17:25,463] [8/MainThread] [INFO] [root] Launching a server
[2019-05-27 12:17:25,789] [9/MainThread] [INFO] [root] Joblib 0.13.2 imported
[2019-05-27 12:17:25,939] [9/MainThread] [DEBUG] [asyncio] Using selector: EpollSelector
[2019-05-27 12:17:26,152] [9/MainThread] [INFO] [root] Distributed 1.28.0+11.g715c05c imported
[2019-05-27 12:17:26,152] [9/MainThread] [INFO] [root] Git revision: 715c05c2449c55345821f9aecb52d62f6d0d4a3b
[2019-05-27 12:17:26,153] [9/MainThread] [INFO] [joblib.dask] _dask import (logger)
[2019-05-27 12:17:26,153] [9/MainThread] [INFO] [root] _dask import (logging)
/usr/local/lib/python3.6/site-packages/bokeh/themes/theme.py:94: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
json = yaml.load(f)
[2019-05-27 12:17:27,091] [9/MainThread] [INFO] [root] setup_log() called
[2019-05-27 12:17:27,091] [9/MainThread] [INFO] [root] Installing debugging signal handler
[2019-05-27 12:17:27,091] [9/MainThread] [INFO] [root] Started reproducible example
[2019-05-27 12:17:27,091] [9/MainThread] [INFO] [root] Dask Scheduler: starting on port 8786
[2019-05-27 12:17:27,093] [9/MainThread] [INFO] [root] Dask Scheduler: started on port 8786
[2019-05-27 12:17:27,093] [9/MainThread] [INFO] [root] Creating Dask Client
[2019-05-27 12:17:27,272] [9/Thread-1] [INFO] [root] Scheduler.__init__ called from start_dask_scheduler
[2019-05-27 12:17:27,273] [9/Thread-1] [INFO] [root] Scheduler._setup_logging called from __init__
[2019-05-27 12:17:27,275] [9/MainThread] [INFO] [root] Client.__init__ called from <module>
[2019-05-27 12:17:27,276] [9/Thread-1] [DEBUG] [asyncio] Using selector: EpollSelector
[2019-05-27 12:17:27,277] [9/MainThread] [DEBUG] [asyncio] Using selector: EpollSelector
[2019-05-27 12:17:27,280] [9/MainThread] [INFO] [root] Client.start called from __init__
[2019-05-27 12:17:27,281] [9/IO loop] [DEBUG] [asyncio] Using selector: EpollSelector
[2019-05-27 12:17:27,287] [9/IO loop] [INFO] [root] Client._start called from wrapper
[2019-05-27 12:17:27,294] [9/Thread-1] [INFO] [root] Scheduler.start called from start_dask_scheduler
[2019-05-27 12:17:27,295] [9/IO loop] [INFO] [root] Client._ensure_connected called from wrapper
[2019-05-27 12:17:27,296] [9/Thread-1] [INFO] [root] Clear task state
[2019-05-27 12:17:27,327] [9/Thread-1] [INFO] [root] Scheduler at: tcp://10.12.3.55:8786
[2019-05-27 12:17:27,327] [9/Thread-1] [INFO] [root] bokeh at: :8787
[2019-05-27 12:17:27,330] [9/Thread-1] [INFO] [root] Scheduler.finished called from wrapper
[2019-05-27 12:17:27,333] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:17:27,342] [9/Thread-1] [INFO] [root] Scheduler.add_client called from wrapper
[2019-05-27 12:17:27,342] [9/Thread-1] [INFO] [root] Receive client connection: Client-643eee0c-8079-11e9-8009-0a580a0c0337
distributed.core - INFO - Starting established connection
[2019-05-27 12:17:27,346] [9/IO loop] [INFO] [root] Client._handle_report called from wrapper
[2019-05-27 12:17:27,347] [9/MainThread] [INFO] [root] Created Dask Client
[2019-05-27 12:17:27,347] [9/MainThread] [INFO] [root] Waiting for Workers to connect
[2019-05-27 12:17:29,349] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:17:31,348] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:17:32,359] [9/MainThread] [INFO] [joblib.parallel] register_parallel_backend called on ; name: dask ; factory: DaskDistributedBackend
[2019-05-27 12:17:32,359] [9/MainThread] [INFO] [joblib.parallel] Parallel.__init__ called
[2019-05-27 12:17:32,360] [9/MainThread] [INFO] [joblib.parallel] Parallel.__init__: getting back_end from given string:'dask'
[2019-05-27 12:17:32,360] [9/MainThread] [INFO] [joblib.dask] DaskDistributedBackend.__init__ called
[2019-05-27 12:17:32,360] [9/MainThread] [INFO] [root] Worker.get_client called
[2019-05-27 12:17:32,360] [9/MainThread] [INFO] [root] Worker.get_worker called
[2019-05-27 12:17:32,360] [9/MainThread] [INFO] [joblib.parallel] Parallel.__init__: back_end=DaskDistributedBackend
[2019-05-27 12:17:32,360] [9/MainThread] [INFO] [root] Entering Dask Context
[2019-05-27 12:17:32,360] [9/MainThread] [INFO] [joblib.parallel] parallel_backend.__init__ called
[2019-05-27 12:17:32,361] [9/MainThread] [INFO] [joblib.dask] DaskDistributedBackend.__init__ called
[2019-05-27 12:17:32,361] [9/MainThread] [INFO] [root] Worker.get_client called
[2019-05-27 12:17:32,361] [9/MainThread] [INFO] [root] Worker.get_worker called
[2019-05-27 12:17:32,361] [9/MainThread] [INFO] [root] Entered Dask Context
[2019-05-27 12:17:32,361] [9/MainThread] [INFO] [root] Running 'fit_and_score_estimator' with 4 jobs and dask as a parallel back-end
[2019-05-27 12:17:32,361] [9/MainThread] [INFO] [joblib.parallel] Parallel.__call__ called
[2019-05-27 12:17:32,361] [9/MainThread] [INFO] [joblib.parallel] Parallel._initialize_backend called
[2019-05-27 12:17:32,361] [9/MainThread] [INFO] [joblib.dask] DaskDistributedBackend.configure called
[2019-05-27 12:17:32,361] [9/MainThread] [INFO] [joblib.dask] DaskDistributedBackend.effective_n_jobs called
[2019-05-27 12:17:32,368] [9/MainThread] [INFO] [root] Client.submit called from effective_n_jobs
[2019-05-27 12:17:32,369] [9/MainThread] [INFO] [root] Client._graph_to_futures called from submit
[2019-05-27 12:17:32,370] [9/MainThread] [INFO] [root] Client._send_to_scheduler called from _graph_to_futures
[2019-05-27 12:17:32,372] [9/IO loop] [INFO] [root] Client._send_to_scheduler_safe called from null_wrapper
[2019-05-27 12:17:32,375] [9/Thread-1] [INFO] [root] Scheduler.update_graph called from handle_stream
[2019-05-27 12:17:32,376] [9/Thread-1] [INFO] [root] Scheduler.transitions called from update_graph
[2019-05-27 12:17:33,347] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:17:35,349] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:17:36,776] [9/Thread-1] [INFO] [root] Scheduler.add_worker called from wrapper
[2019-05-27 12:17:36,777] [9/Thread-1] [INFO] [root] Scheduler.check_idle_saturated called from add_worker
[2019-05-27 12:17:36,781] [9/Thread-1] [INFO] [root] Scheduler.valid_workers called from add_worker
[2019-05-27 12:17:36,782] [9/Thread-1] [INFO] [root] Scheduler.transitions called from add_worker
[2019-05-27 12:17:36,783] [9/Thread-1] [INFO] [root] Scheduler.decide_worker called from transition_waiting_processing
[2019-05-27 12:17:36,784] [9/Thread-1] [INFO] [root] Scheduler.valid_workers called from decide_worker
[2019-05-27 12:17:36,785] [9/Thread-1] [INFO] [root] Scheduler.consumre_resources called from transition_waiting_processing
[2019-05-27 12:17:36,786] [9/Thread-1] [INFO] [root] Scheduler.check_idle_saturated called from transition_waiting_processing
[2019-05-27 12:17:36,787] [9/Thread-1] [INFO] [root] Scheduler.send_task_to_worker called from transition_waiting_processing
[2019-05-27 12:17:36,787] [9/Thread-1] [INFO] [root] Scheduler.worker_send called from send_task_to_worker
[2019-05-27 12:17:36,788] [9/Thread-1] [INFO] [root] Register tcp://10.12.2.243:45497
[2019-05-27 12:17:36,789] [9/Thread-1] [INFO] [root] Scheduler.handle_worker called from wrapper
[2019-05-27 12:17:36,790] [9/Thread-1] [INFO] [root] Starting worker compute stream, tcp://10.12.2.243:45497
distributed.core - INFO - Starting established connection
[2019-05-27 12:17:36,842] [9/Thread-1] [INFO] [root] Scheduler.check_idle_saturated called from _reevaluate_occupancy_worker
[2019-05-27 12:17:36,944] [9/Thread-1] [INFO] [root] Scheduler.check_idle_saturated called from _reevaluate_occupancy_worker
[2019-05-27 12:17:36,992] [9/Thread-1] [INFO] [root] Scheduler.handle_task_finished called from handle_stream
[2019-05-27 12:17:36,993] [9/Thread-1] [INFO] [root] Scheduler._remove_from_processing called from transition_processing_memory
[2019-05-27 12:17:36,994] [9/Thread-1] [INFO] [root] Scheduler.check_idle_saturated called from _remove_from_processing
[2019-05-27 12:17:36,995] [9/Thread-1] [INFO] [root] Scheduler.release_resources called from _remove_from_processing
[2019-05-27 12:17:36,996] [9/Thread-1] [INFO] [root] Scheduler._add_to_memory called from transition_processing_memory
[2019-05-27 12:17:36,997] [9/Thread-1] [INFO] [root] Scheduler.report called from _add_to_memory
[2019-05-27 12:17:36,998] [9/Thread-1] [INFO] [root] Scheduler.transitions called from handle_task_finished
[2019-05-27 12:17:37,003] [9/IO loop] [INFO] [root] Client._gather called from wrapper
[2019-05-27 12:17:37,003] [9/IO loop] [INFO] [root] Worker.get_worker called
[2019-05-27 12:17:37,006] [9/Thread-1] [INFO] [root] Scheduler.gather called from wrapper
[2019-05-27 12:17:37,006] [9/Thread-1] [INFO] [root] Worker.get_data_from_worker called
[2019-05-27 12:17:37,017] [9/IO loop] [INFO] [root] Client._send_to_scheduler called from _release_key
[2019-05-27 12:17:37,018] [9/IO loop] [INFO] [root] Client._send_to_scheduler_safe called from null_wrapper
[2019-05-27 12:17:37,024] [9/Thread-1] [INFO] [root] Scheduler.transitions called from client_releases_keys
[2019-05-27 12:17:37,025] [9/Thread-1] [INFO] [root] Scheduler.worker_send called from _propagate_forgotten
[2019-05-27 12:17:37,025] [9/Thread-1] [INFO] [root] Scheduler.report_on_key called from transition_memory_forgotten
[2019-05-27 12:17:37,026] [9/Thread-1] [INFO] [root] Scheduler.report called from report_on_key
[2019-05-27 12:17:37,029] [9/MainThread] [INFO] [joblib.parallel] Parallel._print called
[2019-05-27 12:17:37,030] [9/MainThread] [INFO] [joblib.parallel] [Parallel(n_jobs=4)]: Using backend DaskDistributedBackend with 8 concurrent workers.
[2019-05-27 12:17:37,030] [9/MainThread] [INFO] [joblib.dask] DaskDistributedBackend.start_call called
[2019-05-27 12:17:37,030] [9/MainThread] [INFO] [joblib.parallel] Parallel.__call__: iterator created
[2019-05-27 12:17:37,030] [9/MainThread] [INFO] [joblib.parallel] <generator object <genexpr> at 0x7f1ba0a8f990>
[2019-05-27 12:17:37,031] [9/MainThread] [INFO] [joblib.parallel] Parallel.dispatch_one_batch called from __call__
[2019-05-27 12:17:37,031] [9/MainThread] [INFO] [joblib.parallel] Parallel.dispatch_one_batch: took the lock
[2019-05-27 12:17:37,031] [9/MainThread] [INFO] [joblib.dask] DaskDistributedBackend.get_nested_backend called
[2019-05-27 12:17:37,031] [9/MainThread] [INFO] [joblib.dask] DaskDistributedBackend.__init__ called
[2019-05-27 12:17:37,031] [9/MainThread] [INFO] [joblib.parallel] BatchedCalls.__init__ called
[2019-05-27 12:17:37,033] [9/MainThread] [INFO] [joblib.parallel] BatchedCalls.__init__ : wraps 1 objects
[2019-05-27 12:17:37,034] [9/MainThread] [INFO] [joblib.parallel] Parallel.dispatch: calling self._dispatch
[2019-05-27 12:17:37,034] [9/MainThread] [INFO] [joblib.parallel] Parallel._dispatch called from dispatch_one_batch
[2019-05-27 12:17:37,034] [9/MainThread] [INFO] [joblib.parallel] BatchCompletionCallBack.__init__ called
[2019-05-27 12:17:37,034] [9/MainThread] [INFO] [joblib.parallel] Parallel._dispatch: taking the lock
[2019-05-27 12:17:37,034] [9/MainThread] [INFO] [joblib.parallel] Parallel._dispatch: took the lock
[2019-05-27 12:17:37,034] [9/MainThread] [INFO] [joblib.parallel] Parallel._dispatch: submitting job of id 0 to back-end of size 1
[2019-05-27 12:17:37,035] [9/MainThread] [INFO] [joblib.dask] DaskDistributedBackend.apply_async called
[2019-05-27 12:17:37,457] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:17:37,834] [9/Thread-1] [INFO] [root] Scheduler.add_worker called from wrapper
[2019-05-27 12:17:37,952] [9/Thread-1] [INFO] [root] Scheduler.check_idle_saturated called from add_worker
[2019-05-27 12:17:37,963] [9/Thread-1] [INFO] [root] Register tcp://10.12.3.56:42245
[2019-05-27 12:17:38,162] [9/Thread-1] [INFO] [root] Scheduler.handle_worker called from wrapper
[2019-05-27 12:17:38,178] [9/Thread-1] [INFO] [root] Starting worker compute stream, tcp://10.12.3.56:42245
distributed.core - INFO - Starting established connection
[2019-05-27 12:17:39,395] [9/Thread-1] [INFO] [root] Scheduler.add_worker called from wrapper
[2019-05-27 12:17:39,409] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:17:39,486] [9/Thread-1] [INFO] [root] Scheduler.check_idle_saturated called from add_worker
[2019-05-27 12:17:39,496] [9/Thread-1] [INFO] [root] Register tcp://10.12.3.57:35999
[2019-05-27 12:17:39,735] [9/Thread-1] [INFO] [root] Scheduler.handle_worker called from wrapper
[2019-05-27 12:17:39,741] [9/Thread-1] [INFO] [root] Starting worker compute stream, tcp://10.12.3.57:35999
distributed.core - INFO - Starting established connection
[2019-05-27 12:17:41,453] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:17:43,518] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:17:45,433] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:17:47,393] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:17:49,393] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:17:51,405] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:17:53,388] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:17:55,433] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:17:57,443] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:17:59,410] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:01,454] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:03,411] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:05,435] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:07,419] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:09,435] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:11,416] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:13,424] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:15,393] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:17,407] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:19,371] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:21,414] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:23,391] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:25,404] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:27,448] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:29,404] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:31,445] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:33,457] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:35,414] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:37,411] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:39,412] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:41,397] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:44,259] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:45,421] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:47,438] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:49,425] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:51,434] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:53,413] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:55,419] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:57,414] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:18:59,500] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:19:01,657] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:19:03,435] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:19:05,382] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:19:07,427] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:19:09,379] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:19:11,385] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:19:13,414] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:19:15,465] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
[2019-05-27 12:19:17,376] [9/IO loop] [INFO] [root] Client._update_scheduler_info called from wrapper
Raw
worker-1
Showing logs of joblib-dask-deadlock-test-worker-774b9f6565-dsw7g
[2019-05-27 12:17:26,950] [8/MainThread] [INFO] [root] Launching a worker
[2019-05-27 12:17:27,521] [9/MainThread] [INFO] [root] Distributed 1.28.0+11.g715c05c imported
[2019-05-27 12:17:27,521] [9/MainThread] [INFO] [root] Git revision: 715c05c2449c55345821f9aecb52d62f6d0d4a3b
[2019-05-27 12:17:27,522] [9/MainThread] [INFO] [root] setup_log() called
[2019-05-27 12:17:27,522] [9/MainThread] [INFO] [root] Installing debugging signal handler
[2019-05-27 12:17:27,523] [9/MainThread] [INFO] [root] Worker.__init__ called
[2019-05-27 12:17:27,523] [9/MainThread] [INFO] [root] Worker._setup_logging called
[2019-05-27 12:17:27,532] [9/MainThread] [INFO] [root] Launching Dask Worker via Python API
[2019-05-27 12:17:27,532] [9/MainThread] [INFO] [root] Python version:
[2019-05-27 12:17:27,533] [9/MainThread] [INFO] [root] 3.6.8 (default, Apr 25 2019, 21:02:35)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]
[2019-05-27 12:17:27,533] [9/MainThread] [INFO] [root] Dask version: 1.2.2
[2019-05-27 12:17:27,533] [9/MainThread] [INFO] [root] Distributed version: 1.28.0+11.g715c05c
[2019-05-27 12:17:27,533] [9/MainThread] [INFO] [root] Worker.start called
[2019-05-27 12:17:27,533] [9/MainThread] [INFO] [root] Launching Worker Watchdog Thread
[2019-05-27 12:17:27,534] [9/MainThread] [INFO] [root] Worker._start called
[2019-05-27 12:17:27,538] [9/MainThread] [INFO] [root] Start worker at: tcp://10.12.3.56:42245
[2019-05-27 12:17:27,539] [9/MainThread] [INFO] [root] Listening to: tcp://joblib-dask-deadlock-test-server:42245
[2019-05-27 12:17:27,539] [9/MainThread] [INFO] [root] Waiting to connect to: tcp://joblib-dask-deadlock-test-server:8786
[2019-05-27 12:17:27,539] [9/MainThread] [INFO] [root] -------------------------------------------------
[2019-05-27 12:17:27,539] [9/MainThread] [INFO] [root] Threads: 8
[2019-05-27 12:17:27,539] [9/MainThread] [INFO] [root] Memory: 31.62 GB
[2019-05-27 12:17:27,539] [9/MainThread] [INFO] [root] Local Directory: /home/test_user/dask-worker-space/worker-6sk4e7fj
[2019-05-27 12:17:27,540] [9/MainThread] [INFO] [root] Worker._register_with_scheduler called
[2019-05-27 12:17:27,540] [9/MainThread] [INFO] [root] -------------------------------------------------
[2019-05-27 12:17:32,538] [9/Thread-1] [INFO] [root] Dask Worker status : None
[2019-05-27 12:17:37,543] [9/Thread-1] [INFO] [root] Dask Worker status : None
[2019-05-27 12:17:37,545] [9/MainThread] [INFO] [root] Waiting to connect to: tcp://joblib-dask-deadlock-test-server:8786
[2019-05-27 12:17:38,103] [9/MainThread] [INFO] [root] Registered to: tcp://joblib-dask-deadlock-test-server:8786
[2019-05-27 12:17:38,103] [9/MainThread] [INFO] [root] -------------------------------------------------
[2019-05-27 12:17:38,104] [9/MainThread] [INFO] [root] Worker.handle_scheduler called
distributed.core - INFO - Starting established connection
[2019-05-27 12:17:42,549] [9/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:17:47,554] [9/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:17:52,560] [9/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:17:57,565] [9/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:02,570] [9/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:07,575] [9/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:12,580] [9/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:17,586] [9/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:22,591] [9/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:27,595] [9/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:32,600] [9/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:37,602] [9/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:42,607] [9/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:47,612] [9/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:52,617] [9/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:57,621] [9/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:19:02,627] [9/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:19:07,632] [9/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:19:12,637] [9/Thread-1] [INFO] [root] Dask Worker status : running
Raw
worker-2
Showing logs of joblib-dask-deadlock-test-worker-774b9f6565-tpshr
[2019-05-27 12:17:26,023] [9/MainThread] [INFO] [root] Launching a worker
[2019-05-27 12:17:26,633] [10/MainThread] [INFO] [root] Distributed 1.28.0+11.g715c05c imported
[2019-05-27 12:17:26,633] [10/MainThread] [INFO] [root] Git revision: 715c05c2449c55345821f9aecb52d62f6d0d4a3b
[2019-05-27 12:17:26,634] [10/MainThread] [INFO] [root] setup_log() called
[2019-05-27 12:17:26,635] [10/MainThread] [INFO] [root] Installing debugging signal handler
[2019-05-27 12:17:26,635] [10/MainThread] [INFO] [root] Worker.__init__ called
[2019-05-27 12:17:26,635] [10/MainThread] [INFO] [root] Worker._setup_logging called
[2019-05-27 12:17:26,647] [10/MainThread] [INFO] [root] Launching Dask Worker via Python API
[2019-05-27 12:17:26,647] [10/MainThread] [INFO] [root] Python version:
[2019-05-27 12:17:26,648] [10/MainThread] [INFO] [root] 3.6.8 (default, Apr 25 2019, 21:02:35)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]
[2019-05-27 12:17:26,648] [10/MainThread] [INFO] [root] Dask version: 1.2.2
[2019-05-27 12:17:26,648] [10/MainThread] [INFO] [root] Distributed version: 1.28.0+11.g715c05c
[2019-05-27 12:17:26,648] [10/MainThread] [INFO] [root] Worker.start called
[2019-05-27 12:17:26,648] [10/MainThread] [INFO] [root] Launching Worker Watchdog Thread
[2019-05-27 12:17:26,650] [10/MainThread] [INFO] [root] Worker._start called
[2019-05-27 12:17:26,653] [10/MainThread] [INFO] [root] Start worker at: tcp://10.12.2.243:45497
[2019-05-27 12:17:26,654] [10/MainThread] [INFO] [root] Listening to: tcp://joblib-dask-deadlock-test-server:45497
[2019-05-27 12:17:26,654] [10/MainThread] [INFO] [root] Waiting to connect to: tcp://joblib-dask-deadlock-test-server:8786
[2019-05-27 12:17:26,654] [10/MainThread] [INFO] [root] -------------------------------------------------
[2019-05-27 12:17:26,654] [10/MainThread] [INFO] [root] Threads: 8
[2019-05-27 12:17:26,654] [10/MainThread] [INFO] [root] Memory: 31.62 GB
[2019-05-27 12:17:26,654] [10/MainThread] [INFO] [root] Local Directory: /home/test_user/dask-worker-space/worker-8fyr_piu
[2019-05-27 12:17:26,654] [10/MainThread] [INFO] [root] Worker._register_with_scheduler called
[2019-05-27 12:17:26,654] [10/MainThread] [INFO] [root] -------------------------------------------------
[2019-05-27 12:17:31,654] [10/Thread-1] [INFO] [root] Dask Worker status : None
[2019-05-27 12:17:36,660] [10/Thread-1] [INFO] [root] Dask Worker status : None
[2019-05-27 12:17:36,666] [10/MainThread] [INFO] [root] Waiting to connect to: tcp://joblib-dask-deadlock-test-server:8786
[2019-05-27 12:17:36,790] [10/MainThread] [INFO] [root] Registered to: tcp://joblib-dask-deadlock-test-server:8786
[2019-05-27 12:17:36,790] [10/MainThread] [INFO] [root] -------------------------------------------------
[2019-05-27 12:17:36,790] [10/MainThread] [INFO] [root] Worker.handle_scheduler called
distributed.core - INFO - Starting established connection
[2019-05-27 12:17:36,791] [10/MainThread] [INFO] [root] Worker.add_task called
[2019-05-27 12:17:36,791] [10/MainThread] [INFO] [root] Worker._deserialize called
[2019-05-27 12:17:36,979] [10/MainThread] [INFO] [root] Joblib 0.13.2 imported
[2019-05-27 12:17:36,986] [10/MainThread] [INFO] [joblib.dask] _dask import (logger)
[2019-05-27 12:17:36,986] [10/MainThread] [INFO] [root] _dask import (logging)
[2019-05-27 12:17:36,986] [10/MainThread] [INFO] [root] Worker.ensure_computing called
[2019-05-27 12:17:36,987] [10/MainThread] [INFO] [root] Worker.execute called
[2019-05-27 12:17:36,987] [10/MainThread] [INFO] [root] Worker.executor_submit called
[2019-05-27 12:17:36,989] [10/MainThread] [INFO] [root] Worker.ensure_computing called
[2019-05-27 12:17:37,010] [10/MainThread] [INFO] [root] Worker.get_data called
[2019-05-27 12:17:37,029] [10/MainThread] [INFO] [root] Worker.delete_data called
[2019-05-27 12:17:37,030] [10/MainThread] [INFO] [root] Worker.ensure_computing called
[2019-05-27 12:17:41,665] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:17:46,670] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:17:51,672] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:17:56,677] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:01,683] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:06,688] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:11,693] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:16,698] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:21,704] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:26,709] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:31,715] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:36,716] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:41,722] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:46,728] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:51,733] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:56,739] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:19:01,744] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:19:06,750] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:19:11,755] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:19:16,761] [10/Thread-1] [INFO] [root] Dask Worker status : running
Raw
worker-3
Showing logs of joblib-dask-deadlock-test-worker-774b9f6565-wzz2b
[2019-05-27 12:17:28,540] [9/MainThread] [INFO] [root] Launching a worker
[2019-05-27 12:17:29,104] [10/MainThread] [INFO] [root] Distributed 1.28.0+11.g715c05c imported
[2019-05-27 12:17:29,104] [10/MainThread] [INFO] [root] Git revision: 715c05c2449c55345821f9aecb52d62f6d0d4a3b
[2019-05-27 12:17:29,105] [10/MainThread] [INFO] [root] setup_log() called
[2019-05-27 12:17:29,105] [10/MainThread] [INFO] [root] Installing debugging signal handler
[2019-05-27 12:17:29,105] [10/MainThread] [INFO] [root] Worker.__init__ called
[2019-05-27 12:17:29,106] [10/MainThread] [INFO] [root] Worker._setup_logging called
[2019-05-27 12:17:29,115] [10/MainThread] [INFO] [root] Launching Dask Worker via Python API
[2019-05-27 12:17:29,116] [10/MainThread] [INFO] [root] Python version:
[2019-05-27 12:17:29,116] [10/MainThread] [INFO] [root] 3.6.8 (default, Apr 25 2019, 21:02:35)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]
[2019-05-27 12:17:29,116] [10/MainThread] [INFO] [root] Dask version: 1.2.2
[2019-05-27 12:17:29,116] [10/MainThread] [INFO] [root] Distributed version: 1.28.0+11.g715c05c
[2019-05-27 12:17:29,116] [10/MainThread] [INFO] [root] Worker.start called
[2019-05-27 12:17:29,116] [10/MainThread] [INFO] [root] Launching Worker Watchdog Thread
[2019-05-27 12:17:29,117] [10/MainThread] [INFO] [root] Worker._start called
[2019-05-27 12:17:29,122] [10/MainThread] [INFO] [root] Start worker at: tcp://10.12.3.57:35999
[2019-05-27 12:17:29,122] [10/MainThread] [INFO] [root] Listening to: tcp://joblib-dask-deadlock-test-server:35999
[2019-05-27 12:17:29,122] [10/MainThread] [INFO] [root] Waiting to connect to: tcp://joblib-dask-deadlock-test-server:8786
[2019-05-27 12:17:29,122] [10/MainThread] [INFO] [root] -------------------------------------------------
[2019-05-27 12:17:29,122] [10/MainThread] [INFO] [root] Threads: 8
[2019-05-27 12:17:29,123] [10/MainThread] [INFO] [root] Memory: 31.62 GB
[2019-05-27 12:17:29,123] [10/MainThread] [INFO] [root] Local Directory: /home/test_user/dask-worker-space/worker-pjg6xaq9
[2019-05-27 12:17:29,123] [10/MainThread] [INFO] [root] Worker._register_with_scheduler called
[2019-05-27 12:17:29,123] [10/MainThread] [INFO] [root] -------------------------------------------------
[2019-05-27 12:17:34,122] [10/Thread-1] [INFO] [root] Dask Worker status : None
[2019-05-27 12:17:39,127] [10/Thread-1] [INFO] [root] Dask Worker status : None
[2019-05-27 12:17:39,133] [10/MainThread] [INFO] [root] Waiting to connect to: tcp://joblib-dask-deadlock-test-server:8786
[2019-05-27 12:17:39,634] [10/MainThread] [INFO] [root] Registered to: tcp://joblib-dask-deadlock-test-server:8786
[2019-05-27 12:17:39,635] [10/MainThread] [INFO] [root] -------------------------------------------------
[2019-05-27 12:17:39,635] [10/MainThread] [INFO] [root] Worker.handle_scheduler called
distributed.core - INFO - Starting established connection
[2019-05-27 12:17:44,132] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:17:49,137] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:17:54,143] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:17:59,148] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:04,154] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:09,159] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:14,164] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:19,167] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:24,172] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:29,177] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:34,183] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:39,187] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:44,192] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:49,197] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:54,202] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:18:59,207] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:19:04,212] [10/Thread-1] [INFO] [root] Dask Worker status : running
[2019-05-27 12:19:09,216] [10/Thread-1] [INFO] [root] Dask Worker status : running
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment