Skip to content

Instantly share code, notes, and snippets.

@bnaul

bnaul/worker.log

Created April 11, 2023 21:18
Show Gist options
  • Save bnaul/f40c47e42f4964d1adaaa9872e0fc539 to your computer and use it in GitHub Desktop.
Save bnaul/f40c47e42f4964d1adaaa9872e0fc539 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
worker.log
/usr/local/lib/python3.10/site-packages/distributed/cli/dask_worker.py:264: FutureWarning: dask-worker is deprecated and will be removed in a future release; use `dask worker` instead
warnings.warn(
2023-04-09 15:43:19,990 - distributed.nanny - INFO - Start Nanny at: 'tcp://10.76.9.27:43473'
2023-04-09 15:43:20,788 - distributed.worker - INFO - Start worker at: tcp://10.76.9.27:46429
2023-04-09 15:43:20,788 - distributed.worker - INFO - Listening to: tcp://10.76.9.27:46429
2023-04-09 15:43:20,788 - distributed.worker - INFO - Worker name: brett-1f0ecf6d-daskworkers-large-865bf95dbc-5vqnd
2023-04-09 15:43:20,788 - distributed.worker - INFO - dashboard at: 10.76.9.27:8787
2023-04-09 15:43:20,788 - distributed.worker - INFO - Waiting to connect to: tcp://brett-1f0ecf6d-daskscheduler:8786
2023-04-09 15:43:20,788 - distributed.worker - INFO - -------------------------------------------------
2023-04-09 15:43:20,788 - distributed.worker - INFO - Threads: 4
2023-04-09 15:43:20,788 - distributed.worker - INFO - Memory: 89.41 GiB
2023-04-09 15:43:20,788 - distributed.worker - INFO - Local Directory: /tmp/dask-worker-space/worker-3id3r54g
2023-04-09 15:43:20,788 - distributed.worker - INFO - -------------------------------------------------
2023-04-09 15:43:21,005 - distributed.worker - INFO - Registered to: tcp://brett-1f0ecf6d-daskscheduler:8786
2023-04-09 15:43:21,005 - distributed.worker - INFO - -------------------------------------------------
2023-04-09 15:43:21,006 - distributed.core - INFO - Starting established connection to tcp://brett-1f0ecf6d-daskscheduler:8786
[2023-04-09 15:43:26+0000] INFO - prefect.CloudTaskRunner | Task 'on_demand_auto_calibration-places/configs/south_central/2022_Q4/saturday.yaml': Starting task run...
[2023-04-09 15:43:38+0000] INFO - root | Loading on demand auto data...
[2023-04-09 15:43:47+0000] INFO - root | Loading trips
[2023-04-09 15:43:48+0000] INFO - root | Saving query result to temporary table temp_0330502b7ac44d6da5bddbab79b875c1
[2023-04-09 15:43:49+0000] INFO - root | Running query: https://console.cloud.google.com/bigquery?project=model-159019&j=bq:US:9a48b9bf-7fc7-4c3c-9928-b0493a6fce44&page=queryresults
[2023-04-09 15:44:18+0000] INFO - root | Query was billed for 5.989GB (~$0.03)
[2023-04-09 15:44:20+0000] INFO - root | Computing hourly on-demand coefficients...
2023-04-09 15:45:45,232 - distributed.core - INFO - Removing comms to tcp://10.76.9.47:44359
[2023-04-09 15:45:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.47:44359
2023-04-09 15:45:45,233 - distributed.core - INFO - Removing comms to tcp://10.76.9.45:39307
2023-04-09 15:45:45,233 - distributed.core - INFO - Removing comms to tcp://10.76.9.43:38263
[2023-04-09 15:45:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.45:39307
[2023-04-09 15:45:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.43:38263
[2023-04-09 15:45:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.44:38033
2023-04-09 15:45:45,233 - distributed.core - INFO - Removing comms to tcp://10.76.9.44:38033
2023-04-09 15:45:45,234 - distributed.core - INFO - Removing comms to tcp://10.76.9.46:43001
[2023-04-09 15:45:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.46:43001
[2023-04-09 15:46:08+0000] CRITICAL - root | Web-PDB: starting web-server on brett-1f0ecf6d-daskworkers-large-865bf95dbc-5vqnd:5555...
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.40:46305
2023-04-09 15:46:25,245 - distributed.core - INFO - Removing comms to tcp://10.76.9.40:46305
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.56:39929
2023-04-09 15:46:25,245 - distributed.core - INFO - Removing comms to tcp://10.76.9.56:39929
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.29:38839
2023-04-09 15:46:25,245 - distributed.core - INFO - Removing comms to tcp://10.76.9.29:38839
2023-04-09 15:46:25,245 - distributed.core - INFO - Removing comms to tcp://10.76.9.55:42775
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.55:42775
2023-04-09 15:46:25,245 - distributed.core - INFO - Removing comms to tcp://10.76.9.52:38589
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.52:38589
2023-04-09 15:46:25,246 - distributed.core - INFO - Removing comms to tcp://10.76.9.34:38143
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.34:38143
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.32:41163
2023-04-09 15:46:25,246 - distributed.core - INFO - Removing comms to tcp://10.76.9.32:41163
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.37:42627
2023-04-09 15:46:25,246 - distributed.core - INFO - Removing comms to tcp://10.76.9.37:42627
2023-04-09 15:46:25,246 - distributed.core - INFO - Removing comms to tcp://10.76.9.36:43351
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.36:43351
2023-04-09 15:46:25,246 - distributed.core - INFO - Removing comms to tcp://10.76.9.42:42791
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.42:42791
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.54:46409
2023-04-09 15:46:25,246 - distributed.core - INFO - Removing comms to tcp://10.76.9.54:46409
2023-04-09 15:46:25,246 - distributed.core - INFO - Removing comms to tcp://10.76.9.49:33475
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.49:33475
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.30:40417
2023-04-09 15:46:25,246 - distributed.core - INFO - Removing comms to tcp://10.76.9.30:40417
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.39:44649
2023-04-09 15:46:25,247 - distributed.core - INFO - Removing comms to tcp://10.76.9.39:44649
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.33:42703
2023-04-09 15:46:25,247 - distributed.core - INFO - Removing comms to tcp://10.76.9.33:42703
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.38:40847
2023-04-09 15:46:25,247 - distributed.core - INFO - Removing comms to tcp://10.76.9.38:40847
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.53:46049
2023-04-09 15:46:25,247 - distributed.core - INFO - Removing comms to tcp://10.76.9.53:46049
2023-04-09 15:46:25,247 - distributed.core - INFO - Removing comms to tcp://10.76.9.41:44511
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.41:44511
2023-04-09 15:46:25,247 - distributed.core - INFO - Removing comms to tcp://10.76.9.48:45847
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.48:45847
2023-04-09 15:46:25,247 - distributed.core - INFO - Removing comms to tcp://10.76.9.51:38981
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.51:38981
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.61:39323
2023-04-09 15:46:25,247 - distributed.core - INFO - Removing comms to tcp://10.76.117.61:39323
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.31:34457
2023-04-09 15:46:25,248 - distributed.core - INFO - Removing comms to tcp://10.76.9.31:34457
[2023-04-09 15:46:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.35:33371
2023-04-09 15:46:25,248 - distributed.core - INFO - Removing comms to tcp://10.76.9.35:33371
2023-04-09 15:47:35,232 - distributed.core - INFO - Removing comms to tcp://10.76.9.60:41331
[2023-04-09 15:47:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.60:41331
2023-04-09 15:47:35,232 - distributed.core - INFO - Removing comms to tcp://10.76.9.64:39303
[2023-04-09 15:47:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.64:39303
2023-04-09 15:47:35,232 - distributed.core - INFO - Removing comms to tcp://10.76.9.28:35323
[2023-04-09 15:47:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.28:35323
[2023-04-09 15:47:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.61:33561
2023-04-09 15:47:35,232 - distributed.core - INFO - Removing comms to tcp://10.76.117.61:33561
2023-04-09 15:47:35,232 - distributed.core - INFO - Removing comms to tcp://10.76.9.50:35227
[2023-04-09 15:47:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.50:35227
[2023-04-09 15:47:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.57:38951
2023-04-09 15:47:45,234 - distributed.core - INFO - Removing comms to tcp://10.76.9.57:38951
2023-04-09 15:47:45,234 - distributed.core - INFO - Removing comms to tcp://10.76.9.59:34401
[2023-04-09 15:47:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.59:34401
2023-04-09 15:47:45,235 - distributed.core - INFO - Removing comms to tcp://10.76.9.62:37533
[2023-04-09 15:47:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.62:37533
2023-04-09 15:47:45,235 - distributed.core - INFO - Removing comms to tcp://10.76.9.61:43181
[2023-04-09 15:47:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.61:43181
2023-04-09 15:47:45,235 - distributed.core - INFO - Removing comms to tcp://10.76.9.58:38979
[2023-04-09 15:47:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.58:38979
[2023-04-09 15:47:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.29:41347
2023-04-09 15:47:45,235 - distributed.core - INFO - Removing comms to tcp://10.76.9.29:41347
[2023-04-09 15:47:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.63:35491
2023-04-09 15:47:45,235 - distributed.core - INFO - Removing comms to tcp://10.76.9.63:35491
<stdin>:1: UserWarning: `meta` is not specified, inferred from partial data. Please provide `meta` if the result is unexpected.
Before: .apply(func)
After: .apply(func, meta={'x': 'f8', 'y': 'f8'}) for dataframe result
or: .apply(func, meta=('x', 'f8')) for series result
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.61:34413
2023-04-09 15:49:25,238 - distributed.core - INFO - Removing comms to tcp://10.76.117.61:34413
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.75:37509
2023-04-09 15:49:25,238 - distributed.core - INFO - Removing comms to tcp://10.76.9.75:37509
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.72:33713
2023-04-09 15:49:25,238 - distributed.core - INFO - Removing comms to tcp://10.76.9.72:33713
2023-04-09 15:49:25,238 - distributed.core - INFO - Removing comms to tcp://10.76.9.76:43395
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.76:43395
2023-04-09 15:49:25,239 - distributed.core - INFO - Removing comms to tcp://10.76.9.71:43447
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.71:43447
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.68:45359
2023-04-09 15:49:25,239 - distributed.core - INFO - Removing comms to tcp://10.76.9.68:45359
2023-04-09 15:49:25,239 - distributed.core - INFO - Removing comms to tcp://10.76.9.28:39177
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.28:39177
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.69:34013
2023-04-09 15:49:25,239 - distributed.core - INFO - Removing comms to tcp://10.76.9.69:34013
2023-04-09 15:49:25,239 - distributed.core - INFO - Removing comms to tcp://10.76.9.70:37279
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.70:37279
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.73:45365
2023-04-09 15:49:25,239 - distributed.core - INFO - Removing comms to tcp://10.76.9.73:45365
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.29:33075
2023-04-09 15:49:25,239 - distributed.core - INFO - Removing comms to tcp://10.76.9.29:33075
2023-04-09 15:49:25,239 - distributed.core - INFO - Removing comms to tcp://10.76.9.74:42141
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.74:42141
2023-04-09 15:49:25,239 - distributed.core - INFO - Removing comms to tcp://10.76.9.77:42095
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.77:42095
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.67:42745
2023-04-09 15:49:25,239 - distributed.core - INFO - Removing comms to tcp://10.76.9.67:42745
2023-04-09 15:49:25,239 - distributed.core - INFO - Removing comms to tcp://10.76.9.65:36627
[2023-04-09 15:49:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.65:36627
<stdin>:1: UserWarning: `meta` is not specified, inferred from partial data. Please provide `meta` if the result is unexpected.
Before: .apply(func)
After: .apply(func, meta={'x': 'f8', 'y': 'f8'}) for dataframe result
or: .apply(func, meta=('x', 'f8')) for series result
[2023-04-09 15:49:49+0000] INFO - distributed.utils_perf | full garbage collection released 22.26 MiB from 510 reference cycles (threshold: 9.54 MiB)
2023-04-09 15:49:49,530 - distributed.utils_perf - INFO - full garbage collection released 22.26 MiB from 510 reference cycles (threshold: 9.54 MiB)
[2023-04-09 15:50:09+0000] INFO - distributed.utils_perf | full garbage collection released 51.62 MiB from 570 reference cycles (threshold: 9.54 MiB)
2023-04-09 15:50:09,226 - distributed.utils_perf - INFO - full garbage collection released 51.62 MiB from 570 reference cycles (threshold: 9.54 MiB)
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.82:34499
2023-04-09 15:50:35,240 - distributed.core - INFO - Removing comms to tcp://10.76.9.82:34499
2023-04-09 15:50:35,240 - distributed.core - INFO - Removing comms to tcp://10.76.9.89:38245
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.89:38245
2023-04-09 15:50:35,241 - distributed.core - INFO - Removing comms to tcp://10.76.9.28:35487
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.28:35487
2023-04-09 15:50:35,241 - distributed.core - INFO - Removing comms to tcp://10.76.9.78:39053
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.78:39053
2023-04-09 15:50:35,241 - distributed.core - INFO - Removing comms to tcp://10.76.117.61:38073
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.61:38073
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.86:34919
2023-04-09 15:50:35,241 - distributed.core - INFO - Removing comms to tcp://10.76.9.86:34919
2023-04-09 15:50:35,241 - distributed.core - INFO - Removing comms to tcp://10.76.9.80:34689
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.80:34689
2023-04-09 15:50:35,241 - distributed.core - INFO - Removing comms to tcp://10.76.9.81:43023
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.81:43023
2023-04-09 15:50:35,241 - distributed.core - INFO - Removing comms to tcp://10.76.9.88:38229
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.88:38229
2023-04-09 15:50:35,241 - distributed.core - INFO - Removing comms to tcp://10.76.9.84:32885
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.84:32885
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.85:40007
2023-04-09 15:50:35,241 - distributed.core - INFO - Removing comms to tcp://10.76.9.85:40007
2023-04-09 15:50:35,241 - distributed.core - INFO - Removing comms to tcp://10.76.9.87:38171
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.87:38171
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.83:42677
2023-04-09 15:50:35,242 - distributed.core - INFO - Removing comms to tcp://10.76.9.83:42677
2023-04-09 15:50:35,242 - distributed.core - INFO - Removing comms to tcp://10.76.9.79:37789
[2023-04-09 15:50:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.79:37789
/usr/local/lib/python3.10/site-packages/dask/dataframe/core.py:6984: FutureWarning: Meta is not valid, `map_partitions` and `map_overlap` expects output to be a pandas object. Try passing a pandas object as meta or a dict or tuple representing the (name, dtype) of the columns. In the future the meta you passed will not work.
warnings.warn(
[2023-04-09 15:50:48+0000] INFO - distributed.utils_perf | full garbage collection released 21.95 MiB from 433 reference cycles (threshold: 9.54 MiB)
2023-04-09 15:50:48,801 - distributed.utils_perf - INFO - full garbage collection released 21.95 MiB from 433 reference cycles (threshold: 9.54 MiB)
2023-04-09 15:52:19,721 - distributed.comm.tcp - INFO - Connection from tcp://10.76.9.108:50670 closed before handshake completed
[2023-04-09 15:52:19+0000] INFO - distributed.comm.tcp | Connection from tcp://10.76.9.108:50670 closed before handshake completed
2023-04-09 15:52:35,506 - distributed.core - INFO - Removing comms to tcp://10.76.117.70:40427
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.70:40427
2023-04-09 15:52:35,508 - distributed.core - INFO - Removing comms to tcp://10.76.9.118:34731
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.118:34731
2023-04-09 15:52:35,541 - distributed.core - INFO - Removing comms to tcp://10.76.9.123:41057
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.123:41057
2023-04-09 15:52:35,544 - distributed.core - INFO - Removing comms to tcp://10.76.117.62:35147
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.62:35147
2023-04-09 15:52:35,612 - distributed.core - INFO - Removing comms to tcp://10.76.9.116:35619
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.116:35619
2023-04-09 15:52:35,656 - distributed.core - INFO - Removing comms to tcp://10.76.9.90:43623
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.90:43623
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.120:45829
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.66:43527
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.121:34003
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.106:33747
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.68:37137
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.124:34529
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.71:36633
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.104:35301
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.64:37585
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.109:41427
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.117:35185
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.103:38401
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.72:37413
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.110:44621
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.122:44313
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.67:33027
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.111:35995
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.91:45783
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.119:33227
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.28:34593
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.97:39951
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.65:46807
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.73:43233
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.95:40549
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.114:37299
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.98:46447
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.61:46773
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.63:36409
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.69:45159
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.102:37163
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.115:36053
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.107:42381
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.101:46881
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.112:40069
[2023-04-09 15:52:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.100:37729
2023-04-09 15:52:35,657 - distributed.core - INFO - Removing comms to tcp://10.76.9.120:45829
2023-04-09 15:52:35,657 - distributed.core - INFO - Removing comms to tcp://10.76.117.66:43527
2023-04-09 15:52:35,657 - distributed.core - INFO - Removing comms to tcp://10.76.9.121:34003
2023-04-09 15:52:35,657 - distributed.core - INFO - Removing comms to tcp://10.76.9.106:33747
2023-04-09 15:52:35,658 - distributed.core - INFO - Removing comms to tcp://10.76.117.68:37137
2023-04-09 15:52:35,658 - distributed.core - INFO - Removing comms to tcp://10.76.9.124:34529
2023-04-09 15:52:35,658 - distributed.core - INFO - Removing comms to tcp://10.76.117.71:36633
2023-04-09 15:52:35,658 - distributed.core - INFO - Removing comms to tcp://10.76.9.104:35301
2023-04-09 15:52:35,659 - distributed.core - INFO - Removing comms to tcp://10.76.117.64:37585
2023-04-09 15:52:35,659 - distributed.core - INFO - Removing comms to tcp://10.76.9.109:41427
2023-04-09 15:52:35,659 - distributed.core - INFO - Removing comms to tcp://10.76.9.117:35185
2023-04-09 15:52:35,659 - distributed.core - INFO - Removing comms to tcp://10.76.9.103:38401
2023-04-09 15:52:35,659 - distributed.core - INFO - Removing comms to tcp://10.76.117.72:37413
2023-04-09 15:52:35,659 - distributed.core - INFO - Removing comms to tcp://10.76.9.110:44621
2023-04-09 15:52:35,659 - distributed.core - INFO - Removing comms to tcp://10.76.9.122:44313
2023-04-09 15:52:35,659 - distributed.core - INFO - Removing comms to tcp://10.76.117.67:33027
2023-04-09 15:52:35,659 - distributed.core - INFO - Removing comms to tcp://10.76.9.111:35995
2023-04-09 15:52:35,660 - distributed.core - INFO - Removing comms to tcp://10.76.9.91:45783
2023-04-09 15:52:35,660 - distributed.core - INFO - Removing comms to tcp://10.76.9.119:33227
2023-04-09 15:52:35,660 - distributed.core - INFO - Removing comms to tcp://10.76.9.28:34593
2023-04-09 15:52:35,660 - distributed.core - INFO - Removing comms to tcp://10.76.9.97:39951
2023-04-09 15:52:35,660 - distributed.core - INFO - Removing comms to tcp://10.76.117.65:46807
2023-04-09 15:52:35,660 - distributed.core - INFO - Removing comms to tcp://10.76.117.73:43233
2023-04-09 15:52:35,660 - distributed.core - INFO - Removing comms to tcp://10.76.9.95:40549
2023-04-09 15:52:35,660 - distributed.core - INFO - Removing comms to tcp://10.76.9.114:37299
2023-04-09 15:52:35,660 - distributed.core - INFO - Removing comms to tcp://10.76.9.98:46447
2023-04-09 15:52:35,660 - distributed.core - INFO - Removing comms to tcp://10.76.117.61:46773
2023-04-09 15:52:35,661 - distributed.core - INFO - Removing comms to tcp://10.76.117.63:36409
2023-04-09 15:52:35,661 - distributed.core - INFO - Removing comms to tcp://10.76.117.69:45159
2023-04-09 15:52:35,661 - distributed.core - INFO - Removing comms to tcp://10.76.9.102:37163
2023-04-09 15:52:35,661 - distributed.core - INFO - Removing comms to tcp://10.76.9.115:36053
2023-04-09 15:52:35,661 - distributed.core - INFO - Removing comms to tcp://10.76.9.107:42381
2023-04-09 15:52:35,661 - distributed.core - INFO - Removing comms to tcp://10.76.9.101:46881
2023-04-09 15:52:35,661 - distributed.core - INFO - Removing comms to tcp://10.76.9.112:40069
2023-04-09 15:52:35,661 - distributed.core - INFO - Removing comms to tcp://10.76.9.100:37729
2023-04-09 15:53:25,241 - distributed.core - INFO - Removing comms to tcp://10.76.9.96:40683
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.96:40683
2023-04-09 15:53:25,247 - distributed.core - INFO - Removing comms to tcp://10.76.9.93:46657
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.93:46657
2023-04-09 15:53:25,248 - distributed.core - INFO - Removing comms to tcp://10.76.9.99:38421
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.99:38421
2023-04-09 15:53:25,249 - distributed.core - INFO - Removing comms to tcp://10.76.117.76:44435
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.76:44435
2023-04-09 15:53:25,249 - distributed.core - INFO - Removing comms to tcp://10.76.9.105:33501
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.105:33501
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.108:43421
2023-04-09 15:53:25,249 - distributed.core - INFO - Removing comms to tcp://10.76.9.108:43421
2023-04-09 15:53:25,250 - distributed.core - INFO - Removing comms to tcp://10.76.9.113:43369
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.113:43369
2023-04-09 15:53:25,250 - distributed.core - INFO - Removing comms to tcp://10.76.9.94:38567
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.94:38567
2023-04-09 15:53:25,250 - distributed.core - INFO - Removing comms to tcp://10.76.117.74:46113
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.74:46113
2023-04-09 15:53:25,250 - distributed.core - INFO - Removing comms to tcp://10.76.117.78:33169
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.78:33169
2023-04-09 15:53:25,250 - distributed.core - INFO - Removing comms to tcp://10.76.117.77:38099
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.77:38099
2023-04-09 15:53:25,250 - distributed.core - INFO - Removing comms to tcp://10.76.9.29:43851
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.29:43851
2023-04-09 15:53:25,251 - distributed.core - INFO - Removing comms to tcp://10.76.9.92:38357
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.92:38357
2023-04-09 15:53:25,251 - distributed.core - INFO - Removing comms to tcp://10.76.117.75:43379
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.75:43379
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.79:40527
2023-04-09 15:53:25,251 - distributed.core - INFO - Removing comms to tcp://10.76.117.79:40527
2023-04-09 15:53:25,251 - distributed.core - INFO - Removing comms to tcp://10.76.9.66:35095
[2023-04-09 15:53:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.66:35095
/usr/local/lib/python3.10/site-packages/dask/dataframe/core.py:6984: FutureWarning: Meta is not valid, `map_partitions` and `map_overlap` expects output to be a pandas object. Try passing a pandas object as meta or a dict or tuple representing the (name, dtype) of the columns. In the future the meta you passed will not work.
warnings.warn(
[2023-04-09 15:54:18+0000] INFO - google.cloud.bigquery_storage_v1.writer | RPC termination has signaled streaming pull manager shutdown.
[2023-04-09 15:54:18+0000] INFO - google.api_core.bidi | Thread-ConsumeBidirectionalStream exiting
2023-04-09 15:55:30,089 - distributed.worker - ERROR - failed during get data with tcp://10.76.9.27:46429 -> tcp://10.76.117.100:34073
[2023-04-09 15:55:30+0000] ERROR - distributed.worker | failed during get data with tcp://10.76.9.27:46429 -> tcp://10.76.117.100:34073
tornado.iostream.StreamClosedError: Stream is closed
frames_nbytes = await stream.read_bytes(fmt_size)
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 225, in read
Traceback (most recent call last):
null
The above exception was the direct cause of the following exception:
null
distributed.comm.core.CommClosedError: in <TCP (closed) local=tcp://10.76.9.27:46429 remote=tcp://10.76.117.100:44916>: Stream is closed
raise CommClosedError(f"in {obj}: {exc}") from exc
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 144, in convert_stream_closed_error
convert_stream_closed_error(self, e)
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 241, in read
response = await comm.read(deserializers=serializers)
File "/usr/local/lib/python3.10/site-packages/distributed/worker.py", line 1793, in get_data
Traceback (most recent call last):
2023-04-09 15:55:30,287 - distributed.core - INFO - Lost connection to 'tcp://10.76.117.100:44916'
tornado.iostream.StreamClosedError: Stream is closed
frames_nbytes = await stream.read_bytes(fmt_size)
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 225, in read
Traceback (most recent call last):
null
The above exception was the direct cause of the following exception:
null
distributed.comm.core.CommClosedError: in <TCP (closed) local=tcp://10.76.9.27:46429 remote=tcp://10.76.117.100:44916>: Stream is closed
raise CommClosedError(f"in {obj}: {exc}") from exc
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 144, in convert_stream_closed_error
convert_stream_closed_error(self, e)
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 241, in read
response = await comm.read(deserializers=serializers)
File "/usr/local/lib/python3.10/site-packages/distributed/worker.py", line 1793, in get_data
return await func(self, *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 987, in wrapper
result = await result
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm
Traceback (most recent call last):
tornado.iostream.StreamClosedError: Stream is closed
frames_nbytes = await stream.read_bytes(fmt_size)
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 225, in read
Traceback (most recent call last):
null
The above exception was the direct cause of the following exception:
null
distributed.comm.core.CommClosedError: in <TCP (closed) local=tcp://10.76.9.27:46429 remote=tcp://10.76.117.100:44916>: Stream is closed
raise CommClosedError(f"in {obj}: {exc}") from exc
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 144, in convert_stream_closed_error
convert_stream_closed_error(self, e)
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 241, in read
response = await comm.read(deserializers=serializers)
File "/usr/local/lib/python3.10/site-packages/distributed/worker.py", line 1793, in get_data
Traceback (most recent call last):
[2023-04-09 15:55:30+0000] INFO - distributed.core | Lost connection to 'tcp://10.76.117.100:44916'
tornado.iostream.StreamClosedError: Stream is closed
frames_nbytes = await stream.read_bytes(fmt_size)
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 225, in read
Traceback (most recent call last):
null
The above exception was the direct cause of the following exception:
null
distributed.comm.core.CommClosedError: in <TCP (closed) local=tcp://10.76.9.27:46429 remote=tcp://10.76.117.100:44916>: Stream is closed
raise CommClosedError(f"in {obj}: {exc}") from exc
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 144, in convert_stream_closed_error
convert_stream_closed_error(self, e)
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 241, in read
response = await comm.read(deserializers=serializers)
File "/usr/local/lib/python3.10/site-packages/distributed/worker.py", line 1793, in get_data
return await func(self, *args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 987, in wrapper
result = await result
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm
Traceback (most recent call last):
2023-04-09 15:55:30,717 - distributed.core - INFO - Removing comms to tcp://10.76.117.98:45247
[2023-04-09 15:55:30+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.98:45247
2023-04-09 15:55:30,722 - distributed.core - INFO - Removing comms to tcp://10.76.117.97:39537
[2023-04-09 15:55:30+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.97:39537
2023-04-09 15:55:30,722 - distributed.core - INFO - Removing comms to tcp://10.76.117.99:42495
2023-04-09 15:55:30,724 - distributed.core - INFO - Removing comms to tcp://10.76.117.100:34073
[2023-04-09 15:55:30+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.99:42495
[2023-04-09 15:55:30+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.100:34073
2023-04-09 15:55:33,310 - distributed.core - INFO - Removing comms to tcp://10.76.9.136:45665
[2023-04-09 15:55:33+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.136:45665
2023-04-09 15:56:00,992 - distributed.core - INFO - Removing comms to tcp://10.76.9.138:40117
[2023-04-09 15:56:00+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.138:40117
2023-04-09 15:56:01,196 - distributed.core - INFO - Removing comms to tcp://10.76.9.137:46167
[2023-04-09 15:56:01+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.137:46167
2023-04-09 15:56:07,179 - distributed.client - WARNING - Couldn't gather 34 keys, rescheduling {"('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 151)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 273)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 114)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 270)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 315)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 89)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 369)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 274)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 266)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 238)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 283)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 243)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 284)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 282)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 254)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 245)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 20)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 153)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 262)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 269)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 286)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 276)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 72)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 167)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 250)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 67)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 224)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 158)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 278)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 257)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 235)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 331)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 325)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 96)": ()}
[2023-04-09 15:56:07+0000] WARNING - distributed.client | Couldn't gather 34 keys, rescheduling {"('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 151)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 273)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 114)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 270)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 315)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 89)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 369)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 274)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 266)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 238)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 283)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 243)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 284)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 282)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 254)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 245)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 20)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 153)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 262)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 269)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 286)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 276)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 72)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 167)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 250)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 67)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 224)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 158)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 278)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 257)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 235)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 331)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 325)": (), "('optimize_alpha-c2d668f311c9df5721f07db523a77c59', 96)": ()}
2023-04-09 15:56:35,269 - distributed.core - INFO - Removing comms to tcp://10.76.9.169:42489
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.169:42489
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.170:42145
2023-04-09 15:56:35,269 - distributed.core - INFO - Removing comms to tcp://10.76.9.170:42145
2023-04-09 15:56:35,269 - distributed.core - INFO - Removing comms to tcp://10.76.9.154:42649
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.154:42649
2023-04-09 15:56:35,270 - distributed.core - INFO - Removing comms to tcp://10.76.117.89:45579
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.89:45579
2023-04-09 15:56:35,270 - distributed.core - INFO - Removing comms to tcp://10.76.9.156:38383
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.156:38383
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.80:44355
2023-04-09 15:56:35,270 - distributed.core - INFO - Removing comms to tcp://10.76.117.80:44355
2023-04-09 15:56:35,270 - distributed.core - INFO - Removing comms to tcp://10.76.9.165:40627
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.165:40627
2023-04-09 15:56:35,270 - distributed.core - INFO - Removing comms to tcp://10.76.9.166:42423
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.166:42423
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.87:41569
2023-04-09 15:56:35,270 - distributed.core - INFO - Removing comms to tcp://10.76.117.87:41569
2023-04-09 15:56:35,271 - distributed.core - INFO - Removing comms to tcp://10.76.65.77:35861
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.65.77:35861
2023-04-09 15:56:35,271 - distributed.core - INFO - Removing comms to tcp://10.76.9.168:37409
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.168:37409
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.157:32993
2023-04-09 15:56:35,271 - distributed.core - INFO - Removing comms to tcp://10.76.9.157:32993
2023-04-09 15:56:35,271 - distributed.core - INFO - Removing comms to tcp://10.76.9.171:46407
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.171:46407
2023-04-09 15:56:35,271 - distributed.core - INFO - Removing comms to tcp://10.76.117.86:38891
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.86:38891
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.83:43401
2023-04-09 15:56:35,271 - distributed.core - INFO - Removing comms to tcp://10.76.117.83:43401
2023-04-09 15:56:35,271 - distributed.core - INFO - Removing comms to tcp://10.76.117.82:43551
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.82:43551
2023-04-09 15:56:35,272 - distributed.core - INFO - Removing comms to tcp://10.76.9.167:38061
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.167:38061
2023-04-09 15:56:35,272 - distributed.core - INFO - Removing comms to tcp://10.76.9.153:45781
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.153:45781
2023-04-09 15:56:35,272 - distributed.core - INFO - Removing comms to tcp://10.76.117.94:40615
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.94:40615
[2023-04-09 15:56:35+0000] ERROR - distributed.shuffle | shuffle_barrier failed during shuffle ca23d72b3d695085e9da34db46f8233c
2023-04-09 15:56:35,279 - distributed.shuffle - ERROR - shuffle_barrier failed during shuffle ca23d72b3d695085e9da34db46f8233c
tornado.iostream.StreamClosedError: Stream is closed
frames_nbytes = await stream.read_bytes(fmt_size)
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 225, in read
Traceback (most recent call last):
tornado.iostream.StreamClosedError: Stream is closed
frames_nbytes = await stream.read_bytes(fmt_size)
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 225, in read
Traceback (most recent call last):
null
The above exception was the direct cause of the following exception:
null
distributed.comm.core.CommClosedError: in <TCP (closed) ConnectionPool.shuffle_barrier local=tcp://10.76.9.27:34274 remote=tcp://brett-1f0ecf6d-daskscheduler:8786>: Stream is closed
raise CommClosedError(f"in {obj}: {exc}") from exc
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 144, in convert_stream_closed_error
convert_stream_closed_error(self, e)
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 241, in read
response = await comm.read(deserializers=deserializers)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1024, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
await self.scheduler.shuffle_barrier(id=self.id, run_id=self.run_id)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 117, in barrier
await shuffle.barrier()
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 652, in _barrier
value = future.result()
File "/usr/local/lib/python3.10/site-packages/tornado/gen.py", line 762, in run
result = yield future
File "/usr/local/lib/python3.10/site-packages/distributed/utils.py", line 389, in f
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/utils.py", line 416, in sync
result = sync(self.worker.loop, self._barrier, shuffle_id, run_ids)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 865, in barrier
return _get_worker_extension().barrier(id, run_ids)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_shuffle.py", line 91, in shuffle_barrier
Traceback (most recent call last):
null
The above exception was the direct cause of the following exception:
null
distributed.comm.core.CommClosedError: in <TCP (closed) ConnectionPool.shuffle_barrier local=tcp://10.76.9.27:34274 remote=tcp://brett-1f0ecf6d-daskscheduler:8786>: Stream is closed
raise CommClosedError(f"in {obj}: {exc}") from exc
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 144, in convert_stream_closed_error
convert_stream_closed_error(self, e)
File "/usr/local/lib/python3.10/site-packages/distributed/comm/tcp.py", line 241, in read
response = await comm.read(deserializers=deserializers)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1024, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
await self.scheduler.shuffle_barrier(id=self.id, run_id=self.run_id)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 117, in barrier
await shuffle.barrier()
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 652, in _barrier
value = future.result()
File "/usr/local/lib/python3.10/site-packages/tornado/gen.py", line 762, in run
result = yield future
File "/usr/local/lib/python3.10/site-packages/distributed/utils.py", line 389, in f
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/utils.py", line 416, in sync
result = sync(self.worker.loop, self._barrier, shuffle_id, run_ids)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 865, in barrier
return _get_worker_extension().barrier(id, run_ids)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_shuffle.py", line 91, in shuffle_barrier
Traceback (most recent call last):
2023-04-09 15:56:35,283 - distributed.worker - WARNING - Compute Failed
[2023-04-09 15:56:35+0000] WARNING - distributed.worker | Compute Failed
Key: shuffle-barrier-ca23d72b3d695085e9da34db46f8233c
Function: shuffle_barrier
args: ('ca23d72b3d695085e9da34db46f8233c', [4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
Key: shuffle-barrier-ca23d72b3d695085e9da34db46f8233c
Function: shuffle_barrier
args: ('ca23d72b3d695085e9da34db46f8233c', [4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
kwargs: {}
Exception: "RuntimeError('shuffle_barrier failed during shuffle ca23d72b3d695085e9da34db46f8233c')"
kwargs: {}
Exception: "RuntimeError('shuffle_barrier failed during shuffle ca23d72b3d695085e9da34db46f8233c')"
2023-04-09 15:56:35,660 - distributed.core - INFO - Removing comms to tcp://10.76.9.127:38379
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.127:38379
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.61:36685
2023-04-09 15:56:35,712 - distributed.core - INFO - Removing comms to tcp://10.76.117.61:36685
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.84:36653
2023-04-09 15:56:35,719 - distributed.core - INFO - Removing comms to tcp://10.76.117.84:36653
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.159:36777
2023-04-09 15:56:35,720 - distributed.core - INFO - Removing comms to tcp://10.76.9.159:36777
2023-04-09 15:56:35,720 - distributed.core - INFO - Removing comms to tcp://10.76.9.144:38619
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.144:38619
2023-04-09 15:56:35,720 - distributed.core - INFO - Removing comms to tcp://10.76.9.148:36611
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.148:36611
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.102:46079
2023-04-09 15:56:35,720 - distributed.core - INFO - Removing comms to tcp://10.76.117.102:46079
2023-04-09 15:56:35,720 - distributed.core - INFO - Removing comms to tcp://10.76.9.158:43367
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.158:43367
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.145:44429
2023-04-09 15:56:35,721 - distributed.core - INFO - Removing comms to tcp://10.76.9.145:44429
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.126:35341
2023-04-09 15:56:35,721 - distributed.core - INFO - Removing comms to tcp://10.76.9.126:35341
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.101:39155
2023-04-09 15:56:35,721 - distributed.core - INFO - Removing comms to tcp://10.76.117.101:39155
2023-04-09 15:56:35,721 - distributed.core - INFO - Removing comms to tcp://10.76.9.134:35609
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.134:35609
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.65.79:40717
2023-04-09 15:56:35,721 - distributed.core - INFO - Removing comms to tcp://10.76.65.79:40717
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.152:33859
2023-04-09 15:56:35,722 - distributed.core - INFO - Removing comms to tcp://10.76.9.152:33859
2023-04-09 15:56:35,722 - distributed.core - INFO - Removing comms to tcp://10.76.9.155:34533
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.155:34533
2023-04-09 15:56:35,722 - distributed.core - INFO - Removing comms to tcp://10.76.117.85:46813
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.85:46813
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.81:44087
2023-04-09 15:56:35,722 - distributed.core - INFO - Removing comms to tcp://10.76.117.81:44087
2023-04-09 15:56:35,722 - distributed.core - INFO - Removing comms to tcp://10.76.9.142:46145
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.142:46145
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.141:34333
2023-04-09 15:56:35,722 - distributed.core - INFO - Removing comms to tcp://10.76.9.141:34333
2023-04-09 15:56:35,722 - distributed.core - INFO - Removing comms to tcp://10.76.9.139:46663
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.139:46663
2023-04-09 15:56:35,722 - distributed.core - INFO - Removing comms to tcp://10.76.9.162:36021
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.162:36021
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.140:35963
2023-04-09 15:56:35,722 - distributed.core - INFO - Removing comms to tcp://10.76.9.140:35963
2023-04-09 15:56:35,722 - distributed.core - INFO - Removing comms to tcp://10.76.9.160:36573
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.160:36573
2023-04-09 15:56:35,722 - distributed.core - INFO - Removing comms to tcp://10.76.9.150:42363
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.150:42363
2023-04-09 15:56:35,723 - distributed.core - INFO - Removing comms to tcp://10.76.9.146:36013
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.146:36013
2023-04-09 15:56:35,723 - distributed.core - INFO - Removing comms to tcp://10.76.9.164:42399
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.164:42399
2023-04-09 15:56:35,723 - distributed.core - INFO - Removing comms to tcp://10.76.9.149:38583
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.149:38583
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.143:43253
2023-04-09 15:56:35,723 - distributed.core - INFO - Removing comms to tcp://10.76.9.143:43253
2023-04-09 15:56:35,723 - distributed.core - INFO - Removing comms to tcp://10.76.9.151:38761
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.151:38761
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.163:34535
2023-04-09 15:56:35,723 - distributed.core - INFO - Removing comms to tcp://10.76.9.163:34535
2023-04-09 15:56:35,723 - distributed.core - INFO - Removing comms to tcp://10.76.9.161:40815
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.161:40815
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.29:45303
2023-04-09 15:56:35,723 - distributed.core - INFO - Removing comms to tcp://10.76.9.29:45303
2023-04-09 15:56:35,723 - distributed.core - INFO - Removing comms to tcp://10.76.9.135:39905
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.135:39905
2023-04-09 15:56:35,724 - distributed.core - INFO - Removing comms to tcp://10.76.9.132:36293
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.132:36293
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.129:38587
2023-04-09 15:56:35,724 - distributed.core - INFO - Removing comms to tcp://10.76.9.129:38587
[2023-04-09 15:56:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.130:39149
2023-04-09 15:56:35,724 - distributed.core - INFO - Removing comms to tcp://10.76.9.130:39149
2023-04-09 15:56:45,485 - distributed.core - INFO - Removing comms to tcp://10.76.117.88:36365
[2023-04-09 15:56:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.88:36365
[2023-04-09 15:56:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.65.80:42509
2023-04-09 15:56:45,486 - distributed.core - INFO - Removing comms to tcp://10.76.65.80:42509
2023-04-09 15:56:45,486 - distributed.core - INFO - Removing comms to tcp://10.76.9.28:38943
[2023-04-09 15:56:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.28:38943
2023-04-09 15:56:45,486 - distributed.core - INFO - Removing comms to tcp://10.76.117.91:38851
[2023-04-09 15:56:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.91:38851
2023-04-09 15:56:45,486 - distributed.core - INFO - Removing comms to tcp://10.76.9.133:46883
[2023-04-09 15:56:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.133:46883
[2023-04-09 15:56:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.93:44355
2023-04-09 15:56:45,486 - distributed.core - INFO - Removing comms to tcp://10.76.117.93:44355
[2023-04-09 15:56:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.96:37269
2023-04-09 15:56:45,486 - distributed.core - INFO - Removing comms to tcp://10.76.117.96:37269
[2023-04-09 15:56:45+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.131:44185
2023-04-09 15:56:45,487 - distributed.core - INFO - Removing comms to tcp://10.76.9.131:44185
2023-04-09 15:56:48,307 - distributed.shuffle._comms - ERROR - 'ca23d72b3d695085e9da34db46f8233c'
KeyError: 'ca23d72b3d695085e9da34db46f8233c'
state = self.states[id]
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get
result = handler(**msg)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
result = await self.worker.scheduler.shuffle_get(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle
shuffle = await self._refresh_shuffle(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run
shuffle = await self._get_shuffle_run(shuffle_id, run_id)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive
result = await result
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
return await self.rpc(address).shuffle_receive(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 123, in send
await self.send(address, shards)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_comms.py", line 71, in _process
Traceback (most recent call last):
[2023-04-09 15:56:48+0000] ERROR - distributed.shuffle._comms | 'ca23d72b3d695085e9da34db46f8233c'
KeyError: 'ca23d72b3d695085e9da34db46f8233c'
state = self.states[id]
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get
result = handler(**msg)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
result = await self.worker.scheduler.shuffle_get(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle
shuffle = await self._refresh_shuffle(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run
shuffle = await self._get_shuffle_run(shuffle_id, run_id)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive
result = await result
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
return await self.rpc(address).shuffle_receive(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 123, in send
await self.send(address, shards)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_comms.py", line 71, in _process
Traceback (most recent call last):
[2023-04-09 15:56:48+0000] ERROR - distributed.shuffle._comms | 'ca23d72b3d695085e9da34db46f8233c'
2023-04-09 15:56:48,320 - distributed.shuffle._comms - ERROR - 'ca23d72b3d695085e9da34db46f8233c'
KeyError: 'ca23d72b3d695085e9da34db46f8233c'
state = self.states[id]
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get
result = handler(**msg)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
result = await self.worker.scheduler.shuffle_get(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle
shuffle = await self._refresh_shuffle(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run
shuffle = await self._get_shuffle_run(shuffle_id, run_id)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive
result = await result
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
return await self.rpc(address).shuffle_receive(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 123, in send
await self.send(address, shards)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_comms.py", line 71, in _process
Traceback (most recent call last):
KeyError: 'ca23d72b3d695085e9da34db46f8233c'
state = self.states[id]
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get
result = handler(**msg)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
result = await self.worker.scheduler.shuffle_get(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle
shuffle = await self._refresh_shuffle(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run
shuffle = await self._get_shuffle_run(shuffle_id, run_id)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive
result = await result
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
return await self.rpc(address).shuffle_receive(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 123, in send
await self.send(address, shards)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_comms.py", line 71, in _process
Traceback (most recent call last):
[2023-04-09 15:56:48+0000] ERROR - distributed.shuffle._comms | 'ca23d72b3d695085e9da34db46f8233c'
KeyError: 'ca23d72b3d695085e9da34db46f8233c'
state = self.states[id]
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get
result = handler(**msg)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
result = await self.worker.scheduler.shuffle_get(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle
shuffle = await self._refresh_shuffle(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run
shuffle = await self._get_shuffle_run(shuffle_id, run_id)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive
result = await result
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
return await self.rpc(address).shuffle_receive(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 123, in send
await self.send(address, shards)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_comms.py", line 71, in _process
Traceback (most recent call last):
2023-04-09 15:56:48,331 - distributed.shuffle._comms - ERROR - 'ca23d72b3d695085e9da34db46f8233c'
KeyError: 'ca23d72b3d695085e9da34db46f8233c'
state = self.states[id]
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get
result = handler(**msg)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
result = await self.worker.scheduler.shuffle_get(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle
shuffle = await self._refresh_shuffle(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run
shuffle = await self._get_shuffle_run(shuffle_id, run_id)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive
result = await result
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
return await self.rpc(address).shuffle_receive(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 123, in send
await self.send(address, shards)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_comms.py", line 71, in _process
Traceback (most recent call last):
2023-04-09 15:56:48,342 - distributed.shuffle._comms - ERROR - 'ca23d72b3d695085e9da34db46f8233c'
KeyError: 'ca23d72b3d695085e9da34db46f8233c'
state = self.states[id]
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get
result = handler(**msg)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
result = await self.worker.scheduler.shuffle_get(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle
shuffle = await self._refresh_shuffle(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run
shuffle = await self._get_shuffle_run(shuffle_id, run_id)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive
result = await result
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
return await self.rpc(address).shuffle_receive(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 123, in send
await self.send(address, shards)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_comms.py", line 71, in _process
Traceback (most recent call last):
[2023-04-09 15:56:48+0000] ERROR - distributed.shuffle._comms | 'ca23d72b3d695085e9da34db46f8233c'
KeyError: 'ca23d72b3d695085e9da34db46f8233c'
state = self.states[id]
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get
result = handler(**msg)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
result = await self.worker.scheduler.shuffle_get(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle
shuffle = await self._refresh_shuffle(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run
shuffle = await self._get_shuffle_run(shuffle_id, run_id)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive
result = await result
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
return await self.rpc(address).shuffle_receive(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 123, in send
await self.send(address, shards)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_comms.py", line 71, in _process
Traceback (most recent call last):
2023-04-09 15:56:48,352 - distributed.shuffle._worker_extension - ERROR - 'ca23d72b3d695085e9da34db46f8233c'
KeyError: 'ca23d72b3d695085e9da34db46f8233c'
state = self.states[id]
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get
result = handler(**msg)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
result = await self.worker.scheduler.shuffle_get(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle
shuffle = await self._refresh_shuffle(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run
shuffle = await self._get_shuffle_run(shuffle_id, run_id)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive
result = await result
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
return await self.rpc(address).shuffle_receive(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 123, in send
await self.send(address, shards)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_comms.py", line 71, in _process
await self._process(id, shards)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_buffer.py", line 107, in process
raise self._exception
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_buffer.py", line 222, in raise_on_exception
self._comm_buffer.raise_on_exception()
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 173, in inputs_done
await shuffle.inputs_done()
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 597, in shuffle_inputs_done
Traceback (most recent call last):
[2023-04-09 15:56:48+0000] ERROR - distributed.shuffle._worker_extension | 'ca23d72b3d695085e9da34db46f8233c'
KeyError: 'ca23d72b3d695085e9da34db46f8233c'
state = self.states[id]
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get
result = handler(**msg)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
result = await self.worker.scheduler.shuffle_get(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle
shuffle = await self._refresh_shuffle(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run
shuffle = await self._get_shuffle_run(shuffle_id, run_id)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive
result = await result
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
return await self.rpc(address).shuffle_receive(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 123, in send
await self.send(address, shards)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_comms.py", line 71, in _process
await self._process(id, shards)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_buffer.py", line 107, in process
raise self._exception
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_buffer.py", line 222, in raise_on_exception
self._comm_buffer.raise_on_exception()
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 173, in inputs_done
await shuffle.inputs_done()
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 597, in shuffle_inputs_done
Traceback (most recent call last):
2023-04-09 15:56:48,352 - distributed.core - ERROR - Exception while handling op shuffle_inputs_done
KeyError: 'ca23d72b3d695085e9da34db46f8233c'
state = self.states[id]
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get
result = handler(**msg)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
result = await self.worker.scheduler.shuffle_get(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle
shuffle = await self._refresh_shuffle(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run
shuffle = await self._get_shuffle_run(shuffle_id, run_id)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive
result = await result
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
return await self.rpc(address).shuffle_receive(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 123, in send
await self.send(address, shards)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_comms.py", line 71, in _process
await self._process(id, shards)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_buffer.py", line 107, in process
raise self._exception
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_buffer.py", line 222, in raise_on_exception
self._comm_buffer.raise_on_exception()
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 173, in inputs_done
await shuffle.inputs_done()
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 597, in shuffle_inputs_done
result = await result
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm
Traceback (most recent call last):
[2023-04-09 15:56:48+0000] ERROR - distributed.core | Exception while handling op shuffle_inputs_done
KeyError: 'ca23d72b3d695085e9da34db46f8233c'
state = self.states[id]
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get
result = handler(**msg)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
result = await self.worker.scheduler.shuffle_get(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle
shuffle = await self._refresh_shuffle(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run
shuffle = await self._get_shuffle_run(shuffle_id, run_id)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive
result = await result
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
return await self.rpc(address).shuffle_receive(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 123, in send
await self.send(address, shards)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_comms.py", line 71, in _process
await self._process(id, shards)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_buffer.py", line 107, in process
raise self._exception
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_buffer.py", line 222, in raise_on_exception
self._comm_buffer.raise_on_exception()
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 173, in inputs_done
await shuffle.inputs_done()
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 597, in shuffle_inputs_done
result = await result
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm
Traceback (most recent call last):
/usr/local/lib/python3.10/site-packages/dask/dataframe/core.py:6984: FutureWarning: Meta is not valid, `map_partitions` and `map_overlap` expects output to be a pandas object. Try passing a pandas object as meta or a dict or tuple representing the (name, dtype) of the columns. In the future the meta you passed will not work.
warnings.warn(
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.104:37161
2023-04-09 15:57:51,606 - distributed.core - INFO - Removing comms to tcp://10.76.117.104:37161
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.125:39587
2023-04-09 15:57:51,606 - distributed.core - INFO - Removing comms to tcp://10.76.9.125:39587
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.90:40771
2023-04-09 15:57:51,606 - distributed.core - INFO - Removing comms to tcp://10.76.117.90:40771
2023-04-09 15:57:51,609 - distributed.core - INFO - Event loop was unresponsive in Worker for 63.60s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
[2023-04-09 15:57:51+0000] INFO - distributed.core | Event loop was unresponsive in Worker for 63.60s. This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.
2023-04-09 15:57:51,623 - distributed.core - INFO - Removing comms to tcp://10.76.17.3:33923
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.3:33923
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.147:38979
2023-04-09 15:57:51,627 - distributed.core - INFO - Removing comms to tcp://10.76.9.147:38979
2023-04-09 15:57:51,627 - distributed.core - INFO - Removing comms to tcp://10.76.9.128:34777
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.9.128:34777
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.92:38415
2023-04-09 15:57:51,627 - distributed.core - INFO - Removing comms to tcp://10.76.117.92:38415
2023-04-09 15:57:51,627 - distributed.core - INFO - Removing comms to tcp://10.76.7.6:40431
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.6:40431
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.3:36247
2023-04-09 15:57:51,627 - distributed.core - INFO - Removing comms to tcp://10.76.7.3:36247
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.65.78:43525
2023-04-09 15:57:51,627 - distributed.core - INFO - Removing comms to tcp://10.76.65.78:43525
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.4:42465
2023-04-09 15:57:51,627 - distributed.core - INFO - Removing comms to tcp://10.76.7.4:42465
2023-04-09 15:57:51,628 - distributed.core - INFO - Removing comms to tcp://10.76.117.103:41355
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.103:41355
2023-04-09 15:57:51,628 - distributed.core - INFO - Removing comms to tcp://10.76.7.5:33549
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.5:33549
2023-04-09 15:57:51,628 - distributed.core - INFO - Removing comms to tcp://10.76.117.95:32937
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.95:32937
2023-04-09 15:57:51,628 - distributed.core - INFO - Removing comms to tcp://10.76.19.3:37473
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.3:37473
2023-04-09 15:57:51,630 - distributed.core - INFO - Removing comms to tcp://10.76.28.3:45377
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.3:45377
2023-04-09 15:57:51,630 - distributed.core - INFO - Removing comms to tcp://10.76.17.4:34897
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.4:34897
2023-04-09 15:57:51,630 - distributed.core - INFO - Removing comms to tcp://10.76.28.5:40339
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.5:40339
2023-04-09 15:57:51,630 - distributed.core - INFO - Removing comms to tcp://10.76.28.4:38529
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.4:38529
2023-04-09 15:57:51,631 - distributed.core - INFO - Removing comms to tcp://10.76.19.4:36373
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.4:36373
2023-04-09 15:57:51,631 - distributed.core - INFO - Removing comms to tcp://10.76.7.7:46001
[2023-04-09 15:57:51+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.7:46001
[2023-04-09 15:57:54+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.11:37173
2023-04-09 15:57:54,416 - distributed.core - INFO - Removing comms to tcp://10.76.7.11:37173
[2023-04-09 15:57:54+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.8:36697
2023-04-09 15:57:54,425 - distributed.core - INFO - Removing comms to tcp://10.76.7.8:36697
[2023-04-09 15:57:54+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.10:39997
2023-04-09 15:57:54,435 - distributed.core - INFO - Removing comms to tcp://10.76.7.10:39997
[2023-04-09 15:57:54+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.9:46433
2023-04-09 15:57:54,435 - distributed.core - INFO - Removing comms to tcp://10.76.7.9:46433
[2023-04-09 15:58:00+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.12:38649
2023-04-09 15:58:00,159 - distributed.core - INFO - Removing comms to tcp://10.76.7.12:38649
[2023-04-09 15:58:00+0000] ERROR - distributed.core | Exception while handling op shuffle_receive
RuntimeError: Worker tcp://10.76.7.12:38649 left during active shuffle a4d98659bee6af96be24ed925c9a884f
raise self._exception
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 163, in raise_if_closed
self.raise_if_closed()
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 155, in _write_to_disk
await self._write_to_disk(groups)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 480, in _receive
await self._receive(data)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 463, in receive
await shuffle.receive(data)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 588, in shuffle_receive
result = await result
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm
Traceback (most recent call last):
2023-04-09 15:58:00,236 - distributed.core - ERROR - Exception while handling op shuffle_receive
RuntimeError: Worker tcp://10.76.7.12:38649 left during active shuffle a4d98659bee6af96be24ed925c9a884f
raise self._exception
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 163, in raise_if_closed
self.raise_if_closed()
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 155, in _write_to_disk
await self._write_to_disk(groups)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 480, in _receive
await self._receive(data)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 463, in receive
await shuffle.receive(data)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 588, in shuffle_receive
result = await result
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm
Traceback (most recent call last):
2023-04-09 15:58:00,249 - distributed.core - ERROR - Exception while handling op shuffle_receive
[2023-04-09 15:58:00+0000] ERROR - distributed.core | Exception while handling op shuffle_receive
RuntimeError: Stale shuffle
raise RuntimeError("Stale shuffle")
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 682, in _get_shuffle_run
shuffle = await self._get_shuffle_run(shuffle_id, run_id)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive
result = await result
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm
Traceback (most recent call last):
[2023-04-09 15:58:00+0000] ERROR - distributed.core | Exception while handling op shuffle_receive
KeyError: 'a4d98659bee6af96be24ed925c9a884f'
state = self.states[id]
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get
result = handler(**msg)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
result = await self.worker.scheduler.shuffle_get(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle
shuffle = await self._refresh_shuffle(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run
shuffle = await self._get_shuffle_run(shuffle_id, run_id)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive
result = await result
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm
Traceback (most recent call last):
RuntimeError: Stale shuffle
raise RuntimeError("Stale shuffle")
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 682, in _get_shuffle_run
shuffle = await self._get_shuffle_run(shuffle_id, run_id)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive
result = await result
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm
Traceback (most recent call last):
[2023-04-09 15:58:00+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.14:45377
2023-04-09 15:58:00,311 - distributed.core - ERROR - Exception while handling op shuffle_receive
KeyError: 'a4d98659bee6af96be24ed925c9a884f'
state = self.states[id]
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_scheduler_extension.py", line 127, in get
result = handler(**msg)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 828, in _handle_comm
raise exc.with_traceback(tb)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1049, in send_recv
return await send_recv(comm=comm, op=key, **kwargs)
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 1265, in send_recv_from_rpc
result = await self.worker.scheduler.shuffle_get(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 749, in _refresh_shuffle
shuffle = await self._refresh_shuffle(
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 678, in _get_shuffle_run
shuffle = await self._get_shuffle_run(shuffle_id, run_id)
File "/usr/local/lib/python3.10/site-packages/distributed/shuffle/_worker_extension.py", line 587, in shuffle_receive
result = await result
File "/usr/local/lib/python3.10/site-packages/distributed/core.py", line 830, in _handle_comm
Traceback (most recent call last):
2023-04-09 15:58:00,645 - distributed.core - INFO - Removing comms to tcp://10.76.7.14:45377
/usr/local/lib/python3.10/site-packages/dask/dataframe/core.py:6984: FutureWarning: Meta is not valid, `map_partitions` and `map_overlap` expects output to be a pandas object. Try passing a pandas object as meta or a dict or tuple representing the (name, dtype) of the columns. In the future the meta you passed will not work.
[2023-04-09 15:58:02+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.13:36351
warnings.warn(
2023-04-09 15:58:02,976 - distributed.core - INFO - Removing comms to tcp://10.76.7.13:36351
2023-04-09 15:58:03,985 - distributed.core - INFO - Removing comms to tcp://10.76.7.15:45883
[2023-04-09 15:58:03+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.15:45883
2023-04-09 15:58:06,430 - distributed.core - INFO - Removing comms to tcp://10.76.7.16:43913
[2023-04-09 15:58:06+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.16:43913
2023-04-09 15:58:07,441 - distributed.core - INFO - Removing comms to tcp://10.76.7.17:33859
[2023-04-09 15:58:07+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.17:33859
2023-04-09 15:58:09,160 - distributed.core - INFO - Removing comms to tcp://10.76.7.18:38483
[2023-04-09 15:58:09+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.18:38483
[2023-04-09 15:58:10+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.19:39703
2023-04-09 15:58:10,302 - distributed.core - INFO - Removing comms to tcp://10.76.7.19:39703
[2023-04-09 15:58:12+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.20:37461
2023-04-09 15:58:12,022 - distributed.core - INFO - Removing comms to tcp://10.76.7.20:37461
[2023-04-09 15:58:13+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.21:42639
2023-04-09 15:58:13,031 - distributed.core - INFO - Removing comms to tcp://10.76.7.21:42639
2023-04-09 15:58:15,056 - distributed.core - INFO - Removing comms to tcp://10.76.7.22:37661
[2023-04-09 15:58:15+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.22:37661
[2023-04-09 15:58:17+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.23:39713
2023-04-09 15:58:17,212 - distributed.core - INFO - Removing comms to tcp://10.76.7.23:39713
2023-04-09 15:58:19,193 - distributed.core - INFO - Removing comms to tcp://10.76.7.24:41493
[2023-04-09 15:58:19+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.24:41493
[2023-04-09 15:58:20+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.25:45635
2023-04-09 15:58:20,074 - distributed.core - INFO - Removing comms to tcp://10.76.7.25:45635
2023-04-09 15:58:22,083 - distributed.core - INFO - Removing comms to tcp://10.76.7.26:43345
[2023-04-09 15:58:22+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.26:43345
[2023-04-09 15:58:24+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.27:43925
2023-04-09 15:58:24,092 - distributed.core - INFO - Removing comms to tcp://10.76.7.27:43925
[2023-04-09 15:58:25+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.28:36047
2023-04-09 15:58:25,098 - distributed.core - INFO - Removing comms to tcp://10.76.7.28:36047
2023-04-09 15:58:27,118 - distributed.core - INFO - Removing comms to tcp://10.76.7.29:36805
[2023-04-09 15:58:27+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.29:36805
[2023-04-09 15:58:29+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.31:45077
2023-04-09 15:58:29,124 - distributed.core - INFO - Removing comms to tcp://10.76.7.31:45077
[2023-04-09 15:58:30+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.32:39645
2023-04-09 15:58:30,128 - distributed.core - INFO - Removing comms to tcp://10.76.7.32:39645
2023-04-09 15:58:32,142 - distributed.core - INFO - Removing comms to tcp://10.76.7.30:39415
[2023-04-09 15:58:32+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.30:39415
2023-04-09 15:58:34,151 - distributed.core - INFO - Removing comms to tcp://10.76.7.33:43941
[2023-04-09 15:58:34+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.33:43941
2023-04-09 15:58:35,287 - distributed.core - INFO - Removing comms to tcp://10.76.28.14:38443
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.14:38443
2023-04-09 15:58:35,287 - distributed.core - INFO - Removing comms to tcp://10.76.19.16:34879
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.16:34879
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.15:33929
2023-04-09 15:58:35,287 - distributed.core - INFO - Removing comms to tcp://10.76.17.15:33929
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.39:35533
2023-04-09 15:58:35,287 - distributed.core - INFO - Removing comms to tcp://10.76.7.39:35533
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.18:43819
2023-04-09 15:58:35,287 - distributed.core - INFO - Removing comms to tcp://10.76.28.18:43819
2023-04-09 15:58:35,287 - distributed.core - INFO - Removing comms to tcp://10.76.7.49:41661
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.49:41661
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.45:39845
2023-04-09 15:58:35,287 - distributed.core - INFO - Removing comms to tcp://10.76.7.45:39845
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.16:43729
2023-04-09 15:58:35,288 - distributed.core - INFO - Removing comms to tcp://10.76.28.16:43729
2023-04-09 15:58:35,288 - distributed.core - INFO - Removing comms to tcp://10.76.7.48:40677
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.48:40677
2023-04-09 15:58:35,288 - distributed.core - INFO - Removing comms to tcp://10.76.7.34:33577
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.34:33577
2023-04-09 15:58:35,288 - distributed.core - INFO - Removing comms to tcp://10.76.17.22:38489
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.22:38489
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.17:33719
2023-04-09 15:58:35,288 - distributed.core - INFO - Removing comms to tcp://10.76.17.17:33719
2023-04-09 15:58:35,288 - distributed.core - INFO - Removing comms to tcp://10.76.19.11:37067
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.11:37067
2023-04-09 15:58:35,288 - distributed.core - INFO - Removing comms to tcp://10.76.7.46:39677
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.46:39677
2023-04-09 15:58:35,288 - distributed.core - INFO - Removing comms to tcp://10.76.17.6:46027
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.6:46027
2023-04-09 15:58:35,288 - distributed.core - INFO - Removing comms to tcp://10.76.19.17:37697
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.17:37697
2023-04-09 15:58:35,288 - distributed.core - INFO - Removing comms to tcp://10.76.19.15:32769
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.15:32769
2023-04-09 15:58:35,288 - distributed.core - INFO - Removing comms to tcp://10.76.28.19:41825
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.19:41825
2023-04-09 15:58:35,288 - distributed.core - INFO - Removing comms to tcp://10.76.28.15:37167
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.15:37167
2023-04-09 15:58:35,288 - distributed.core - INFO - Removing comms to tcp://10.76.17.20:35041
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.20:35041
2023-04-09 15:58:35,289 - distributed.core - INFO - Removing comms to tcp://10.76.28.17:46221
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.17:46221
2023-04-09 15:58:35,289 - distributed.core - INFO - Removing comms to tcp://10.76.28.20:36963
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.20:36963
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.19:33049
2023-04-09 15:58:35,289 - distributed.core - INFO - Removing comms to tcp://10.76.19.19:33049
2023-04-09 15:58:35,289 - distributed.core - INFO - Removing comms to tcp://10.76.7.37:36807
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.37:36807
2023-04-09 15:58:35,289 - distributed.core - INFO - Removing comms to tcp://10.76.19.18:43969
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.18:43969
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.36:38533
2023-04-09 15:58:35,289 - distributed.core - INFO - Removing comms to tcp://10.76.7.36:38533
2023-04-09 15:58:35,289 - distributed.core - INFO - Removing comms to tcp://10.76.17.11:46091
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.11:46091
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.41:38825
2023-04-09 15:58:35,289 - distributed.core - INFO - Removing comms to tcp://10.76.7.41:38825
2023-04-09 15:58:35,289 - distributed.core - INFO - Removing comms to tcp://10.76.7.50:44429
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.50:44429
2023-04-09 15:58:35,289 - distributed.core - INFO - Removing comms to tcp://10.76.28.12:40011
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.12:40011
2023-04-09 15:58:35,289 - distributed.core - INFO - Removing comms to tcp://10.76.28.6:42893
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.6:42893
2023-04-09 15:58:35,289 - distributed.core - INFO - Removing comms to tcp://10.76.19.7:44433
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.7:44433
2023-04-09 15:58:35,289 - distributed.core - INFO - Removing comms to tcp://10.76.28.9:45017
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.9:45017
2023-04-09 15:58:35,290 - distributed.core - INFO - Removing comms to tcp://10.76.19.5:38911
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.5:38911
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.9:35877
2023-04-09 15:58:35,290 - distributed.core - INFO - Removing comms to tcp://10.76.19.9:35877
2023-04-09 15:58:35,290 - distributed.core - INFO - Removing comms to tcp://10.76.28.7:42501
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.7:42501
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.61:43793
2023-04-09 15:58:35,290 - distributed.core - INFO - Removing comms to tcp://10.76.117.61:43793
2023-04-09 15:58:35,290 - distributed.core - INFO - Removing comms to tcp://10.76.7.35:39413
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.35:39413
2023-04-09 15:58:35,290 - distributed.core - INFO - Removing comms to tcp://10.76.19.21:40021
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.21:40021
2023-04-09 15:58:35,290 - distributed.core - INFO - Removing comms to tcp://10.76.17.14:42727
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.14:42727
2023-04-09 15:58:35,290 - distributed.core - INFO - Removing comms to tcp://10.76.7.42:34953
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.42:34953
2023-04-09 15:58:35,290 - distributed.core - INFO - Removing comms to tcp://10.76.17.16:37455
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.16:37455
2023-04-09 15:58:35,290 - distributed.core - INFO - Removing comms to tcp://10.76.17.21:35309
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.21:35309
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.13:39117
2023-04-09 15:58:35,290 - distributed.core - INFO - Removing comms to tcp://10.76.19.13:39117
2023-04-09 15:58:35,290 - distributed.core - INFO - Removing comms to tcp://10.76.17.9:35497
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.9:35497
2023-04-09 15:58:35,290 - distributed.core - INFO - Removing comms to tcp://10.76.28.11:33319
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.11:33319
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.18:44755
2023-04-09 15:58:35,291 - distributed.core - INFO - Removing comms to tcp://10.76.17.18:44755
2023-04-09 15:58:35,291 - distributed.core - INFO - Removing comms to tcp://10.76.17.8:36251
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.8:36251
2023-04-09 15:58:35,291 - distributed.core - INFO - Removing comms to tcp://10.76.19.10:36831
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.10:36831
2023-04-09 15:58:35,291 - distributed.core - INFO - Removing comms to tcp://10.76.19.20:37825
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.20:37825
2023-04-09 15:58:35,291 - distributed.core - INFO - Removing comms to tcp://10.76.7.43:38071
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.43:38071
2023-04-09 15:58:35,291 - distributed.core - INFO - Removing comms to tcp://10.76.7.40:46691
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.40:46691
2023-04-09 15:58:35,291 - distributed.core - INFO - Removing comms to tcp://10.76.17.13:33681
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.13:33681
2023-04-09 15:58:35,291 - distributed.core - INFO - Removing comms to tcp://10.76.19.6:34521
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.6:34521
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.12:40935
2023-04-09 15:58:35,292 - distributed.core - INFO - Removing comms to tcp://10.76.17.12:40935
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.19:34327
2023-04-09 15:58:35,292 - distributed.core - INFO - Removing comms to tcp://10.76.17.19:34327
2023-04-09 15:58:35,292 - distributed.core - INFO - Removing comms to tcp://10.76.7.47:45667
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.47:45667
2023-04-09 15:58:35,292 - distributed.core - INFO - Removing comms to tcp://10.76.28.8:45155
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.8:45155
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.5:40253
2023-04-09 15:58:35,292 - distributed.core - INFO - Removing comms to tcp://10.76.17.5:40253
2023-04-09 15:58:35,292 - distributed.core - INFO - Removing comms to tcp://10.76.17.7:46267
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.7:46267
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.8:41017
2023-04-09 15:58:35,292 - distributed.core - INFO - Removing comms to tcp://10.76.19.8:41017
2023-04-09 15:58:35,292 - distributed.core - INFO - Removing comms to tcp://10.76.7.38:45473
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.38:45473
2023-04-09 15:58:35,292 - distributed.core - INFO - Removing comms to tcp://10.76.19.14:34803
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.14:34803
2023-04-09 15:58:35,293 - distributed.core - INFO - Removing comms to tcp://10.76.7.44:38215
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.7.44:38215
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.17.10:40415
2023-04-09 15:58:35,293 - distributed.core - INFO - Removing comms to tcp://10.76.17.10:40415
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.13:42999
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.28.10:42223
[2023-04-09 15:58:35+0000] INFO - distributed.core | Removing comms to tcp://10.76.19.12:45525
2023-04-09 15:58:35,293 - distributed.core - INFO - Removing comms to tcp://10.76.28.13:42999
2023-04-09 15:58:35,293 - distributed.core - INFO - Removing comms to tcp://10.76.28.10:42223
2023-04-09 15:58:35,293 - distributed.core - INFO - Removing comms to tcp://10.76.19.12:45525
[2023-04-09 16:01:03+0000] ERROR - asyncore_wsgi.SimpleWebSocketServer | Exception in <web_pdb.web_console.WebConsoleSocket connected 127.0.0.1:39698 at 0x7f768dbcba90>!
asyncore_wsgi.SimpleWebSocketServer.WebSocketError: remote socket closed
raise WebSocketError("remote socket closed")
File "/usr/local/lib/python3.10/site-packages/asyncore_wsgi/SimpleWebSocketServer.py", line 290, in _handleData
self._handleData()
File "/usr/local/lib/python3.10/site-packages/asyncore_wsgi/SimpleWebSocketServer.py", line 596, in handle_read
Traceback (most recent call last):
2023-04-09 16:11:20,280 - distributed.core - INFO - Removing comms to tcp://10.76.117.61:37003
[2023-04-09 16:11:20+0000] INFO - distributed.core | Removing comms to tcp://10.76.117.61:37003
2023-04-09 16:36:16,462 - distributed._signals - INFO - Received signal SIGTERM (15)
2023-04-09 16:36:16,463 - distributed.nanny - INFO - Closing Nanny at 'tcp://10.76.9.27:43473'. Reason: nanny-close
2023-04-09 16:36:16,463 - distributed.nanny - INFO - Nanny asking worker to close. Reason: nanny-close
2023-04-09 16:36:16,464 - distributed.worker - INFO - Stopping worker at tcp://10.76.9.27:46429. Reason: nanny-close
[2023-04-09 16:36:16+0000] INFO - distributed.worker | Stopping worker at tcp://10.76.9.27:46429. Reason: nanny-close
2023-04-09 16:36:16,472 - distributed.core - INFO - Connection to tcp://brett-1f0ecf6d-daskscheduler:8786 has been closed.
[2023-04-09 16:36:16+0000] INFO - distributed.core | Connection to tcp://brett-1f0ecf6d-daskscheduler:8786 has been closed.
/usr/local/lib/python3.10/site-packages/distributed/cli/dask_worker.py:264: FutureWarning: dask-worker is deprecated and will be removed in a future release; use `dask worker` instead
warnings.warn(
2023-04-09 16:36:52,371 - distributed.nanny - INFO - Start Nanny at: 'tcp://10.76.34.20:35471'
2023-04-09 16:36:53,088 - distributed.worker - INFO - Start worker at: tcp://10.76.34.20:45213
2023-04-09 16:36:53,088 - distributed.worker - INFO - Listening to: tcp://10.76.34.20:45213
2023-04-09 16:36:53,088 - distributed.worker - INFO - Worker name: brett-1f0ecf6d-daskworkers-large-865bf95dbc-fbk8j
2023-04-09 16:36:53,088 - distributed.worker - INFO - dashboard at: 10.76.34.20:8787
2023-04-09 16:36:53,088 - distributed.worker - INFO - Waiting to connect to: tcp://brett-1f0ecf6d-daskscheduler:8786
2023-04-09 16:36:53,088 - distributed.worker - INFO - -------------------------------------------------
2023-04-09 16:36:53,088 - distributed.worker - INFO - Threads: 4
2023-04-09 16:36:53,088 - distributed.worker - INFO - Memory: 89.41 GiB
2023-04-09 16:36:53,088 - distributed.worker - INFO - Local Directory: /tmp/dask-worker-space/worker-v6e12mq5
2023-04-09 16:36:53,088 - distributed.worker - INFO - -------------------------------------------------
2023-04-09 16:36:53,316 - distributed.worker - INFO - Registered to: tcp://brett-1f0ecf6d-daskscheduler:8786
2023-04-09 16:36:53,316 - distributed.worker - INFO - -------------------------------------------------
2023-04-09 16:36:53,317 - distributed.core - INFO - Starting established connection to tcp://brett-1f0ecf6d-daskscheduler:8786
[2023-04-09 16:36:56+0000] INFO - prefect.CloudTaskRunner | Task 'on_demand_auto_calibration-places/configs/south_central/2022_Q4/saturday.yaml': Starting task run...
[2023-04-09 16:36:56+0000] ERROR - prefect | Worker died while running task; retrying (attempt 2 of 3)
[2023-04-09 16:37:07+0000] INFO - root | Sources already exist, skipping DataManager(name='on_demand_auto_calibration', data_config='places/configs/south_central/2022_Q4/saturday.yaml')
[2023-04-09 16:37:07+0000] INFO - prefect.CloudTaskRunner | SKIP signal raised: SKIP(None)
[2023-04-09 16:37:07+0000] INFO - prefect.CloudTaskRunner | Task 'on_demand_auto_calibration-places/configs/south_central/2022_Q4/saturday.yaml': Finished task run for task with final state: 'Skipped'
2023-04-09 16:37:09,386 - distributed._signals - INFO - Received signal SIGTERM (15)
2023-04-09 16:37:09,386 - distributed.nanny - INFO - Closing Nanny at 'tcp://10.76.34.20:35471'. Reason: nanny-close
2023-04-09 16:37:09,386 - distributed.nanny - INFO - Nanny asking worker to close. Reason: nanny-close
2023-04-09 16:37:09,387 - distributed.worker - INFO - Stopping worker at tcp://10.76.34.20:45213. Reason: nanny-close
[2023-04-09 16:37:09+0000] INFO - distributed.worker | Stopping worker at tcp://10.76.34.20:45213. Reason: nanny-close
[2023-04-09 16:37:09+0000] INFO - distributed.core | Connection to tcp://brett-1f0ecf6d-daskscheduler:8786 has been closed.
2023-04-09 16:37:09,388 - distributed.core - INFO - Connection to tcp://brett-1f0ecf6d-daskscheduler:8786 has been closed.
[2023-04-09 16:37:09+0000] INFO - distributed.nanny | Worker closed
2023-04-09 16:37:09,389 - distributed.nanny - INFO - Worker closed
2023-04-09 16:37:10,664 - distributed.dask_worker - INFO - End worker
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment