Created
August 16, 2022 15:16
-
-
Save clarng/cd25c7308cd9fa10e1d59fd07cc2cb39 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| (base) ray@ip-172-31-89-194:~/workspace-project-rtoom_pipeline_data_ingest2/release/nightly_tests/dataset$ python data_ingest_benchmark.py --dataset-size-gb=200 --num-workers=20 --streaming | |
| 2022-08-16 07:55:17,950 INFO worker.py:1203 -- Using address localhost:9031 set in the environment variable RAY_ADDRESS | |
| 2022-08-16 07:55:20,407 INFO worker.py:1312 -- Connecting to existing Ray cluster at address: 172.31.89.194:9031... | |
| 2022-08-16 07:55:20,414 INFO worker.py:1487 -- Connected to Ray cluster. View the dashboard at https://session-aqwatdmgevphyalje148wqvj.i.anyscaleuserdata-staging.com/auth/?token=agh0_CkcwRQIgIk7mkj-gUcqPe1OsvZ_H8CPE_wsJBiwWJBszQZGOg8gCIQDWWn_c_wIHFEow6P51__K6H3uqXn3u3z5ucA_mHK8lJBJmEiD6fIFsE2xA7OC9UAjVkUdqDfyAqnjZHYJO1YMBnRNSfRgCIgNuL2E6DAii24bjBhCIutWeA0IMCKLY7pcGEIi61Z4D-gEeChxzZXNfQXF3YVREbWdFdnBoeUFMSkUxNDhXUXZK&redirect_to=dashboard. | |
| 2022-08-16 07:55:20,682 INFO packaging.py:342 -- Pushing file package 'gcs://_ray_pkg_b140455d05812ebf9ae368f7805fc29c.zip' (91.13MiB) to Ray cluster... | |
| 2022-08-16 07:55:21,765 INFO packaging.py:351 -- Successfully pushed file package 'gcs://_ray_pkg_b140455d05812ebf9ae368f7805fc29c.zip'. | |
| 2022-08-16 07:55:24,763 WARNING read_api.py:281 -- ⚠️ The blocks of this dataset are estimated to be 2.0x larger than the target block size of 512 MiB. This may lead to out-of-memory errors during processing. Consider reducing the size of input files or using `.repartition(n)` to increase the number of dataset blocks. | |
| Created dataset Dataset(num_blocks=201, num_rows=20971520, schema={__value__: ArrowTensorType(shape=(1280,), dtype=int64)}) of size 214748364800 | |
| 2022-08-16 07:55:24,970 INFO dataset.py:3282 -- Created DatasetPipeline with 6 windows: 1.17MiB min, 40.0GiB max, 33.33GiB mean | |
| 2022-08-16 07:55:24,970 INFO dataset.py:3291 -- Blocks per window: 1 min, 40 max, 33 mean | |
| 2022-08-16 07:55:24,972 WARNING dataset.py:3299 -- ⚠️ This pipeline's parallelism is limited by its blocks per window to ~33 concurrent tasks per window. To maximize performance, increase the blocks per window to at least 320. This may require increasing the base dataset's parallelism and/or adjusting the windowing parameters. | |
| 2022-08-16 07:55:24,973 INFO dataset.py:3326 -- ✔️ This pipeline's windows likely fit in object store memory without spilling. | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: : 2it [00:03, 1.80s/it] | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s].60s/it] | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: : 3it [00:05, 1.84s/it] pid=3184) | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: : 4it [00:07, 1.89s/it] pid=3184) | |
| Stage 0: : 2it [00:02, 1.17s/it] | |
| Stage 0: : 2it [00:02, 1.22s/it] | |
| Stage 0: : 2it [00:09, 4.58s/it] | |
| Stage 0: : 2it [00:09, 4.62s/it] | |
| Stage 0: : 2it [00:09, 4.77s/it] | |
| Stage 0: : 2it [00:09, 4.76s/it] | |
| (raylet, ip=172.31.79.34) Spilled 5121 MiB, 5 objects, write throughput 501 MiB/s. Set RAY_verbose_spill_logs=0 to disable this message. | |
| Stage 0: : 5it [00:18, 5.07s/it] pid=3184) | |
| Stage 0: : 3it [00:13, 5.48s/it]1.68.8) 4) | |
| (raylet, ip=172.31.79.34) Spilled 6146 MiB, 6 objects, write throughput 502 MiB/s. | |
| Stage 0: : 3it [00:17, 7.15s/it]1.79.34) | |
| (raylet, ip=172.31.79.34) Spilled 8195 MiB, 8 objects, write throughput 615 MiB/s. | |
| Stage 0: : 3it [00:18, 6.60s/it]1.68.8) | |
| Stage 0: : 3it [00:18, 6.66s/it]1.68.8) | |
| Stage 0: : 3it [00:19, 6.80s/it]1.68.8) | |
| Stage 0: : 3it [00:19, 6.74s/it]1.68.8) | |
| (raylet, ip=172.31.68.8) Spilled 8195 MiB, 8 objects, write throughput 875 MiB/s. Set RAY_verbose_spill_logs=0 to disable this message. | |
| (raylet, ip=172.31.68.8) Spilled 10243 MiB, 10 objects, write throughput 916 MiB/s. | |
| (ConsumingActor pid=764, ip=172.31.68.8) Warning: reader on shard 7 of the pipeline has been blocked more than 10.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| Stage 0: : 2it [00:25, 12.67s/it] | |
| Stage 0: : 2it [00:26, 13.10s/it] | |
| Stage 0: : 2it [00:26, 13.14s/it] | |
| Stage 0: : 2it [00:26, 13.26s/it] | |
| Stage 0: : 2it [00:26, 13.33s/it] | |
| Stage 0: : 2it [00:26, 13.45s/it] | |
| Stage 0: : 2it [00:26, 13.45s/it] | |
| Stage 0: : 2it [00:27, 13.51s/it] | |
| Stage 0: : 2it [00:27, 13.56s/it] | |
| Stage 0: : 2it [00:27, 13.59s/it] | |
| Stage 0: : 2it [00:27, 13.60s/it] | |
| Stage 0: : 2it [00:27, 13.55s/it] | |
| Stage 0: : 2it [00:27, 13.63s/it] | |
| Stage 0: : 2it [00:27, 13.63s/it] | |
| Stage 0: : 3it [01:43, 39.86s/it]1.79.34) | |
| Stage 0: : 3it [01:45, 40.54s/it]1.79.34) | |
| Stage 0: : 3it [01:45, 40.81s/it]1.79.34) | |
| Stage 0: : 3it [01:46, 41.01s/it]1.79.34) | |
| Stage 0: : 3it [01:48, 41.90s/it]1.79.34) | |
| Stage 0: : 3it [01:48, 41.99s/it]1.79.34) | |
| Stage 0: : 3it [01:49, 42.27s/it]1.79.34) | |
| Stage 0: : 3it [01:50, 42.53s/it]1.79.34) | |
| Stage 0: : 3it [01:50, 42.82s/it]1.79.34) | |
| Stage 0: : 3it [01:51, 42.98s/it]1.79.34) | |
| Stage 0: : 3it [01:51, 42.99s/it]1.79.34) | |
| Stage 0: : 3it [01:51, 43.02s/it]1.79.34) | |
| Stage 0: : 3it [01:55, 44.52s/it]1.79.34) | |
| Stage 0: : 6it [02:03, 38.32s/it] pid=3184) | |
| Stage 0: : 4it [01:58, 42.62s/it]1.68.8) 4) | |
| Stage 0: : 4it [02:02, 42.68s/it]1.68.8) | |
| Stage 0: : 4it [02:04, 43.43s/it]1.68.8) | |
| Stage 0: : 4it [02:04, 43.62s/it]1.68.8) | |
| Stage 0: : 4it [02:04, 43.66s/it]1.68.8) | |
| (ConsumingActor pid=768, ip=172.31.68.8) Warning: reader on shard 17 of the pipeline has been blocked more than 10.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=765, ip=172.31.68.8) Warning: reader on shard 8 of the pipeline has been blocked more than 10.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=766, ip=172.31.68.8) Warning: reader on shard 12 of the pipeline has been blocked more than 10.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=767, ip=172.31.68.8) Warning: reader on shard 15 of the pipeline has been blocked more than 10.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| Stage 0: : 4it [02:15, 48.37s/it]1.79.34) | |
| (ConsumingActor pid=764, ip=172.31.68.8) Warning: reader on shard 7 of the pipeline has been blocked more than 20.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=768, ip=172.31.68.8) Warning: reader on shard 17 of the pipeline has been blocked more than 20.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=765, ip=172.31.68.8) Warning: reader on shard 8 of the pipeline has been blocked more than 20.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=766, ip=172.31.68.8) Warning: reader on shard 12 of the pipeline has been blocked more than 20.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=767, ip=172.31.68.8) Warning: reader on shard 15 of the pipeline has been blocked more than 20.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=765, ip=172.31.79.34) Warning: reader on shard 0 of the pipeline has been blocked more than 10.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=765, ip=172.31.79.34) Warning: reader on shard 0 of the pipeline has been blocked more than 20.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=764, ip=172.31.68.8) Warning: reader on shard 7 of the pipeline has been blocked more than 40.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=768, ip=172.31.68.8) Warning: reader on shard 17 of the pipeline has been blocked more than 40.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=765, ip=172.31.68.8) Warning: reader on shard 8 of the pipeline has been blocked more than 40.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=766, ip=172.31.68.8) Warning: reader on shard 12 of the pipeline has been blocked more than 40.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=767, ip=172.31.68.8) Warning: reader on shard 15 of the pipeline has been blocked more than 40.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=765, ip=172.31.79.34) Warning: reader on shard 0 of the pipeline has been blocked more than 40.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| Stage 0: : 4it [03:08, 55.92s/it]1.79.34) | |
| Stage 0: : 4it [03:09, 56.50s/it]1.79.34) | |
| Stage 0: : 4it [03:10, 57.32s/it]1.79.34) | |
| Stage 0: : 4it [03:11, 57.41s/it]1.79.34) | |
| Stage 0: : 4it [03:11, 57.66s/it]1.79.34) | |
| Stage 0: : 4it [03:13, 57.73s/it]1.79.34) | |
| Stage 0: : 4it [03:13, 57.67s/it]1.79.34) | |
| Stage 0: : 4it [03:13, 57.69s/it]1.79.34) | |
| Stage 0: : 3it [03:13, 77.30s/it]1.79.34) | |
| Stage 0: : 7it [03:19, 50.30s/it] pid=3184) | |
| (_map_block_nosplit pid=762, ip=172.31.89.245) E0816 07:58:45.902934426 815 chttp2_transport.cc:1103] Received a GOAWAY with error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings" | |
| Stage 0: : 4it [03:14, 58.05s/it]1.79.34) | |
| Stage 0: : 5it [03:15, 54.56s/it]1.68.8) | |
| Stage 0: : 4it [03:17, 59.11s/it]1.79.34) | |
| Stage 0: : 5it [03:19, 54.50s/it]1.68.8) | |
| (raylet, ip=172.31.68.8) Spilled 12292 MiB, 12 objects, write throughput 830 MiB/s. | |
| Stage 0: : 5it [03:20, 54.81s/it]1.68.8) | |
| Stage 0: : 4it [03:20, 60.31s/it]1.79.34) | |
| Stage 0: : 5it [03:20, 55.20s/it]1.68.8) | |
| Stage 0: : 5it [03:20, 55.01s/it]1.68.8) | |
| Stage 0: : 4it [03:22, 62.05s/it]1.79.34) | |
| Stage 0: : 5it [03:26, 56.07s/it]1.79.34) | |
| Stage 0: : 4it [03:32, 64.14s/it]1.79.34) | |
| Stage 0: : 5it [03:59, 54.13s/it]1.79.34) | |
| Stage 0: : 5it [03:59, 54.38s/it]1.79.34) | |
| Stage 0: : 5it [04:00, 54.57s/it]1.79.34) | |
| Stage 0: : 5it [04:02, 55.26s/it]1.79.34) | |
| Stage 0: : 4it [04:02, 66.59s/it]1.79.34) | |
| Stage 0: : 5it [04:03, 55.70s/it]1.79.34) | |
| Stage 0: : 5it [04:06, 55.87s/it]1.79.34) | |
| Stage 0: : 5it [04:17, 59.91s/it]1.79.34) | |
| Stage 0: : 5it [04:18, 59.61s/it]1.79.34) | |
| Stage 0: : 5it [04:22, 61.21s/it]1.79.34) | |
| Stage 0: : 5it [04:23, 61.84s/it]1.79.34) | |
| Stage 0: : 5it [04:25, 62.91s/it]1.79.34) | |
| Stage 0: : 5it [04:34, 63.49s/it]1.79.34) | |
| Stage 0: : 5it [04:34, 54.62s/it]1.79.34) | |
| Stage 0: : 5it [04:36, 66.07s/it]1.79.34) | |
| Stage 0: : 8it [04:49, 62.89s/it] pid=3184) | |
| Stage 0: : 6it [04:44, 65.99s/it]1.68.8) 4) | |
| Stage 0: : 6it [04:49, 66.32s/it]1.68.8) | |
| Stage 0: : 6it [04:49, 66.31s/it]1.68.8) | |
| Stage 0: : 6it [04:49, 66.38s/it]1.68.8) | |
| Stage 0: : 6it [04:50, 66.41s/it]1.68.8) | |
| Stage 0: : 6it [05:16, 74.22s/it]1.79.34) | |
| (raylet, ip=172.31.95.112) Spilled 13317 MiB, 13 objects, write throughput 185 MiB/s. Set RAY_verbose_spill_logs=0 to disable this message. | |
| Stage 0: : 6it [06:13, 80.62s/it]1.79.34) | |
| Stage 0: : 6it [06:14, 80.95s/it]1.79.34) | |
| Stage 0: : 6it [06:15, 78.30s/it]1.79.34) | |
| Stage 0: : 6it [06:16, 78.75s/it]1.79.34) | |
| Stage 0: : 6it [06:16, 77.58s/it]1.79.34) | |
| Stage 0: : 6it [06:16, 78.94s/it]1.79.34) | |
| Stage 0: : 6it [06:18, 77.03s/it]1.79.34) | |
| Stage 0: : 6it [06:20, 71.66s/it]1.79.34) | |
| Stage 0: : 6it [06:22, 80.79s/it]1.79.34) | |
| Stage 0: : 6it [06:28, 83.26s/it]1.79.34) | |
| Stage 0: : 6it [06:29, 86.12s/it]1.79.34) | |
| Stage 0: : 6it [06:29, 86.19s/it]1.79.34) | |
| Stage 0: : 6it [06:31, 86.09s/it]1.79.34) | |
| Stage 0: : 6it [06:31, 85.56s/it]1.79.34) | |
| Stage 0: : 9it [06:37, 76.75s/it] pid=3184) | |
| Stage 0: : 6it [06:32, 65.34s/it]1.68.8) 4) | |
| Stage 0: : 6it [06:32, 65.34s/it]1.68.8) | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: : 6it [06:32, 65.34s/it]1.68.8) | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: : 6it [06:32, 65.34s/it]1.68.8) | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| (raylet, ip=172.31.68.8) [2022-08-16 08:02:04,959 E 527 527] (raylet) node_manager.cc:1715: Lost worker for wait request 0x56454f392480 | |
| Stage 0: : 6it [06:33, 65.52s/it]1.68.8) | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: : 6it [06:37, 66.27s/it]1.79.34) | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| (raylet, ip=172.31.71.11) Spilled 11268 MiB, 11 objects, write throughput 507 MiB/s. Set RAY_verbose_spill_logs=0 to disable this message. | |
| (raylet, ip=172.31.75.219) Spilled 13317 MiB, 13 objects, write throughput 446 MiB/s. Set RAY_verbose_spill_logs=0 to disable this message. | |
| (raylet, ip=172.31.72.218) Spilled 13317 MiB, 13 objects, write throughput 406 MiB/s. Set RAY_verbose_spill_logs=0 to disable this message. | |
| (raylet, ip=172.31.83.120) Spilled 13317 MiB, 13 objects, write throughput 403 MiB/s. Set RAY_verbose_spill_logs=0 to disable this message. | |
| (raylet, ip=172.31.75.249) Spilled 13317 MiB, 13 objects, write throughput 385 MiB/s. Set RAY_verbose_spill_logs=0 to disable this message. | |
| Stage 0: : 6it [07:08, 71.45s/it]1.79.34) | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| (raylet, ip=172.31.82.24) Spilled 14341 MiB, 14 objects, write throughput 386 MiB/s. Set RAY_verbose_spill_logs=0 to disable this message. | |
| (raylet, ip=172.31.67.11) Spilled 14341 MiB, 14 objects, write throughput 376 MiB/s. Set RAY_verbose_spill_logs=0 to disable this message. | |
| Stage 0: : 6it [07:12, 72.04s/it]1.79.34) | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: : 6it [07:12, 72.07s/it]1.79.34) | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: : 6it [07:18, 73.09s/it]1.79.34) | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: : 6it [07:26, 74.35s/it]1.79.34) | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| (raylet, ip=172.31.83.3) Spilled 13317 MiB, 13 objects, write throughput 234 MiB/s. Set RAY_verbose_spill_logs=0 to disable this message. | |
| (raylet, ip=172.31.69.57) Spilled 13317 MiB, 13 objects, write throughput 229 MiB/s. Set RAY_verbose_spill_logs=0 to disable this message. | |
| (raylet, ip=172.31.77.97) Spilled 14341 MiB, 14 objects, write throughput 234 MiB/s. Set RAY_verbose_spill_logs=0 to disable this message. | |
| (raylet, ip=172.31.95.12) Spilled 14341 MiB, 14 objects, write throughput 233 MiB/s. Set RAY_verbose_spill_logs=0 to disable this message. | |
| (raylet, ip=172.31.70.42) Spilled 14341 MiB, 14 objects, write throughput 231 MiB/s. Set RAY_verbose_spill_logs=0 to disable this message. | |
| (raylet) Spilled 14341 MiB, 213 objects, write throughput 226 MiB/s. Set RAY_verbose_spill_logs=0 to disable this message. | |
| Stage 0: : 10it [07:51, 75.91s/it]pid=3184) | |
| Stage 0: : 2it [01:13, 36.71s/it] | |
| Stage 0: : 2it [00:38, 19.23s/it] | |
| Stage 0: : 2it [00:41, 20.99s/it] | |
| Stage 0: : 2it [00:32, 16.13s/it] | |
| Stage 0: : 2it [00:38, 19.20s/it] | |
| Stage 0: : 2it [00:24, 12.30s/it] | |
| Stage 0: : 2it [01:19, 39.60s/it] | |
| Stage 0: : 2it [01:19, 39.97s/it] | |
| Stage 0: : 2it [01:20, 40.30s/it] | |
| Stage 0: : 2it [01:20, 40.37s/it] | |
| (raylet, ip=172.31.68.8) Spilled 18439 MiB, 18 objects, write throughput 778 MiB/s. | |
| (raylet, ip=172.31.79.34) Spilled 16390 MiB, 16 objects, write throughput 506 MiB/s. | |
| (ConsumingActor pid=774, ip=172.31.79.34) Warning: reader on shard 11 of the pipeline has been blocked more than 10.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=773, ip=172.31.79.34) Warning: reader on shard 10 of the pipeline has been blocked more than 10.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=775, ip=172.31.79.34) Warning: reader on shard 13 of the pipeline has been blocked more than 10.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=769, ip=172.31.79.34) Warning: reader on shard 4 of the pipeline has been blocked more than 10.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=771, ip=172.31.79.34) Warning: reader on shard 6 of the pipeline has been blocked more than 10.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| Stage 0: : 2it [01:28, 44.18s/it] | |
| (ConsumingActor pid=773, ip=172.31.79.34) Warning: reader on shard 10 of the pipeline has been blocked more than 20.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=774, ip=172.31.79.34) Warning: reader on shard 11 of the pipeline has been blocked more than 20.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=771, ip=172.31.79.34) Warning: reader on shard 6 of the pipeline has been blocked more than 20.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=775, ip=172.31.79.34) Warning: reader on shard 13 of the pipeline has been blocked more than 20.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=769, ip=172.31.79.34) Warning: reader on shard 4 of the pipeline has been blocked more than 20.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=774, ip=172.31.79.34) Warning: reader on shard 11 of the pipeline has been blocked more than 40.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=775, ip=172.31.79.34) Warning: reader on shard 13 of the pipeline has been blocked more than 40.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=769, ip=172.31.79.34) Warning: reader on shard 4 of the pipeline has been blocked more than 40.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| Stage 0: : 6it [08:32, 85.42s/it]1.79.34) | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| (ConsumingActor pid=773, ip=172.31.79.34) Warning: reader on shard 10 of the pipeline has been blocked more than 40.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=771, ip=172.31.79.34) Warning: reader on shard 6 of the pipeline has been blocked more than 40.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| Stage 0: : 2it [00:02, 1.16s/it] | |
| Stage 0: : 6it [08:38, 86.44s/it]1.79.34) | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: : 6it [08:39, 86.59s/it]1.79.34) | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: : 6it [08:40, 86.70s/it]1.79.34) | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: : 6it [08:42, 87.01s/it]1.79.34) | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: : 6it [08:43, 87.22s/it]1.79.34) | |
| Stage 0: : 2it [00:05, 2.60s/it] | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: : 6it [08:44, 87.41s/it]1.79.34) | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: : 2it [00:06, 3.09s/it] | |
| Stage 0: : 6it [08:45, 87.64s/it]1.79.34) | |
| Stage 0: : 6it [08:46, 87.69s/it]1.79.34) | |
| (ConsumingActor pid=778, ip=172.31.79.34) Warning: reader on shard 18 of the pipeline has been blocked more than 10.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: : 11it [08:53, 71.62s/it]pid=3184) | |
| Stage 0: 0%| | 0/1 [00:00<?, ?it/s] | |
| Stage 0: : 3it [02:15, 47.19s/it]1.68.8) | |
| Stage 0: : 3it [02:18, 47.97s/it]1.68.8) | |
| Stage 0: : 3it [02:20, 48.31s/it]1.68.8) | |
| Stage 0: : 2it [00:11, 5.62s/it] | |
| Stage 0: : 3it [02:20, 48.39s/it]1.68.8) | |
| Stage 0: : 2it [00:11, 5.72s/it] | |
| Stage 0: : 2it [00:15, 7.88s/it] | |
| Stage 0: : 2it [00:15, 7.56s/it] | |
| Stage 0: : 2it [00:14, 7.15s/it] | |
| Stage 0: : 3it [02:31, 52.29s/it]1.79.34) | |
| Stage 0: : 3it [02:46, 59.32s/it]1.68.8) | |
| Stage 0: : 3it [02:12, 50.57s/it]1.79.34) | |
| Stage 0: : 2it [00:40, 20.39s/it] | |
| Stage 0: : 3it [02:19, 52.95s/it]1.79.34) | |
| Stage 0: : 3it [00:56, 23.09s/it]1.79.34) | |
| Stage 0: : 3it [02:03, 48.23s/it]1.79.34) | |
| Stage 0: : 3it [00:52, 21.16s/it]1.79.34) | |
| Stage 0: : 3it [00:52, 19.74s/it]1.79.34) | |
| Stage 0: : 3it [02:26, 56.25s/it]1.79.34) | |
| Stage 0: : 3it [00:55, 21.15s/it]1.79.34) | |
| Stage 0: : 3it [01:06, 26.20s/it]1.79.34) | |
| Stage 0: : 3it [01:06, 26.35s/it]1.79.34) | |
| Stage 0: : 3it [01:10, 28.58s/it]1.79.34) | |
| Stage 0: : 3it [01:10, 24.32s/it]1.79.34) | |
| Stage 0: : 3it [02:38, 61.82s/it]1.79.34) | |
| Stage 0: : 3it [01:11, 28.14s/it]1.79.34) | |
| Stage 0: : 12it [10:15, 74.96s/it]pid=3184) | |
| Stage 0: : 13it [10:16, 52.62s/it]pid=3184) | |
| Stage 0: : 4it [03:38, 60.46s/it]1.68.8) 4) | |
| Stage 0: : 5it [03:39, 39.78s/it]1.68.8) | |
| Stage 0: : 4it [03:43, 61.51s/it]1.68.8) | |
| Stage 0: : 4it [03:43, 58.34s/it]1.68.8) | |
| Stage 0: : 4it [03:43, 61.31s/it]1.68.8) | |
| Stage 0: : 4it [03:43, 61.40s/it]1.68.8) | |
| Stage 0: : 5it [03:50, 40.50s/it]1.68.8) | |
| Stage 0: : 5it [03:50, 42.67s/it]1.68.8) | |
| Stage 0: : 5it [03:50, 42.49s/it]1.68.8) | |
| Stage 0: : 5it [03:51, 42.77s/it]1.68.8) | |
| Stage 0: : 4it [03:56, 64.16s/it]1.79.34) | |
| Stage 0: : 5it [04:30, 53.68s/it]1.79.34) | |
| Stage 0: : 4it [04:06, 76.25s/it]1.79.34) | |
| (ConsumingActor pid=764, ip=172.31.68.8) Warning: reader on shard 7 of the pipeline has been blocked more than 80.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| Stage 0: : 4it [02:51, 54.83s/it]1.79.34) | |
| Stage 0: : 4it [04:20, 77.00s/it]1.79.34) | |
| Stage 0: : 4it [04:28, 80.81s/it]1.79.34) | |
| Stage 0: : 4it [04:29, 82.54s/it]1.79.34) | |
| (ConsumingActor pid=767, ip=172.31.68.8) Warning: reader on shard 15 of the pipeline has been blocked more than 80.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=768, ip=172.31.68.8) Warning: reader on shard 17 of the pipeline has been blocked more than 80.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=766, ip=172.31.68.8) Warning: reader on shard 12 of the pipeline has been blocked more than 80.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| (ConsumingActor pid=765, ip=172.31.68.8) Warning: reader on shard 8 of the pipeline has been blocked more than 80.0s waiting for other readers to catch up. All pipeline shards must be read from concurrently. | |
| Stage 0: : 5it [04:21, 55.02s/it]1.79.34) | |
| Stage 0: : 4it [03:18, 67.93s/it]1.79.34) | |
| Stage 0: : 4it [03:23, 67.43s/it]1.79.34) | |
| Stage 0: : 4it [03:26, 68.65s/it]1.79.34) | |
| Stage 0: : 4it [03:22, 64.44s/it]1.79.34) | |
| Stage 0: : 4it [05:00, 93.06s/it]1.79.34) | |
| Stage 0: : 4it [03:22, 68.37s/it]1.79.34) | |
| Stage 0: : 4it [03:29, 70.00s/it]1.79.34) | |
| Stage 0: : 4it [03:38, 74.88s/it]1.79.34) | |
| Stage 0: : 4it [03:25, 69.65s/it]1.79.34) | |
| Stage 0: : 5it [05:03, 64.90s/it]1.79.34) | |
| Stage 0: : 5it [03:31, 49.64s/it]1.79.34) | |
| Stage 0: : 5it [05:01, 64.50s/it]1.79.34) | |
| Stage 0: : 5it [05:16, 70.23s/it]1.79.34) | |
| Stage 0: : 5it [04:04, 57.58s/it]1.79.34) | |
| Stage 0: : 5it [04:12, 60.89s/it]1.79.34) | |
| Stage 0: : 5it [04:00, 57.45s/it]1.79.34) | |
| Stage 0: : 5it [04:07, 59.00s/it]1.79.34) | |
| Stage 0: : 5it [04:03, 56.17s/it]1.79.34) | |
| Stage 0: : 5it [04:04, 58.47s/it]1.79.34) | |
| Stage 0: : 5it [04:06, 59.66s/it]1.79.34) | |
| Stage 0: : 14it [13:05, 87.70s/it]pid=3184) | |
| Stage 0: : 6it [06:28, 82.86s/it]1.68.8) 4) | |
| Stage 0: : 6it [06:33, 82.43s/it]1.68.8) | |
| Stage 0: : 6it [06:33, 82.65s/it]1.68.8) | |
| Stage 0: : 6it [06:33, 82.50s/it]1.68.8) | |
| Stage 0: : 6it [06:34, 81.52s/it]1.68.8) | |
| Stage 0: : 5it [06:09, 84.95s/it]1.79.34) | |
| Stage 0: : 6it [06:48, 81.88s/it]1.79.34) | |
| Stage 0: : 5it [05:18, 85.96s/it]1.79.34) | |
| (raylet, ip=172.31.89.245) Spilled 13317 MiB, 13 objects, write throughput 229 MiB/s. Set RAY_verbose_spill_logs=0 to disable this message. | |
| (raylet, ip=172.31.70.143) Spilled 13317 MiB, 13 objects, write throughput 219 MiB/s. Set RAY_verbose_spill_logs=0 to disable this message. | |
| (raylet, ip=172.31.72.157) Spilled 14341 MiB, 14 objects, write throughput 221 MiB/s. Set RAY_verbose_spill_logs=0 to disable this message. | |
| Stage 0: : 6it [06:48, 85.45s/it]1.79.34) | |
| Stage 0: : 6it [07:14, 86.80s/it]1.79.34) | |
| Stage 0: : 6it [05:48, 73.07s/it]1.79.34) | |
| Stage 0: : 6it [05:49, 73.35s/it]1.79.34) | |
| Stage 0: : 6it [05:44, 72.33s/it]1.79.34) | |
| Stage 0: : 6it [07:12, 86.47s/it]1.79.34) | |
| Stage 0: : 6it [05:59, 76.08s/it]1.79.34) | |
| Stage 0: : 6it [07:19, 87.95s/it]1.79.34) | |
| Stage 0: : 6it [05:51, 74.37s/it]1.79.34) | |
| Stage 0: : 6it [05:47, 78.32s/it]1.79.34) | |
| Stage 0: : 6it [05:51, 73.40s/it]1.79.34) | |
| Stage 0: : 6it [05:50, 74.87s/it]1.79.34) | |
| Stage 0: : 6it [07:34, 84.77s/it]1.79.34) | |
| Stage 0: : 15it [14:52, 93.34s/it]pid=3184) | |
| Stage 0: : 6it [08:14, 82.47s/it]1.68.8) 4) | |
| (ConsumingActor pid=765, ip=172.31.68.8) Time to read all data 892.7161115059998 seconds | |
| (ConsumingActor pid=765, ip=172.31.68.8) P50/P95/Max batch delay (s) 0.011408727000002727 0.033786879350009236 156.80731947099991 | |
| (ConsumingActor pid=765, ip=172.31.68.8) Num epochs read 2 | |
| (ConsumingActor pid=765, ip=172.31.68.8) Num batches read 512 | |
| (ConsumingActor pid=765, ip=172.31.68.8) Num bytes read 20480.0 MiB | |
| (ConsumingActor pid=765, ip=172.31.68.8) Mean throughput 22.94 MiB/s | |
| (ConsumingActor pid=766, ip=172.31.68.8) Time to read all data 892.6902119490001 seconds | |
| (ConsumingActor pid=766, ip=172.31.68.8) P50/P95/Max batch delay (s) 0.011779398499982108 0.029583962550009345 157.6520220289999 | |
| (ConsumingActor pid=766, ip=172.31.68.8) Num epochs read 2 | |
| (ConsumingActor pid=766, ip=172.31.68.8) Num batches read 512 | |
| (ConsumingActor pid=766, ip=172.31.68.8) Num bytes read 20480.0 MiB | |
| (ConsumingActor pid=766, ip=172.31.68.8) Mean throughput 22.94 MiB/s | |
| Stage 0: : 6it [08:14, 82.47s/it]1.68.8) | |
| Stage 0: : 6it [08:14, 82.47s/it]1.68.8) | |
| (ConsumingActor pid=768, ip=172.31.68.8) Time to read all data 892.5783806469999 seconds | |
| (ConsumingActor pid=768, ip=172.31.68.8) P50/P95/Max batch delay (s) 0.012018109000052846 0.01714436454998349 157.7415475570001 | |
| (ConsumingActor pid=768, ip=172.31.68.8) Num epochs read 2 | |
| (ConsumingActor pid=768, ip=172.31.68.8) Num batches read 512 | |
| (ConsumingActor pid=768, ip=172.31.68.8) Num bytes read 20480.0 MiB | |
| (ConsumingActor pid=768, ip=172.31.68.8) Mean throughput 22.94 MiB/s | |
| (ConsumingActor pid=767, ip=172.31.68.8) Time to read all data 892.6128069389999 seconds | |
| (ConsumingActor pid=767, ip=172.31.68.8) P50/P95/Max batch delay (s) 0.011590173500053425 0.026528185049994456 158.040519442 | |
| (ConsumingActor pid=767, ip=172.31.68.8) Num epochs read 2 | |
| (ConsumingActor pid=767, ip=172.31.68.8) Num batches read 512 | |
| (ConsumingActor pid=767, ip=172.31.68.8) Num bytes read 20480.0 MiB | |
| (ConsumingActor pid=767, ip=172.31.68.8) Mean throughput 22.94 MiB/s | |
| Stage 0: : 6it [08:14, 82.46s/it]1.68.8) | |
| (raylet, ip=172.31.68.8) [2022-08-16 08:10:19,226 E 527 527] (raylet) node_manager.cc:1715: Lost worker for wait request 0x56454ed18690 | |
| (ConsumingActor pid=764, ip=172.31.68.8) Time to read all data 893.2619461500001 seconds | |
| (ConsumingActor pid=764, ip=172.31.68.8) P50/P95/Max batch delay (s) 0.01205218099994454 0.023874584000009236 168.99451536100014 | |
| (ConsumingActor pid=764, ip=172.31.68.8) Num epochs read 2 | |
| (ConsumingActor pid=764, ip=172.31.68.8) Num batches read 512 | |
| (ConsumingActor pid=764, ip=172.31.68.8) Num bytes read 20480.0 MiB | |
| (ConsumingActor pid=764, ip=172.31.68.8) Mean throughput 22.93 MiB/s | |
| Stage 0: : 6it [08:14, 82.37s/it]1.68.8) | |
| (ConsumingActor pid=765, ip=172.31.79.34) Time to read all data 903.114247122 seconds | |
| (ConsumingActor pid=765, ip=172.31.79.34) P50/P95/Max batch delay (s) 0.011462264499982666 0.01547176514996238 118.33785075900005 | |
| (ConsumingActor pid=765, ip=172.31.79.34) Num epochs read 2 | |
| (ConsumingActor pid=765, ip=172.31.79.34) Num batches read 512 | |
| (ConsumingActor pid=765, ip=172.31.79.34) Num bytes read 20480.0 MiB | |
| (ConsumingActor pid=765, ip=172.31.79.34) Mean throughput 22.68 MiB/s | |
| Stage 0: : 6it [08:20, 83.43s/it]1.79.34) | |
| (ConsumingActor pid=765, ip=172.31.79.34) Ingest stats from rank=0: | |
| (ConsumingActor pid=765, ip=172.31.79.34) | |
| (ConsumingActor pid=765, ip=172.31.79.34) == Pipeline Window 10 == | |
| (ConsumingActor pid=765, ip=172.31.79.34) Stage 1 read->map_batches: 40/40 blocks executed in 82.63s | |
| (ConsumingActor pid=765, ip=172.31.79.34) * Remote wall time: 1.39s min, 6.72s max, 1.74s mean, 69.62s total | |
| (ConsumingActor pid=765, ip=172.31.79.34) * Remote cpu time: 1.43s min, 6.28s max, 1.74s mean, 69.61s total | |
| (ConsumingActor pid=765, ip=172.31.79.34) * Peak heap memory usage (MiB): 6538548000.0 min, 17368024000.0 max, 10805282500 mean | |
| (ConsumingActor pid=765, ip=172.31.79.34) * Output num rows: 104857 min, 104857 max, 104857 mean, 4194280 total | |
| (ConsumingActor pid=765, ip=172.31.79.34) * Output size bytes: 1074155212 min, 1074155212 max, 1074155212 mean, 42966208480 total | |
| (ConsumingActor pid=765, ip=172.31.79.34) * Tasks per node: 2 min, 2 max, 2 mean; 20 nodes used | |
| (ConsumingActor pid=765, ip=172.31.79.34) | |
| (ConsumingActor pid=765, ip=172.31.79.34) == Pipeline Window 11 == | |
| (ConsumingActor pid=765, ip=172.31.79.34) Stage 1 read->map_batches: 1/1 blocks executed in 0.02s | |
| (ConsumingActor pid=765, ip=172.31.79.34) * Remote wall time: 3.86ms min, 3.86ms max, 3.86ms mean, 3.86ms total | |
| (ConsumingActor pid=765, ip=172.31.79.34) * Remote cpu time: 3.88ms min, 3.88ms max, 3.88ms mean, 3.88ms total | |
| (ConsumingActor pid=765, ip=172.31.79.34) * Peak heap memory usage (MiB): 9686424000.0 min, 9686424000.0 max, 9686424000 mean | |
| (ConsumingActor pid=765, ip=172.31.79.34) * Output num rows: 120 min, 120 max, 120 mean, 120 total | |
| (ConsumingActor pid=765, ip=172.31.79.34) * Output size bytes: 1229284 min, 1229284 max, 1229284 mean, 1229284 total | |
| (ConsumingActor pid=765, ip=172.31.79.34) * Tasks per node: 1 min, 1 max, 1 mean; 1 nodes used | |
| (ConsumingActor pid=765, ip=172.31.79.34) | |
| (ConsumingActor pid=765, ip=172.31.79.34) == Pipeline Window 12 == | |
| (ConsumingActor pid=765, ip=172.31.79.34) Stage 1 read->map_batches: 40/40 blocks executed in 168.91s | |
| (ConsumingActor pid=765, ip=172.31.79.34) * Remote wall time: 1.38s min, 8.26s max, 1.8s mean, 72.1s total | |
| (ConsumingActor pid=765, ip=172.31.79.34) * Remote cpu time: 1.39s min, 8.0s max, 1.77s mean, 70.99s total | |
| (ConsumingActor pid=765, ip=172.31.79.34) * Peak heap memory usage (MiB): 2341120000.0 min, 21078148000.0 max, 11304942000 mean | |
| (ConsumingActor pid=765, ip=172.31.79.34) * Output num rows: 104857 min, 104857 max, 104857 mean, 4194280 total | |
| (ConsumingActor pid=765, ip=172.31.79.34) * Output size bytes: 1074155212 min, 1074155212 max, 1074155212 mean, 42966208480 total | |
| (ConsumingActor pid=765, ip=172.31.79.34) * Tasks per node: 1 min, 3 max, 2 mean; 20 nodes used | |
| (ConsumingActor pid=765, ip=172.31.79.34) | |
| (ConsumingActor pid=765, ip=172.31.79.34) ##### Overall Pipeline Time Breakdown ##### | |
| (ConsumingActor pid=765, ip=172.31.79.34) * Time stalled waiting for next dataset: 2.61ms min, 112.29s max, 57.27s mean, 687.18s total | |
| (ConsumingActor pid=765, ip=172.31.79.34) | |
| (ConsumingActor pid=771, ip=172.31.79.34) Time to read all data 916.317054734 seconds | |
| (ConsumingActor pid=771, ip=172.31.79.34) P50/P95/Max batch delay (s) 0.012271740500040096 0.039645962300011106 108.31087832899993 | |
| (ConsumingActor pid=771, ip=172.31.79.34) Num epochs read 2 | |
| (ConsumingActor pid=771, ip=172.31.79.34) Num batches read 512 | |
| (ConsumingActor pid=771, ip=172.31.79.34) Num bytes read 20480.0 MiB | |
| (ConsumingActor pid=771, ip=172.31.79.34) Mean throughput 22.35 MiB/s | |
| Stage 0: : 6it [07:45, 77.55s/it]1.79.34) | |
| (raylet, ip=172.31.95.112) Spilled 25609 MiB, 25 objects, write throughput 260 MiB/s. | |
| Stage 0: : 6it [06:42, 85.38s/it]1.79.34) | |
| (ConsumingActor pid=773, ip=172.31.79.34) Time to read all data 937.091727557 seconds | |
| (ConsumingActor pid=773, ip=172.31.79.34) P50/P95/Max batch delay (s) 0.011725475999980972 3.3577996468000206 108.61269720999996 | |
| (ConsumingActor pid=773, ip=172.31.79.34) Num epochs read 2 | |
| (ConsumingActor pid=773, ip=172.31.79.34) Num batches read 512 | |
| (ConsumingActor pid=773, ip=172.31.79.34) Num bytes read 20480.0 MiB | |
| (ConsumingActor pid=773, ip=172.31.79.34) Mean throughput 21.85 MiB/s | |
| Stage 0: : 6it [08:23, 83.90s/it]1.79.34) | |
| (ConsumingActor pid=774, ip=172.31.79.34) Time to read all data 937.826357503 seconds | |
| (ConsumingActor pid=774, ip=172.31.79.34) P50/P95/Max batch delay (s) 0.012096495500031779 0.03022132854996471 116.709431688 | |
| (ConsumingActor pid=774, ip=172.31.79.34) Num epochs read 2 | |
| (ConsumingActor pid=774, ip=172.31.79.34) Num batches read 512 | |
| (ConsumingActor pid=774, ip=172.31.79.34) Num bytes read 20480.0 MiB | |
| (ConsumingActor pid=774, ip=172.31.79.34) Mean throughput 21.84 MiB/s | |
| Stage 0: : 6it [08:20, 83.43s/it]1.79.34) | |
| (ConsumingActor pid=766, ip=172.31.79.34) Time to read all data 938.2858994989999 seconds | |
| (ConsumingActor pid=766, ip=172.31.79.34) P50/P95/Max batch delay (s) 0.012524688499979675 4.744923306150052 97.71694012600005 | |
| (ConsumingActor pid=766, ip=172.31.79.34) Num epochs read 2 | |
| (ConsumingActor pid=766, ip=172.31.79.34) Num batches read 512 | |
| (ConsumingActor pid=766, ip=172.31.79.34) Num bytes read 20480.0 MiB | |
| (ConsumingActor pid=766, ip=172.31.79.34) Mean throughput 21.83 MiB/s | |
| Stage 0: : 6it [06:50, 68.44s/it]1.79.34) | |
| (ConsumingActor pid=778, ip=172.31.79.34) Time to read all data 938.6569642789999 seconds | |
| (ConsumingActor pid=778, ip=172.31.79.34) P50/P95/Max batch delay (s) 0.012363616000016009 4.202708905149948 107.77246190599999 | |
| (ConsumingActor pid=778, ip=172.31.79.34) Num epochs read 2 | |
| (ConsumingActor pid=778, ip=172.31.79.34) Num batches read 512 | |
| (ConsumingActor pid=778, ip=172.31.79.34) Num bytes read 20480.0 MiB | |
| (ConsumingActor pid=778, ip=172.31.79.34) Mean throughput 21.82 MiB/s | |
| Stage 0: : 6it [07:00, 70.16s/it]1.79.34) | |
| Stage 0: : 6it [06:49, 68.25s/it]1.79.34) | |
| (ConsumingActor pid=776, ip=172.31.79.34) Time to read all data 938.743634147 seconds | |
| (ConsumingActor pid=776, ip=172.31.79.34) P50/P95/Max batch delay (s) 0.01297037600005524 4.284427989649991 83.03103318299998 | |
| (ConsumingActor pid=776, ip=172.31.79.34) Num epochs read 2 | |
| (ConsumingActor pid=776, ip=172.31.79.34) Num batches read 512 | |
| (ConsumingActor pid=776, ip=172.31.79.34) Num bytes read 20480.0 MiB | |
| (ConsumingActor pid=776, ip=172.31.79.34) Mean throughput 21.82 MiB/s | |
| Stage 0: : 6it [08:18, 83.08s/it]1.79.34) | |
| (ConsumingActor pid=769, ip=172.31.79.34) Time to read all data 941.9682744959999 seconds | |
| (ConsumingActor pid=769, ip=172.31.79.34) P50/P95/Max batch delay (s) 0.011906692999986035 3.16686551350001 102.59341769999992 | |
| (ConsumingActor pid=769, ip=172.31.79.34) Num epochs read 2 | |
| (ConsumingActor pid=769, ip=172.31.79.34) Num batches read 512 | |
| (ConsumingActor pid=769, ip=172.31.79.34) Num bytes read 20480.0 MiB | |
| (ConsumingActor pid=769, ip=172.31.79.34) Mean throughput 21.74 MiB/s | |
| Stage 0: : 6it [06:52, 68.75s/it]1.79.34) | |
| (ConsumingActor pid=777, ip=172.31.79.34) Time to read all data 943.1299037519999 seconds | |
| (ConsumingActor pid=777, ip=172.31.79.34) P50/P95/Max batch delay (s) 0.012432896999996501 4.394826554099982 103.65832109600001 | |
| (ConsumingActor pid=777, ip=172.31.79.34) Num epochs read 2 | |
| (ConsumingActor pid=777, ip=172.31.79.34) Num batches read 512 | |
| (ConsumingActor pid=777, ip=172.31.79.34) Num bytes read 20480.0 MiB | |
| (ConsumingActor pid=777, ip=172.31.79.34) Mean throughput 21.71 MiB/s | |
| (ConsumingActor pid=772, ip=172.31.79.34) Time to read all data 952.236804772 seconds | |
| (ConsumingActor pid=772, ip=172.31.79.34) P50/P95/Max batch delay (s) 0.012445422499979486 4.428248120599975 96.38540541400016 | |
| (ConsumingActor pid=772, ip=172.31.79.34) Num epochs read 2 | |
| (ConsumingActor pid=772, ip=172.31.79.34) Num batches read 512 | |
| (ConsumingActor pid=772, ip=172.31.79.34) Num bytes read 20480.0 MiB | |
| (ConsumingActor pid=772, ip=172.31.79.34) Mean throughput 21.51 MiB/s | |
| Stage 0: : 6it [06:59, 69.97s/it]1.79.34) | |
| Stage 0: : 6it [07:08, 71.46s/it]1.79.34) | |
| (ConsumingActor pid=803, ip=172.31.79.34) Time to read all data 955.5510917729999 seconds | |
| (ConsumingActor pid=803, ip=172.31.79.34) P50/P95/Max batch delay (s) 0.01286023099993372 4.44200972965001 94.12082756999996 | |
| (ConsumingActor pid=803, ip=172.31.79.34) Num epochs read 2 | |
| (ConsumingActor pid=803, ip=172.31.79.34) Num batches read 512 | |
| (ConsumingActor pid=803, ip=172.31.79.34) Num bytes read 20480.0 MiB | |
| (ConsumingActor pid=803, ip=172.31.79.34) Mean throughput 21.43 MiB/s | |
| (ConsumingActor pid=770, ip=172.31.79.34) Time to read all data 960.8499515880001 seconds | |
| (ConsumingActor pid=770, ip=172.31.79.34) P50/P95/Max batch delay (s) 0.012361509499953627 4.200363388049969 102.34274103099995 | |
| (ConsumingActor pid=770, ip=172.31.79.34) Num epochs read 2 | |
| (ConsumingActor pid=770, ip=172.31.79.34) Num batches read 512 | |
| (ConsumingActor pid=770, ip=172.31.79.34) Num bytes read 20480.0 MiB | |
| (ConsumingActor pid=770, ip=172.31.79.34) Mean throughput 21.31 MiB/s | |
| Stage 0: : 6it [07:17, 72.85s/it]1.79.34) | |
| Stage 0: : 6it [07:24, 74.07s/it]1.79.34) | |
| (ConsumingActor pid=768, ip=172.31.79.34) Time to read all data 969.978806003 seconds | |
| (ConsumingActor pid=768, ip=172.31.79.34) P50/P95/Max batch delay (s) 0.01248161850003271 4.485902221100013 99.13621175800006 | |
| (ConsumingActor pid=768, ip=172.31.79.34) Num epochs read 2 | |
| (ConsumingActor pid=768, ip=172.31.79.34) Num batches read 512 | |
| (ConsumingActor pid=768, ip=172.31.79.34) Num bytes read 20480.0 MiB | |
| (ConsumingActor pid=768, ip=172.31.79.34) Mean throughput 21.11 MiB/s | |
| Stage 0: : 6it [07:24, 74.02s/it]1.79.34) | |
| (ConsumingActor pid=767, ip=172.31.79.34) Time to read all data 976.211592077 seconds | |
| (ConsumingActor pid=767, ip=172.31.79.34) P50/P95/Max batch delay (s) 0.01349875300002168 4.400145288200017 94.41242023699999 | |
| (ConsumingActor pid=767, ip=172.31.79.34) Num epochs read 2 | |
| (ConsumingActor pid=767, ip=172.31.79.34) Num batches read 512 | |
| (ConsumingActor pid=767, ip=172.31.79.34) Num bytes read 20480.0 MiB | |
| (ConsumingActor pid=767, ip=172.31.79.34) Mean throughput 20.98 MiB/s | |
| Stage 0: : 6it [09:02, 90.37s/it]1.79.34) | |
| success! total time 988.9462239742279 | |
| (ConsumingActor pid=775, ip=172.31.79.34) Time to read all data 979.7272809620001 seconds | |
| (ConsumingActor pid=775, ip=172.31.79.34) P50/P95/Max batch delay (s) 0.012410168999963389 3.8293381860500175 103.67797986900007 | |
| (ConsumingActor pid=775, ip=172.31.79.34) Num epochs read 2 | |
| (ConsumingActor pid=775, ip=172.31.79.34) Num batches read 512 | |
| (ConsumingActor pid=775, ip=172.31.79.34) Num bytes read 20480.0 MiB | |
| (ConsumingActor pid=775, ip=172.31.79.34) Mean throughput 20.9 MiB/s |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment