Skip to content

Instantly share code, notes, and snippets.

@bveeramani

bveeramani/aws.txt

Created April 19, 2023 23:13
Show Gist options
  • Save bveeramani/809c5b24e7d7d8c3382618d6d6104f72 to your computer and use it in GitHub Desktop.
Save bveeramani/809c5b24e7d7d8c3382618d6d6104f72 to your computer and use it in GitHub Desktop.
Download ZIP
read_tfrecords_benchmark
Raw
aws.txt
[INFO 2023-04-13 16:18:26,072] anyscale_job_wrapper.py: 256 ### Starting ###
[INFO 2023-04-13 16:18:26,072] anyscale_job_wrapper.py: 280 ### Starting entrypoint ###
[INFO 2023-04-13 16:18:26,072] anyscale_job_wrapper.py: 162 Running command python read_tfrecords_benchmark.py
[INFO 2023-04-13 16:18:26,076] anyscale_job_wrapper.py: 176 Starting process 1561.
+ python read_tfrecords_benchmark.py
2023-04-13 16:18:27,133 INFO worker.py:1315 -- Using address 10.0.14.144:6379 set in the environment variable RAY_ADDRESS
2023-04-13 16:18:27,133 INFO worker.py:1432 -- Connecting to existing Ray cluster at address: 10.0.14.144:6379...
2023-04-13 16:18:27,139 INFO worker.py:1613 -- Connected to Ray cluster. View the dashboard at https://console.anyscale-staging.com/api/v2/sessions/ses_hapzispyymkvr3f7jsmrbi7txz/services?redirect_to=dashboard 
Running benchmark: read-tfrecords
2023-04-13 16:18:28,874 INFO streaming_executor.py:87 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[ReadImage->MapBatches(images_to_bytes)->Write]
2023-04-13 16:18:28,875 INFO streaming_executor.py:88 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=False, actor_locality_enabled=True, verbose_progress=False)
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 1.98 MiB/4.29 GiB object_store_memory: 0%| | 0/32 [00:00<?, ?it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=655) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 1.98 MiB/4.29 GiB object_store_memory: 0%| | 0/32 [00:00<?, ?it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=655) 2023-04-13 16:18:31.939072: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 1.98 MiB/4.29 GiB object_store_memory: 0%| | 0/32 [00:03<?, ?it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=655) 2023-04-13 16:18:31.939413: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 1.98 MiB/4.29 GiB object_store_memory: 0%| | 0/32 [00:03<?, ?it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=655) 2023-04-13 16:18:31.939426: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 1.98 MiB/4.29 GiB object_store_memory: 0%| | 0/32 [00:03<?, ?it/s]
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 0%| | 0/32 [00:05<?, ?it/s]
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 3%|▎ | 1/32 [00:05<02:51, 5.52s/it]
2023-04-13 16:18:34,453 WARNING plan.py:557 -- Warning: The Ray cluster currently does not have any available CPUs. The Dataset job will hang unless more CPUs are freed up. A common reason is that cluster resources are used by Actors or Tune trials; see the following link for more details: https://docs.ray.io/en/master/data/dataset-internals.html#datasets-and-tune
2023-04-13 16:20:02,769 INFO streaming_executor.py:87 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[ReadImage->MapBatches(images_to_bytes)->Write]
2023-04-13 16:20:02,769 INFO streaming_executor.py:88 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=False, actor_locality_enabled=True, verbose_progress=False)
Running 0: 0%| | 0/100 [00:00<?, ?it/s]
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 0%| | 0/100 [00:00<?, ?it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.29 GiB object_store_memory: 0%| | 0/100 [00:00<?, ?it/s]
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 0%| | 0/100 [00:00<?, ?it/s]
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 1%| | 1/100 [00:00<00:33, 2.98it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 36.06 MiB/4.29 GiB object_store_memory: 12%|█▏ | 12/100 [00:00<00:29, 2.98it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 36.06 MiB/4.29 GiB object_store_memory: 13%|█▎ | 13/100 [00:00<00:02, 36.71it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.29 GiB object_store_memory: 13%|█▎ | 13/100 [00:00<00:02, 36.71it/s]
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 13%|█▎ | 13/100 [00:00<00:02, 36.71it/s]
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 20%|██ | 20/100 [00:00<00:02, 35.10it/s]
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 26%|██▌ | 26/100 [00:00<00:01, 39.92it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 36.06 MiB/4.29 GiB object_store_memory: 26%|██▌ | 26/100 [00:00<00:01, 39.92it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.29 GiB object_store_memory: 26%|██▌ | 26/100 [00:00<00:01, 39.92it/s]
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 26%|██▌ | 26/100 [00:00<00:01, 39.92it/s]
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 32%|███▏ | 32/100 [00:00<00:01, 34.91it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 36.07 MiB/4.29 GiB object_store_memory: 38%|███▊ | 38/100 [00:01<00:01, 34.91it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 36.07 MiB/4.29 GiB object_store_memory: 39%|███▉ | 39/100 [00:01<00:01, 41.91it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.29 GiB object_store_memory: 39%|███▉ | 39/100 [00:01<00:01, 41.91it/s]
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 39%|███▉ | 39/100 [00:01<00:01, 41.91it/s]
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 45%|████▌ | 45/100 [00:01<00:01, 36.64it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 36.07 MiB/4.29 GiB object_store_memory: 51%|█████ | 51/100 [00:01<00:01, 36.64it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 36.07 MiB/4.29 GiB object_store_memory: 52%|█████▏ | 52/100 [00:01<00:01, 41.78it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.29 GiB object_store_memory: 52%|█████▏ | 52/100 [00:01<00:01, 41.78it/s]
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 52%|█████▏ | 52/100 [00:01<00:01, 41.78it/s]
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 57%|█████▋ | 57/100 [00:01<00:01, 35.74it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 36.06 MiB/4.29 GiB object_store_memory: 64%|██████▍ | 64/100 [00:01<00:01, 35.74it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 36.06 MiB/4.29 GiB object_store_memory: 65%|██████▌ | 65/100 [00:01<00:00, 43.54it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.29 GiB object_store_memory: 65%|██████▌ | 65/100 [00:01<00:00, 43.54it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=3024) 2023-04-13 16:20:04.683674: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA [repeated 16x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/ray-logging.html#log-deduplication for more options.)
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.29 GiB object_store_memory: 65%|██████▌ | 65/100 [00:01<00:00, 43.54it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=3024) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. [repeated 16x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.29 GiB object_store_memory: 65%|██████▌ | 65/100 [00:01<00:00, 43.54it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=1694) 2023-04-13 16:18:32.073948: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 [repeated 30x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.29 GiB object_store_memory: 65%|██████▌ | 65/100 [00:01<00:00, 43.54it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=1694) 2023-04-13 16:18:32.073959: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. [repeated 15x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.29 GiB object_store_memory: 65%|██████▌ | 65/100 [00:01<00:00, 43.54it/s]
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 65%|██████▌ | 65/100 [00:01<00:00, 43.54it/s]
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 70%|███████ | 70/100 [00:01<00:00, 36.33it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 36.07 MiB/4.29 GiB object_store_memory: 77%|███████▋ | 77/100 [00:02<00:00, 36.33it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 36.07 MiB/4.29 GiB object_store_memory: 78%|███████▊ | 78/100 [00:02<00:00, 44.06it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.29 GiB object_store_memory: 78%|███████▊ | 78/100 [00:02<00:00, 44.06it/s]
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 78%|███████▊ | 78/100 [00:02<00:00, 44.06it/s]
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 9.62 MiB/4.29 GiB object_store_memory: 84%|████████▍ | 84/100 [00:02<00:00, 38.41it/s]
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 21.64 MiB/4.29 GiB object_store_memory: 90%|█████████ | 90/100 [00:02<00:00, 38.41it/s]
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 21.64 MiB/4.29 GiB object_store_memory: 91%|█████████ | 91/100 [00:02<00:00, 44.72it/s]
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 12.02 MiB/4.29 GiB object_store_memory: 91%|█████████ | 91/100 [00:02<00:00, 44.72it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 7.21 MiB/4.29 GiB object_store_memory: 95%|█████████▌| 95/100 [00:02<00:00, 44.72it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 7.21 MiB/4.29 GiB object_store_memory: 97%|█████████▋| 97/100 [00:02<00:00, 39.16it/s]
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 97%|█████████▋| 97/100 [00:04<00:00, 39.16it/s]
2023-04-13 16:20:15,642 INFO streaming_executor.py:87 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[ReadImage->MapBatches(images_to_bytes)->Write]
2023-04-13 16:20:15,642 INFO streaming_executor.py:88 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=False, actor_locality_enabled=True, verbose_progress=False)
Running 0: 0%| | 0/51 [00:00<?, ?it/s]
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 0%| | 0/51 [00:00<?, ?it/s]
Running: 6.0/16.0 CPU, 0.0/0.0 GPU, 6.84 MiB/4.29 GiB object_store_memory: 0%| | 0/51 [00:00<?, ?it/s]
Running: 6.0/16.0 CPU, 0.0/0.0 GPU, 6.84 MiB/4.29 GiB object_store_memory: 2%|▏ | 1/51 [00:00<00:05, 8.70it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.29 GiB object_store_memory: 20%|█▉ | 10/51 [00:00<00:04, 8.70it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.29 GiB object_store_memory: 22%|██▏ | 11/51 [00:00<00:00, 58.15it/s]
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 5.05 MiB/4.29 GiB object_store_memory: 45%|████▌ | 23/51 [00:00<00:00, 58.15it/s]
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 5.05 MiB/4.29 GiB object_store_memory: 47%|████▋ | 24/51 [00:00<00:00, 84.89it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.29 GiB object_store_memory: 67%|██████▋ | 34/51 [00:00<00:00, 84.89it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.29 GiB object_store_memory: 69%|██████▊ | 35/51 [00:00<00:00, 92.03it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.29 GiB object_store_memory: 94%|█████████▍| 48/51 [00:00<00:00, 104.62it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=3397) 2023-04-13 16:20:16.952631: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA [repeated 3x across cluster]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.29 GiB object_store_memory: 94%|█████████▍| 48/51 [00:01<00:00, 104.62it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=3397) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. [repeated 3x across cluster]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.29 GiB object_store_memory: 94%|█████████▍| 48/51 [00:01<00:00, 104.62it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=3025) 2023-04-13 16:20:05.756329: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 [repeated 6x across cluster]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.29 GiB object_store_memory: 94%|█████████▍| 48/51 [00:01<00:00, 104.62it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=3025) 2023-04-13 16:20:05.756336: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. [repeated 3x across cluster]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.29 GiB object_store_memory: 94%|█████████▍| 48/51 [00:01<00:00, 104.62it/s]
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 0.9 MiB/4.29 GiB object_store_memory: 94%|█████████▍| 48/51 [00:03<00:00, 104.62it/s]
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 98%|█████████▊| 50/51 [00:03<00:00, 104.62it/s]
2023-04-13 16:20:19,142 INFO streaming_executor.py:87 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[ReadRange->MapBatches(generate_features)->Write]
2023-04-13 16:20:19,142 INFO streaming_executor.py:88 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=False, actor_locality_enabled=True, verbose_progress=False)
Running 0: 0%| | 0/80 [00:00<?, ?it/s]
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 0%| | 0/80 [00:00<?, ?it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 0%| | 0/80 [00:00<?, ?it/s]
(ReadRange->MapBatches(generate_features)->Write pid=3396) 2023-04-13 16:20:52.665776: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA [repeated 3x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 0%| | 0/80 [00:33<?, ?it/s]
(ReadRange->MapBatches(generate_features)->Write pid=3396) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. [repeated 3x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 0%| | 0/80 [00:33<?, ?it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=3399) 2023-04-13 16:20:17.954977: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 [repeated 6x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 0%| | 0/80 [00:33<?, ?it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=3399) 2023-04-13 16:20:17.954984: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. [repeated 3x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 0%| | 0/80 [00:33<?, ?it/s]
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.29 GiB object_store_memory: 0%| | 0/80 [01:03<?, ?it/s]
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.29 GiB object_store_memory: 1%|▏ | 1/80 [01:03<1:24:05, 63.87s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 4%|▍ | 3/80 [01:03<1:21:58, 63.87s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 5%|▌ | 4/80 [01:04<15:22, 12.14s/it]
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.29 GiB object_store_memory: 6%|▋ | 5/80 [01:04<15:10, 12.14s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 8%|▊ | 6/80 [01:04<14:58, 12.14s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 9%|▉ | 7/80 [01:04<06:52, 5.65s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 10%|█ | 8/80 [01:04<06:46, 5.65s/it]
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.29 GiB object_store_memory: 10%|█ | 8/80 [01:04<06:46, 5.65s/it]
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.29 GiB object_store_memory: 12%|█▎ | 10/80 [01:04<03:45, 3.23s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 14%|█▍ | 11/80 [01:04<03:42, 3.23s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 15%|█▌ | 12/80 [01:04<02:36, 2.30s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 15%|█▌ | 12/80 [01:04<02:36, 2.30s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 15%|█▌ | 12/80 [01:08<02:36, 2.30s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 16%|█▋ | 13/80 [01:08<02:33, 2.30s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 16%|█▋ | 13/80 [01:08<02:33, 2.30s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 18%|█▊ | 14/80 [01:08<02:27, 2.24s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 18%|█▊ | 14/80 [01:08<02:27, 2.24s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 19%|█▉ | 15/80 [01:09<02:25, 2.24s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 19%|█▉ | 15/80 [01:09<02:25, 2.24s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 19%|█▉ | 15/80 [01:09<02:25, 2.24s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 20%|██ | 16/80 [01:09<01:48, 1.69s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 20%|██ | 16/80 [01:09<01:48, 1.69s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 20%|██ | 16/80 [02:07<01:48, 1.69s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 21%|██▏ | 17/80 [02:07<12:02, 11.46s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 22%|██▎ | 18/80 [02:07<11:50, 11.46s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 22%|██▎ | 18/80 [02:07<11:50, 11.46s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 24%|██▍ | 19/80 [02:07<07:48, 7.68s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 24%|██▍ | 19/80 [02:07<07:48, 7.68s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 24%|██▍ | 19/80 [02:07<07:48, 7.68s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 25%|██▌ | 20/80 [02:07<06:14, 6.25s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 26%|██▋ | 21/80 [02:07<06:08, 6.25s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 26%|██▋ | 21/80 [02:07<06:08, 6.25s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 28%|██▊ | 22/80 [02:07<03:56, 4.07s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 29%|██▉ | 23/80 [02:07<03:51, 4.07s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 29%|██▉ | 23/80 [02:07<03:51, 4.07s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 30%|███ | 24/80 [02:07<02:33, 2.74s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 31%|███▏ | 25/80 [02:08<02:02, 2.23s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 31%|███▏ | 25/80 [02:08<02:02, 2.23s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 34%|███▍ | 27/80 [02:08<01:58, 2.23s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 35%|███▌ | 28/80 [02:08<01:05, 1.27s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 35%|███▌ | 28/80 [02:08<01:05, 1.27s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 35%|███▌ | 28/80 [02:08<01:05, 1.27s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 36%|███▋ | 29/80 [02:11<01:25, 1.67s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 36%|███▋ | 29/80 [02:11<01:25, 1.67s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 36%|███▋ | 29/80 [02:12<01:25, 1.67s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 36%|███▋ | 29/80 [02:12<01:25, 1.67s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 38%|███▊ | 30/80 [02:12<01:12, 1.44s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 38%|███▊ | 30/80 [02:12<01:12, 1.44s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 38%|███▊ | 30/80 [02:12<01:12, 1.44s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 39%|███▉ | 31/80 [02:12<00:57, 1.17s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 39%|███▉ | 31/80 [02:12<00:57, 1.17s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 39%|███▉ | 31/80 [02:13<00:57, 1.17s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 40%|████ | 32/80 [02:13<00:51, 1.06s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 40%|████ | 32/80 [02:13<00:51, 1.06s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 41%|████▏ | 33/80 [03:10<12:13, 15.60s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 41%|████▏ | 33/80 [03:10<12:13, 15.60s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 41%|████▏ | 33/80 [03:11<12:13, 15.60s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 41%|████▏ | 33/80 [03:11<12:13, 15.60s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 42%|████▎ | 34/80 [03:11<08:48, 11.49s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 44%|████▍ | 35/80 [03:11<06:14, 8.33s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 44%|████▍ | 35/80 [03:11<06:14, 8.33s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 45%|████▌ | 36/80 [03:11<04:25, 6.03s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 45%|████▌ | 36/80 [03:11<04:25, 6.03s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 46%|████▋ | 37/80 [03:11<03:05, 4.32s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 48%|████▊ | 38/80 [03:11<02:09, 3.09s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 48%|████▊ | 38/80 [03:11<02:09, 3.09s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 49%|████▉ | 39/80 [03:11<01:30, 2.21s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 50%|█████ | 40/80 [03:12<01:28, 2.21s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 50%|█████ | 40/80 [03:12<01:28, 2.21s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 51%|█████▏ | 41/80 [03:12<00:48, 1.25s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 51%|█████▏ | 41/80 [03:12<00:48, 1.25s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 51%|█████▏ | 41/80 [03:12<00:48, 1.25s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 52%|█████▎ | 42/80 [03:12<00:37, 1.00it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 54%|█████▍ | 43/80 [03:12<00:36, 1.00it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 55%|█████▌ | 44/80 [03:12<00:21, 1.65it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 55%|█████▌ | 44/80 [03:12<00:21, 1.65it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 55%|█████▌ | 44/80 [03:15<00:21, 1.65it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 56%|█████▋ | 45/80 [03:15<00:21, 1.65it/s]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 56%|█████▋ | 45/80 [03:15<00:21, 1.65it/s]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 57%|█████▊ | 46/80 [03:15<00:34, 1.03s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 59%|█████▉ | 47/80 [03:16<00:33, 1.03s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 59%|█████▉ | 47/80 [03:17<00:33, 1.03s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 60%|██████ | 48/80 [03:17<00:28, 1.14it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 60%|██████ | 48/80 [03:17<00:28, 1.14it/s]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 60%|██████ | 48/80 [04:14<00:28, 1.14it/s]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 61%|██████▏ | 49/80 [04:14<06:16, 12.14s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 62%|██████▎ | 50/80 [04:14<06:04, 12.14s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 64%|██████▍ | 51/80 [04:15<03:45, 7.79s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 64%|██████▍ | 51/80 [04:15<03:45, 7.79s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 65%|██████▌ | 52/80 [04:15<02:53, 6.21s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 65%|██████▌ | 52/80 [04:15<02:53, 6.21s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 65%|██████▌ | 52/80 [04:15<02:53, 6.21s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 66%|██████▋ | 53/80 [04:15<02:11, 4.88s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 66%|██████▋ | 53/80 [04:15<02:11, 4.88s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 66%|██████▋ | 53/80 [04:15<02:11, 4.88s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 68%|██████▊ | 54/80 [04:15<01:37, 3.73s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 69%|██████▉ | 55/80 [04:15<01:33, 3.73s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 70%|███████ | 56/80 [04:15<00:53, 2.22s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 70%|███████ | 56/80 [04:15<00:53, 2.22s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 72%|███████▎ | 58/80 [04:15<00:48, 2.22s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 72%|███████▎ | 58/80 [04:16<00:48, 2.22s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 74%|███████▍ | 59/80 [04:16<00:25, 1.23s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 74%|███████▍ | 59/80 [04:16<00:25, 1.23s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 74%|███████▍ | 59/80 [04:16<00:25, 1.23s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 75%|███████▌ | 60/80 [04:16<00:20, 1.04s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 75%|███████▌ | 60/80 [04:16<00:20, 1.04s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 75%|███████▌ | 60/80 [04:18<00:20, 1.04s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 76%|███████▋ | 61/80 [04:18<00:25, 1.36s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 76%|███████▋ | 61/80 [04:19<00:25, 1.36s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 76%|███████▋ | 61/80 [04:19<00:25, 1.36s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 78%|███████▊ | 62/80 [04:19<00:22, 1.23s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 78%|███████▊ | 62/80 [04:19<00:22, 1.23s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 78%|███████▊ | 62/80 [04:19<00:22, 1.23s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 79%|███████▉ | 63/80 [04:19<00:16, 1.02it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 79%|███████▉ | 63/80 [04:19<00:16, 1.02it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 79%|███████▉ | 63/80 [04:21<00:16, 1.02it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 80%|████████ | 64/80 [04:21<00:17, 1.12s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 80%|████████ | 64/80 [04:21<00:17, 1.12s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 81%|████████▏ | 65/80 [05:18<04:04, 16.30s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 81%|████████▏ | 65/80 [05:18<04:04, 16.30s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 81%|████████▏ | 65/80 [05:18<04:04, 16.30s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 82%|████████▎ | 66/80 [05:18<02:45, 11.84s/it]
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.29 GiB object_store_memory: 82%|████████▎ | 66/80 [05:19<02:45, 11.84s/it]
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.29 GiB object_store_memory: 84%|████████▍ | 67/80 [05:19<01:50, 8.50s/it]
Running: 11.0/16.0 CPU, 0.0/0.0 GPU, 11.0 MiB/4.29 GiB object_store_memory: 84%|████████▍ | 67/80 [05:19<01:50, 8.50s/it]
Running: 11.0/16.0 CPU, 0.0/0.0 GPU, 11.0 MiB/4.29 GiB object_store_memory: 85%|████████▌ | 68/80 [05:19<01:12, 6.07s/it]
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 9.0 MiB/4.29 GiB object_store_memory: 86%|████████▋ | 69/80 [05:19<01:06, 6.07s/it]
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 9.0 MiB/4.29 GiB object_store_memory: 88%|████████▊ | 70/80 [05:19<00:33, 3.35s/it]
Running: 8.0/16.0 CPU, 0.0/0.0 GPU, 8.0 MiB/4.29 GiB object_store_memory: 89%|████████▉ | 71/80 [05:19<00:30, 3.35s/it]
Running: 8.0/16.0 CPU, 0.0/0.0 GPU, 8.0 MiB/4.29 GiB object_store_memory: 90%|█████████ | 72/80 [05:19<00:16, 2.05s/it]
Running: 6.0/16.0 CPU, 0.0/0.0 GPU, 6.0 MiB/4.29 GiB object_store_memory: 90%|█████████ | 72/80 [05:19<00:16, 2.05s/it]
Running: 6.0/16.0 CPU, 0.0/0.0 GPU, 6.0 MiB/4.29 GiB object_store_memory: 92%|█████████▎| 74/80 [05:19<00:08, 1.35s/it]
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 92%|█████████▎| 74/80 [05:19<00:08, 1.35s/it]
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 95%|█████████▌| 76/80 [05:19<00:03, 1.06it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.0 MiB/4.29 GiB object_store_memory: 95%|█████████▌| 76/80 [05:21<00:03, 1.06it/s]
Running: 2.0/16.0 CPU, 0.0/0.0 GPU, 2.0 MiB/4.29 GiB object_store_memory: 96%|█████████▋| 77/80 [05:21<00:02, 1.06it/s]
Running: 2.0/16.0 CPU, 0.0/0.0 GPU, 2.0 MiB/4.29 GiB object_store_memory: 98%|█████████▊| 78/80 [05:21<00:01, 1.09it/s]
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 1.0 MiB/4.29 GiB object_store_memory: 98%|█████████▊| 78/80 [05:21<00:01, 1.09it/s]
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 1.0 MiB/4.29 GiB object_store_memory: 99%|█████████▉| 79/80 [05:21<00:00, 1.31it/s]
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 99%|█████████▉| 79/80 [05:22<00:00, 1.31it/s]
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 100%|██████████| 80/80 [05:22<00:00, 1.22it/s]
2023-04-13 16:25:41,909 INFO streaming_executor.py:87 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[ReadRange->MapBatches(generate_features)->Write]
2023-04-13 16:25:41,909 INFO streaming_executor.py:88 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=False, actor_locality_enabled=True, verbose_progress=False)
Running 0: 0%| | 0/80 [00:00<?, ?it/s]
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 0%| | 0/80 [00:00<?, ?it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 0%| | 0/80 [00:00<?, ?it/s]
(ReadRange->MapBatches(generate_features)->Write pid=7423) 2023-04-13 16:25:54.395985: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA [repeated 4x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 0%| | 0/80 [00:12<?, ?it/s]
(ReadRange->MapBatches(generate_features)->Write pid=7423) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. [repeated 4x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 0%| | 0/80 [00:12<?, ?it/s]
(ReadRange->MapBatches(generate_features)->Write pid=3652) 2023-04-13 16:20:55.354167: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 [repeated 8x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 0%| | 0/80 [00:12<?, ?it/s]
(ReadRange->MapBatches(generate_features)->Write pid=3652) 2023-04-13 16:20:55.354178: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. [repeated 4x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 0%| | 0/80 [00:12<?, ?it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 1%|▏ | 1/80 [00:39<51:52, 39.40s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 2%|▎ | 2/80 [00:39<51:13, 39.40s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 2%|▎ | 2/80 [00:39<51:13, 39.40s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 4%|▍ | 3/80 [00:39<13:09, 10.26s/it]
Running: 11.0/16.0 CPU, 0.0/0.0 GPU, 11.0 MiB/4.29 GiB object_store_memory: 5%|▌ | 4/80 [00:39<12:59, 10.26s/it]
Running: 11.0/16.0 CPU, 0.0/0.0 GPU, 11.0 MiB/4.29 GiB object_store_memory: 6%|▋ | 5/80 [00:39<06:17, 5.04s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 11%|█▏ | 9/80 [00:39<05:57, 5.04s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 11%|█▏ | 9/80 [00:39<05:57, 5.04s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 12%|█▎ | 10/80 [00:39<02:06, 1.80s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 12%|█▎ | 10/80 [00:40<02:06, 1.80s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 12%|█▎ | 10/80 [00:40<02:06, 1.80s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 14%|█▍ | 11/80 [00:40<02:04, 1.80s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 15%|█▌ | 12/80 [00:40<01:32, 1.36s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 16%|█▋ | 13/80 [00:40<01:31, 1.36s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 18%|█▊ | 14/80 [00:43<01:38, 1.49s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 18%|█▊ | 14/80 [00:43<01:38, 1.49s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 18%|█▊ | 14/80 [00:44<01:38, 1.49s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 18%|█▊ | 14/80 [00:44<01:38, 1.49s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 19%|█▉ | 15/80 [00:44<01:23, 1.28s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 20%|██ | 16/80 [00:44<01:07, 1.05s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 20%|██ | 16/80 [00:44<01:07, 1.05s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 21%|██▏ | 17/80 [01:18<08:39, 8.24s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 21%|██▏ | 17/80 [01:18<08:39, 8.24s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 21%|██▏ | 17/80 [01:18<08:39, 8.24s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 22%|██▎ | 18/80 [01:18<06:32, 6.33s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 24%|██▍ | 19/80 [01:18<06:26, 6.33s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 24%|██▍ | 19/80 [01:19<06:26, 6.33s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 25%|██▌ | 20/80 [01:19<03:49, 3.83s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 26%|██▋ | 21/80 [01:19<03:46, 3.83s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 28%|██▊ | 22/80 [01:19<02:22, 2.45s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 28%|██▊ | 22/80 [01:19<02:22, 2.45s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 30%|███ | 24/80 [01:19<01:31, 1.63s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 30%|███ | 24/80 [01:19<01:31, 1.63s/it]
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 12.0 MiB/4.29 GiB object_store_memory: 30%|███ | 24/80 [01:19<01:31, 1.63s/it]
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 12.0 MiB/4.29 GiB object_store_memory: 32%|███▎ | 26/80 [01:19<01:01, 1.15s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 35%|███▌ | 28/80 [01:19<00:59, 1.15s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 35%|███▌ | 28/80 [01:19<00:59, 1.15s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 36%|███▋ | 29/80 [01:19<00:35, 1.42it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 36%|███▋ | 29/80 [01:19<00:35, 1.42it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 36%|███▋ | 29/80 [01:22<00:35, 1.42it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 38%|███▊ | 30/80 [01:22<00:35, 1.42it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 38%|███▊ | 30/80 [01:22<00:35, 1.42it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 39%|███▉ | 31/80 [01:22<00:46, 1.05it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 40%|████ | 32/80 [01:23<00:38, 1.23it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 40%|████ | 32/80 [01:23<00:38, 1.23it/s]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 40%|████ | 32/80 [01:58<00:38, 1.23it/s]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 41%|████▏ | 33/80 [01:58<05:49, 7.44s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 44%|████▍ | 35/80 [01:58<03:36, 4.81s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 45%|████▌ | 36/80 [01:58<03:31, 4.81s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 45%|████▌ | 36/80 [01:58<03:31, 4.81s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 46%|████▋ | 37/80 [01:58<02:18, 3.23s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 49%|████▉ | 39/80 [01:58<01:30, 2.20s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 50%|█████ | 40/80 [01:58<01:27, 2.20s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 50%|█████ | 40/80 [01:58<01:27, 2.20s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 51%|█████▏ | 41/80 [01:58<00:59, 1.53s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 52%|█████▎ | 42/80 [01:58<00:58, 1.53s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 54%|█████▍ | 43/80 [01:58<00:39, 1.07s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 56%|█████▋ | 45/80 [01:59<00:27, 1.29it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 56%|█████▋ | 45/80 [01:59<00:27, 1.29it/s]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 56%|█████▋ | 45/80 [02:01<00:27, 1.29it/s]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 59%|█████▉ | 47/80 [02:01<00:31, 1.05it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 59%|█████▉ | 47/80 [02:01<00:31, 1.05it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 59%|█████▉ | 47/80 [02:02<00:31, 1.05it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 60%|██████ | 48/80 [02:02<00:27, 1.17it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 60%|██████ | 48/80 [02:02<00:27, 1.17it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 61%|██████▏ | 49/80 [02:37<04:06, 7.96s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 61%|██████▏ | 49/80 [02:37<04:06, 7.96s/it]
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 12.0 MiB/4.29 GiB object_store_memory: 61%|██████▏ | 49/80 [02:38<04:06, 7.96s/it]
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 12.0 MiB/4.29 GiB object_store_memory: 62%|██████▎ | 50/80 [02:38<03:05, 6.19s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 66%|██████▋ | 53/80 [02:38<02:47, 6.19s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 68%|██████▊ | 54/80 [02:38<01:10, 2.72s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 69%|██████▉ | 55/80 [02:38<01:07, 2.72s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 70%|███████ | 56/80 [02:38<00:46, 1.95s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 72%|███████▎ | 58/80 [02:38<00:31, 1.42s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 74%|███████▍ | 59/80 [02:38<00:29, 1.42s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 75%|███████▌ | 60/80 [02:38<00:20, 1.03s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 76%|███████▋ | 61/80 [02:38<00:19, 1.03s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 76%|███████▋ | 61/80 [02:41<00:19, 1.03s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 78%|███████▊ | 62/80 [02:41<00:19, 1.09s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 79%|███████▉ | 63/80 [02:41<00:15, 1.09it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 79%|███████▉ | 63/80 [02:41<00:15, 1.09it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 79%|███████▉ | 63/80 [02:42<00:15, 1.09it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 80%|████████ | 64/80 [02:42<00:14, 1.13it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 80%|████████ | 64/80 [02:42<00:14, 1.13it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.29 GiB object_store_memory: 81%|████████▏ | 65/80 [03:16<02:06, 8.42s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.29 GiB object_store_memory: 81%|████████▏ | 65/80 [03:16<02:06, 8.42s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 81%|████████▏ | 65/80 [03:17<02:06, 8.42s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.29 GiB object_store_memory: 82%|████████▎ | 66/80 [03:17<01:30, 6.47s/it]
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.29 GiB object_store_memory: 82%|████████▎ | 66/80 [03:17<01:30, 6.47s/it]
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.29 GiB object_store_memory: 84%|████████▍ | 67/80 [03:17<01:03, 4.87s/it]
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 12.0 MiB/4.29 GiB object_store_memory: 84%|████████▍ | 67/80 [03:17<01:03, 4.87s/it]
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 12.0 MiB/4.29 GiB object_store_memory: 85%|████████▌ | 68/80 [03:17<00:43, 3.59s/it]
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 9.0 MiB/4.29 GiB object_store_memory: 85%|████████▌ | 68/80 [03:17<00:43, 3.59s/it]
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 9.0 MiB/4.29 GiB object_store_memory: 86%|████████▋ | 69/80 [03:17<00:28, 2.63s/it]
Running: 8.0/16.0 CPU, 0.0/0.0 GPU, 8.0 MiB/4.29 GiB object_store_memory: 89%|████████▉ | 71/80 [03:17<00:23, 2.63s/it]
Running: 8.0/16.0 CPU, 0.0/0.0 GPU, 8.0 MiB/4.29 GiB object_store_memory: 90%|█████████ | 72/80 [03:17<00:09, 1.22s/it]
Running: 7.0/16.0 CPU, 0.0/0.0 GPU, 7.0 MiB/4.29 GiB object_store_memory: 90%|█████████ | 72/80 [03:17<00:09, 1.22s/it]
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 5.0 MiB/4.29 GiB object_store_memory: 91%|█████████▏| 73/80 [03:17<00:08, 1.22s/it]
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 5.0 MiB/4.29 GiB object_store_memory: 92%|█████████▎| 74/80 [03:17<00:05, 1.19it/s]
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 94%|█████████▍| 75/80 [03:18<00:04, 1.19it/s]
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 95%|█████████▌| 76/80 [03:18<00:02, 1.67it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.0 MiB/4.29 GiB object_store_memory: 95%|█████████▌| 76/80 [03:18<00:02, 1.67it/s]
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 1.0 MiB/4.29 GiB object_store_memory: 96%|█████████▋| 77/80 [03:19<00:01, 1.67it/s]
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 1.0 MiB/4.29 GiB object_store_memory: 98%|█████████▊| 78/80 [03:19<00:01, 1.76it/s]
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 99%|█████████▉| 79/80 [03:19<00:00, 1.76it/s]
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 100%|██████████| 80/80 [03:19<00:00, 2.08it/s]
2023-04-13 16:29:01,700 INFO streaming_executor.py:87 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[ReadRange->MapBatches(generate_features)->Write]
2023-04-13 16:29:01,700 INFO streaming_executor.py:88 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=False, actor_locality_enabled=True, verbose_progress=False)
Running 0: 0%| | 0/32 [00:00<?, ?it/s]
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 0%| | 0/32 [00:00<?, ?it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 0%| | 0/32 [00:00<?, ?it/s]
(ReadRange->MapBatches(generate_features)->Write pid=9864) 2023-04-13 16:29:10.396962: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA [repeated 3x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 0%| | 0/32 [00:08<?, ?it/s]
(ReadRange->MapBatches(generate_features)->Write pid=9864) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. [repeated 3x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 0%| | 0/32 [00:08<?, ?it/s]
(ReadRange->MapBatches(generate_features)->Write pid=7425) 2023-04-13 16:25:56.075771: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 [repeated 6x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 0%| | 0/32 [00:08<?, ?it/s]
(ReadRange->MapBatches(generate_features)->Write pid=7425) 2023-04-13 16:25:56.075783: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. [repeated 3x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 0%| | 0/32 [00:08<?, ?it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 3%|▎ | 1/32 [00:11<06:02, 11.70s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.29 GiB object_store_memory: 6%|▋ | 2/32 [00:11<05:50, 11.70s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 6%|▋ | 2/32 [00:11<05:50, 11.70s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.29 GiB object_store_memory: 6%|▋ | 2/32 [00:11<05:50, 11.70s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.29 GiB object_store_memory: 9%|▉ | 3/32 [00:11<01:30, 3.11s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 12%|█▎ | 4/32 [00:12<01:27, 3.11s/it]
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 3.0 MiB/4.29 GiB object_store_memory: 12%|█▎ | 4/32 [00:12<01:27, 3.11s/it]
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 3.0 MiB/4.29 GiB object_store_memory: 16%|█▌ | 5/32 [00:12<00:42, 1.57s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 25%|██▌ | 8/32 [00:12<00:37, 1.57s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.29 GiB object_store_memory: 25%|██▌ | 8/32 [00:12<00:37, 1.57s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.29 GiB object_store_memory: 28%|██▊ | 9/32 [00:12<00:15, 1.50it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 28%|██▊ | 9/32 [00:12<00:15, 1.50it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.29 GiB object_store_memory: 28%|██▊ | 9/32 [00:12<00:15, 1.50it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.29 GiB object_store_memory: 34%|███▍ | 11/32 [00:12<00:11, 1.88it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 34%|███▍ | 11/32 [00:12<00:11, 1.88it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.29 GiB object_store_memory: 34%|███▍ | 11/32 [00:12<00:11, 1.88it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.29 GiB object_store_memory: 38%|███▊ | 12/32 [00:12<00:09, 2.10it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 38%|███▊ | 12/32 [00:13<00:09, 2.10it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.29 GiB object_store_memory: 38%|███▊ | 12/32 [00:13<00:09, 2.10it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.29 GiB object_store_memory: 41%|████ | 13/32 [00:13<00:10, 1.89it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 41%|████ | 13/32 [00:13<00:10, 1.89it/s]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.29 GiB object_store_memory: 41%|████ | 13/32 [00:16<00:10, 1.89it/s]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.29 GiB object_store_memory: 44%|████▍ | 14/32 [00:16<00:19, 1.11s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 47%|████▋ | 15/32 [00:16<00:18, 1.11s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.29 GiB object_store_memory: 47%|████▋ | 15/32 [00:17<00:18, 1.11s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.29 GiB object_store_memory: 50%|█████ | 16/32 [00:17<00:11, 1.35it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.29 GiB object_store_memory: 50%|█████ | 16/32 [00:17<00:11, 1.35it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.29 GiB object_store_memory: 50%|█████ | 16/32 [00:23<00:11, 1.35it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.29 GiB object_store_memory: 53%|█████▎ | 17/32 [00:23<00:29, 1.94s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.29 GiB object_store_memory: 53%|█████▎ | 17/32 [00:23<00:29, 1.94s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.29 GiB object_store_memory: 56%|█████▋ | 18/32 [00:23<00:21, 1.50s/it]
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 3.25 MiB/4.29 GiB object_store_memory: 56%|█████▋ | 18/32 [00:23<00:21, 1.50s/it]
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 3.25 MiB/4.29 GiB object_store_memory: 59%|█████▉ | 19/32 [00:23<00:14, 1.15s/it]
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 3.0 MiB/4.29 GiB object_store_memory: 59%|█████▉ | 19/32 [00:23<00:14, 1.15s/it]
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 3.0 MiB/4.29 GiB object_store_memory: 62%|██████▎ | 20/32 [00:23<00:10, 1.15it/s]
Running: 10.0/16.0 CPU, 0.0/0.0 GPU, 2.5 MiB/4.29 GiB object_store_memory: 62%|██████▎ | 20/32 [00:23<00:10, 1.15it/s]
Running: 10.0/16.0 CPU, 0.0/0.0 GPU, 2.5 MiB/4.29 GiB object_store_memory: 66%|██████▌ | 21/32 [00:23<00:07, 1.52it/s]
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 2.25 MiB/4.29 GiB object_store_memory: 69%|██████▉ | 22/32 [00:23<00:06, 1.52it/s]
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 2.25 MiB/4.29 GiB object_store_memory: 72%|███████▏ | 23/32 [00:23<00:03, 2.55it/s]
Running: 8.0/16.0 CPU, 0.0/0.0 GPU, 2.0 MiB/4.29 GiB object_store_memory: 72%|███████▏ | 23/32 [00:24<00:03, 2.55it/s]
Running: 7.0/16.0 CPU, 0.0/0.0 GPU, 1.75 MiB/4.29 GiB object_store_memory: 75%|███████▌ | 24/32 [00:24<00:03, 2.55it/s]
Running: 7.0/16.0 CPU, 0.0/0.0 GPU, 1.75 MiB/4.29 GiB object_store_memory: 78%|███████▊ | 25/32 [00:24<00:01, 3.58it/s]
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 1.25 MiB/4.29 GiB object_store_memory: 78%|███████▊ | 25/32 [00:24<00:01, 3.58it/s]
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 1.25 MiB/4.29 GiB object_store_memory: 84%|████████▍ | 27/32 [00:24<00:01, 3.97it/s]
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 1.0 MiB/4.29 GiB object_store_memory: 84%|████████▍ | 27/32 [00:24<00:01, 3.97it/s]
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 1.0 MiB/4.29 GiB object_store_memory: 88%|████████▊ | 28/32 [00:24<00:01, 3.82it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 0.75 MiB/4.29 GiB object_store_memory: 88%|████████▊ | 28/32 [00:25<00:01, 3.82it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 0.75 MiB/4.29 GiB object_store_memory: 91%|█████████ | 29/32 [00:25<00:00, 3.41it/s]
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 0.25 MiB/4.29 GiB object_store_memory: 91%|█████████ | 29/32 [00:26<00:00, 3.41it/s]
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 0.25 MiB/4.29 GiB object_store_memory: 94%|█████████▍| 30/32 [00:26<00:00, 2.04it/s]
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.29 GiB object_store_memory: 100%|██████████| 32/32 [00:26<00:00, 2.04it/s]
Running case: tfrecords-images-100-256
Read progress 0: 0%| | 0/32 [00:00<?, ?it/s]
Result of case tfrecords-images-100-256: {'time': 0.06479635800008055}
Running case: tfrecords-images-100-2048
Read progress 0: 0%| | 0/100 [00:00<?, ?it/s]
Read progress 0: 17%|█▋ | 17/100 [00:00<00:00, 160.91it/s]
Read progress 0: 34%|███▍ | 34/100 [00:00<00:00, 102.90it/s]
Read progress 0: 46%|████▌ | 46/100 [00:00<00:00, 101.94it/s]
Read progress 0: 57%|█████▋ | 57/100 [00:00<00:00, 89.00it/s]
Read progress 0: 67%|██████▋ | 67/100 [00:00<00:00, 89.29it/s]
Read progress 0: 77%|███████▋ | 77/100 [00:00<00:00, 91.52it/s]
Read progress 0: 92%|█████████▏| 92/100 [00:00<00:00, 103.56it/s]
(_execute_read_task_split pid=10387) 2023-04-13 16:29:29.737974: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA [repeated 3x across cluster]
Read progress 0: 97%|█████████▋| 97/100 [00:01<00:00, 103.56it/s]
(_execute_read_task_split pid=10387) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. [repeated 3x across cluster]
Read progress 0: 97%|█████████▋| 97/100 [00:01<00:00, 103.56it/s]
(ReadRange->MapBatches(generate_features)->Write pid=9863) 2023-04-13 16:29:11.921215: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 [repeated 6x across cluster]
Read progress 0: 97%|█████████▋| 97/100 [00:01<00:00, 103.56it/s]
(ReadRange->MapBatches(generate_features)->Write pid=9863) 2023-04-13 16:29:11.921228: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. [repeated 3x across cluster]
Read progress 0: 97%|█████████▋| 97/100 [00:01<00:00, 103.56it/s]
Result of case tfrecords-images-100-2048: {'time': 3.71467114699999}
Running case: tfrecords-images-1000-mix
Read progress 0: 0%| | 0/32 [00:00<?, ?it/s]
Result of case tfrecords-images-1000-mix: {'time': 0.05040225200002624}
Running case: tfrecords-random-int-1g
Read progress 0: 0%| | 0/80 [00:00<?, ?it/s]
(_execute_read_task_split pid=11388) 2023-04-13 16:29:33.910067: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA [repeated 5x across cluster]
Read progress 0: 0%| | 0/80 [00:03<?, ?it/s]
(_execute_read_task_split pid=11388) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. [repeated 5x across cluster]
Read progress 0: 0%| | 0/80 [00:03<?, ?it/s]
(_execute_read_task_split pid=11387) 2023-04-13 16:29:35.346885: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 [repeated 8x across cluster]
Read progress 0: 0%| | 0/80 [00:03<?, ?it/s]
(_execute_read_task_split pid=11387) 2023-04-13 16:29:35.346899: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. [repeated 4x across cluster]
Read progress 0: 0%| | 0/80 [00:03<?, ?it/s]
Read progress 0: 1%|▏ | 1/80 [00:54<1:11:15, 54.12s/it]
Read progress 0: 5%|▌ | 4/80 [00:54<13:09, 10.38s/it]
Read progress 0: 6%|▋ | 5/80 [00:54<09:24, 7.53s/it]
Read progress 0: 9%|▉ | 7/80 [00:55<05:11, 4.26s/it]
Read progress 0: 12%|█▎ | 10/80 [00:55<02:35, 2.22s/it]
Read progress 0: 15%|█▌ | 12/80 [00:55<01:45, 1.55s/it]
Read progress 0: 18%|█▊ | 14/80 [00:58<01:44, 1.58s/it]
Read progress 0: 19%|█▉ | 15/80 [00:59<01:26, 1.33s/it]
Read progress 0: 21%|██▏ | 17/80 [01:47<09:34, 9.11s/it]
Read progress 0: 22%|██▎ | 18/80 [01:47<07:36, 7.36s/it]
Read progress 0: 25%|██▌ | 20/80 [01:47<04:44, 4.74s/it]
Read progress 0: 26%|██▋ | 21/80 [01:47<03:44, 3.81s/it]
Read progress 0: 28%|██▊ | 22/80 [01:48<02:58, 3.08s/it]
Read progress 0: 29%|██▉ | 23/80 [01:48<02:19, 2.44s/it]
Read progress 0: 30%|███ | 24/80 [01:49<01:45, 1.88s/it]
Read progress 0: 32%|███▎ | 26/80 [01:49<00:59, 1.11s/it]
Read progress 0: 35%|███▌ | 28/80 [01:49<00:39, 1.33it/s]
Read progress 0: 36%|███▋ | 29/80 [01:50<00:37, 1.38it/s]
Read progress 0: 38%|███▊ | 30/80 [01:51<00:43, 1.15it/s]
Read progress 0: 39%|███▉ | 31/80 [01:52<00:41, 1.17it/s]
Read progress 0: 41%|████▏ | 33/80 [02:40<08:13, 10.49s/it]
Read progress 0: 44%|████▍ | 35/80 [02:40<04:58, 6.63s/it]
Read progress 0: 45%|████▌ | 36/80 [02:41<03:58, 5.43s/it]
Read progress 0: 46%|████▋ | 37/80 [02:41<03:02, 4.25s/it]
Read progress 0: 48%|████▊ | 38/80 [02:41<02:15, 3.23s/it]
Read progress 0: 49%|████▉ | 39/80 [02:42<01:40, 2.44s/it]
Read progress 0: 50%|█████ | 40/80 [02:42<01:17, 1.93s/it]
Read progress 0: 51%|█████▏ | 41/80 [02:43<01:00, 1.56s/it]
Read progress 0: 54%|█████▍ | 43/80 [02:43<00:33, 1.10it/s]
Read progress 0: 55%|█████▌ | 44/80 [02:43<00:25, 1.39it/s]
Read progress 0: 57%|█████▊ | 46/80 [02:44<00:21, 1.58it/s]
Read progress 0: 59%|█████▉ | 47/80 [02:45<00:19, 1.65it/s]
Read progress 0: 60%|██████ | 48/80 [02:45<00:19, 1.65it/s]
Read progress 0: 61%|██████▏ | 49/80 [03:33<06:33, 12.68s/it]
Read progress 0: 62%|██████▎ | 50/80 [03:33<04:38, 9.30s/it]
Read progress 0: 64%|██████▍ | 51/80 [03:33<03:18, 6.83s/it]
Read progress 0: 65%|██████▌ | 52/80 [03:34<02:25, 5.20s/it]
Read progress 0: 68%|██████▊ | 54/80 [03:35<01:17, 3.00s/it]
Read progress 0: 69%|██████▉ | 55/80 [03:36<01:01, 2.45s/it]
Read progress 0: 70%|███████ | 56/80 [03:36<00:44, 1.85s/it]
Read progress 0: 71%|███████▏ | 57/80 [03:36<00:34, 1.51s/it]
Read progress 0: 72%|███████▎ | 58/80 [03:37<00:25, 1.18s/it]
Read progress 0: 74%|███████▍ | 59/80 [03:37<00:18, 1.12it/s]
Read progress 0: 75%|███████▌ | 60/80 [03:37<00:13, 1.49it/s]
Read progress 0: 79%|███████▉ | 63/80 [03:38<00:07, 2.25it/s]
Read progress 0: 80%|████████ | 64/80 [03:38<00:07, 2.23it/s]
Read progress 0: 81%|████████▏ | 65/80 [04:26<02:48, 11.25s/it]
Read progress 0: 84%|████████▍ | 67/80 [04:26<01:29, 6.85s/it]
Read progress 0: 85%|████████▌ | 68/80 [04:28<01:08, 5.70s/it]
Read progress 0: 89%|████████▉ | 71/80 [04:29<00:28, 3.14s/it]
Read progress 0: 91%|█████████▏| 73/80 [04:29<00:15, 2.17s/it]
Read progress 0: 92%|█████████▎| 74/80 [04:29<00:10, 1.83s/it]
Read progress 0: 94%|█████████▍| 75/80 [04:30<00:07, 1.54s/it]
Read progress 0: 95%|█████████▌| 76/80 [04:30<00:04, 1.22s/it]
Read progress 0: 99%|█████████▉| 79/80 [04:30<00:00, 1.58it/s]
Result of case tfrecords-random-int-1g: {'time': 270.63967487499997}
Running case: tfrecords-random-float-1g
Read progress 0: 0%| | 0/80 [00:00<?, ?it/s]
(_execute_read_task_split pid=14580) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Read progress 0: 0%| | 0/80 [00:01<?, ?it/s]
(_execute_read_task_split pid=14580) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Read progress 0: 0%| | 0/80 [00:01<?, ?it/s]
(_execute_read_task_split pid=11388) 2023-04-13 16:29:35.405076: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 [repeated 4x across cluster]
Read progress 0: 0%| | 0/80 [00:01<?, ?it/s]
(_execute_read_task_split pid=11388) 2023-04-13 16:29:35.405089: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. [repeated 2x across cluster]
Read progress 0: 0%| | 0/80 [00:01<?, ?it/s]
(_execute_read_task_split pid=14579) 2023-04-13 16:34:04.571833: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
Read progress 0: 0%| | 0/80 [00:01<?, ?it/s]
(_execute_read_task_split pid=14579) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Read progress 0: 0%| | 0/80 [00:01<?, ?it/s]
Read progress 0: 1%|▏ | 1/80 [00:50<1:06:59, 50.88s/it]
Read progress 0: 2%|▎ | 2/80 [00:51<28:00, 21.55s/it]
Read progress 0: 6%|▋ | 5/80 [00:52<07:43, 6.17s/it]
Read progress 0: 9%|▉ | 7/80 [00:52<04:30, 3.71s/it]
Read progress 0: 11%|█▏ | 9/80 [00:52<02:48, 2.38s/it]
Read progress 0: 15%|█▌ | 12/80 [00:52<01:33, 1.38s/it]
Read progress 0: 18%|█▊ | 14/80 [00:57<01:47, 1.63s/it]
Read progress 0: 19%|█▉ | 15/80 [00:57<01:35, 1.47s/it]
Read progress 0: 20%|██ | 16/80 [00:58<01:23, 1.31s/it]
Read progress 0: 21%|██▏ | 17/80 [01:42<11:19, 10.78s/it]
Read progress 0: 22%|██▎ | 18/80 [01:43<08:39, 8.37s/it]
Read progress 0: 24%|██▍ | 19/80 [01:43<06:22, 6.28s/it]
Read progress 0: 28%|██▊ | 22/80 [01:43<02:55, 3.02s/it]
Read progress 0: 29%|██▉ | 23/80 [01:44<02:20, 2.47s/it]
Read progress 0: 31%|███▏ | 25/80 [01:44<01:28, 1.62s/it]
Read progress 0: 32%|███▎ | 26/80 [01:44<01:10, 1.31s/it]
Read progress 0: 35%|███▌ | 28/80 [01:44<00:44, 1.16it/s]
Read progress 0: 36%|███▋ | 29/80 [01:44<00:36, 1.39it/s]
Read progress 0: 38%|███▊ | 30/80 [01:48<01:09, 1.39s/it]
Read progress 0: 39%|███▉ | 31/80 [01:48<00:52, 1.08s/it]
Read progress 0: 40%|████ | 32/80 [01:49<00:52, 1.10s/it]
Read progress 0: 41%|████▏ | 33/80 [02:33<10:01, 12.80s/it]
Read progress 0: 42%|████▎ | 34/80 [02:35<07:22, 9.62s/it]
Read progress 0: 45%|████▌ | 36/80 [02:35<03:58, 5.42s/it]
Read progress 0: 48%|████▊ | 38/80 [02:35<02:21, 3.36s/it]
Read progress 0: 51%|█████▏ | 41/80 [02:36<01:13, 1.88s/it]
Read progress 0: 52%|█████▎ | 42/80 [02:36<01:01, 1.61s/it]
Read progress 0: 54%|█████▍ | 43/80 [02:36<00:49, 1.35s/it]
Read progress 0: 55%|█████▌ | 44/80 [02:36<00:39, 1.09s/it]
Read progress 0: 56%|█████▋ | 45/80 [02:37<00:31, 1.12it/s]
Read progress 0: 57%|█████▊ | 46/80 [02:39<00:45, 1.33s/it]
Read progress 0: 60%|██████ | 48/80 [02:41<00:34, 1.09s/it]
Read progress 0: 61%|██████▏ | 49/80 [03:24<05:39, 10.96s/it]
Read progress 0: 62%|██████▎ | 50/80 [03:26<04:17, 8.59s/it]
Read progress 0: 64%|██████▍ | 51/80 [03:27<03:09, 6.53s/it]
Read progress 0: 65%|██████▌ | 52/80 [03:27<02:13, 4.78s/it]
Read progress 0: 66%|██████▋ | 53/80 [03:27<01:33, 3.48s/it]
Read progress 0: 69%|██████▉ | 55/80 [03:27<00:50, 2.03s/it]
Read progress 0: 70%|███████ | 56/80 [03:28<00:37, 1.58s/it]
Read progress 0: 71%|███████▏ | 57/80 [03:28<00:28, 1.26s/it]
Read progress 0: 72%|███████▎ | 58/80 [03:28<00:21, 1.05it/s]
Read progress 0: 74%|███████▍ | 59/80 [03:28<00:15, 1.34it/s]
Read progress 0: 75%|███████▌ | 60/80 [03:28<00:11, 1.70it/s]
Read progress 0: 76%|███████▋ | 61/80 [03:29<00:10, 1.76it/s]
Read progress 0: 78%|███████▊ | 62/80 [03:30<00:13, 1.35it/s]
Read progress 0: 79%|███████▉ | 63/80 [03:31<00:14, 1.20it/s]
Read progress 0: 80%|████████ | 64/80 [03:32<00:14, 1.12it/s]
Read progress 0: 81%|████████▏ | 65/80 [04:17<03:30, 14.00s/it]
Read progress 0: 82%|████████▎ | 66/80 [04:18<02:19, 9.95s/it]
Read progress 0: 84%|████████▍ | 67/80 [04:18<01:31, 7.05s/it]
Read progress 0: 85%|████████▌ | 68/80 [04:19<01:02, 5.19s/it]
Read progress 0: 86%|████████▋ | 69/80 [04:19<00:42, 3.84s/it]
Read progress 0: 90%|█████████ | 72/80 [04:19<00:13, 1.72s/it]
Read progress 0: 94%|█████████▍| 75/80 [04:20<00:04, 1.01it/s]
Read progress 0: 95%|█████████▌| 76/80 [04:20<00:03, 1.16it/s]
Read progress 0: 96%|█████████▋| 77/80 [04:21<00:02, 1.25it/s]
Read progress 0: 98%|█████████▊| 78/80 [04:21<00:01, 1.31it/s]
Read progress 0: 99%|█████████▉| 79/80 [04:21<00:00, 1.59it/s]
Read progress 0: 100%|██████████| 80/80 [04:22<00:00, 1.75it/s]
Result of case tfrecords-random-float-1g: {'time': 262.41669915}
Running case: tfrecords-random-bytes-1g
Read progress 0: 0%| | 0/32 [00:00<?, ?it/s]
(_execute_read_task_split pid=14580) 2023-04-13 16:34:06.195826: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 [repeated 6x across cluster]
Read progress 0: 0%| | 0/32 [00:01<?, ?it/s]
(_execute_read_task_split pid=14580) 2023-04-13 16:34:06.195839: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. [repeated 3x across cluster]
Read progress 0: 0%| | 0/32 [00:01<?, ?it/s]
(_execute_read_task_split pid=17699) 2023-04-13 16:38:26.976407: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA [repeated 2x across cluster]
Read progress 0: 0%| | 0/32 [00:01<?, ?it/s]
(_execute_read_task_split pid=17699) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. [repeated 2x across cluster]
Read progress 0: 0%| | 0/32 [00:01<?, ?it/s]
Read progress 0: 3%|▎ | 1/32 [00:15<08:08, 15.76s/it]
Read progress 0: 6%|▋ | 2/32 [00:15<03:17, 6.58s/it]
Read progress 0: 9%|▉ | 3/32 [00:16<01:48, 3.74s/it]
Read progress 0: 12%|█▎ | 4/32 [00:16<01:05, 2.33s/it]
Read progress 0: 16%|█▌ | 5/32 [00:16<00:41, 1.53s/it]
Read progress 0: 28%|██▊ | 9/32 [00:16<00:12, 1.83it/s]
Read progress 0: 38%|███▊ | 12/32 [00:16<00:06, 2.96it/s]
Read progress 0: 44%|████▍ | 14/32 [00:21<00:16, 1.11it/s]
Read progress 0: 47%|████▋ | 15/32 [00:21<00:13, 1.29it/s]
Read progress 0: 50%|█████ | 16/32 [00:22<00:10, 1.49it/s]
Read progress 0: 53%|█████▎ | 17/32 [00:32<00:41, 2.79s/it]
Read progress 0: 59%|█████▉ | 19/32 [00:32<00:22, 1.75s/it]
Read progress 0: 62%|██████▎ | 20/32 [00:32<00:16, 1.40s/it]
Read progress 0: 69%|██████▉ | 22/32 [00:32<00:08, 1.12it/s]
Read progress 0: 72%|███████▏ | 23/32 [00:33<00:06, 1.35it/s]
Read progress 0: 81%|████████▏ | 26/32 [00:33<00:02, 2.40it/s]
Read progress 0: 91%|█████████ | 29/32 [00:33<00:00, 3.82it/s]
Read progress 0: 97%|█████████▋| 31/32 [00:36<00:00, 1.72it/s]
Result of case tfrecords-random-bytes-1g: {'time': 36.31453858500004}
Finish benchmark: read-tfrecords
(_execute_read_task_split pid=17698) 2023-04-13 16:38:28.675456: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 [repeated 6x across cluster]
(_execute_read_task_split pid=17698) 2023-04-13 16:38:28.675470: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. [repeated 3x across cluster]
(_execute_read_task_split pid=17698) 2023-04-13 16:38:27.000738: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA [repeated 2x across cluster]
(_execute_read_task_split pid=17698) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. [repeated 2x across cluster]
Subprocess return code: 0
[INFO 2023-04-13 16:39:05,205] anyscale_job_wrapper.py: 191 Process 1561 exited with return code 0.
[INFO 2023-04-13 16:39:05,206] anyscale_job_wrapper.py: 294 Finished with return code 0. Time taken: 1239.13400811
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
boto3 1.26.112 requires botocore<1.30.0,>=1.29.112, but you have botocore 1.29.105 which is incompatible.
Completed 344 Bytes/344 Bytes (2.6 KiB/s) with 1 file(s) remaining
upload: ../../../../../release_test_out.json to s3://ray-release-automation-results/working_dirs/read_tfrecords_benchmark_single_node/shzzzxvcxu/tmp/release_test_out.json
Completed 374 Bytes/374 Bytes (3.3 KiB/s) with 1 file(s) remaining
upload: ../../../../../metrics_test_out.json to s3://ray-release-automation-results/working_dirs/read_tfrecords_benchmark_single_node/shzzzxvcxu/tmp/metrics_test_out.json
Completed 243 Bytes/243 Bytes (2.3 KiB/s) with 1 file(s) remaining
upload: ./output.json to s3://ray-release-automation-results/working_dirs/read_tfrecords_benchmark_single_node/shzzzxvcxu/tmp/output.json
[INFO 2023-04-13 16:40:00,389] anyscale_job_wrapper.py: 346 ### Finished ###
[INFO 2023-04-13 16:40:00,389] anyscale_job_wrapper.py: 349 ### JSON |{"collected_metrics":true,"last_prepare_time_taken":null,"prepare_return_codes":[],"return_code":0,"total_time_taken":1293.749378899,"uploaded_artifact":false,"uploaded_metrics":true,"uploaded_results":true,"workload_time_taken":1239.13400811}| ###
Raw
gce.txt
[INFO 2023-04-18 22:27:37,533] anyscale_job_wrapper.py: 256 ### Starting ###
[INFO 2023-04-18 22:27:37,533] anyscale_job_wrapper.py: 280 ### Starting entrypoint ###
[INFO 2023-04-18 22:27:37,534] anyscale_job_wrapper.py: 162 Running command python read_tfrecords_benchmark.py
[INFO 2023-04-18 22:27:37,538] anyscale_job_wrapper.py: 176 Starting process 1447.
+ python read_tfrecords_benchmark.py
2023-04-18 22:27:38,567 INFO worker.py:1315 -- Using address 10.138.0.17:6379 set in the environment variable RAY_ADDRESS
2023-04-18 22:27:38,567 INFO worker.py:1432 -- Connecting to existing Ray cluster at address: 10.138.0.17:6379...
2023-04-18 22:27:38,574 INFO worker.py:1613 -- Connected to Ray cluster. View the dashboard at https://console.anyscale-staging.com/api/v2/sessions/ses_g3nbd6p5xypuyvzw83tu6ssnk4/services?redirect_to=dashboard 
Running benchmark: read-tfrecords
2023-04-18 22:27:40,301 INFO streaming_executor.py:87 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[ReadImage->MapBatches(images_to_bytes)->Write]
2023-04-18 22:27:40,302 INFO streaming_executor.py:88 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=False, actor_locality_enabled=True, verbose_progress=False)
2023-04-18 22:27:40,302 INFO streaming_executor.py:91 -- Tip: To enable per-operator progress reporting, set RAY_DATA_VERBOSE_PROGRESS=1.
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 1.98 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:00<?, ?it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=605) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 1.98 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:00<?, ?it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=605) 2023-04-18 22:27:41.035336: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 1.98 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:00<?, ?it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=605) 2023-04-18 22:27:42.763223: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 1.98 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:02<?, ?it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=605) 2023-04-18 22:27:42.763596: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 1.98 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:02<?, ?it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=605) 2023-04-18 22:27:42.763616: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 1.98 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:02<?, ?it/s]
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 0.65 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:04<?, ?it/s]
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 0.65 MiB/4.28 GiB object_store_memory: 3%|▎ | 1/32 [00:04<02:18, 4.48s/it]
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 0.12 MiB/4.28 GiB object_store_memory: 34%|███▍ | 11/32 [00:04<01:33, 4.48s/it]
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 0.12 MiB/4.28 GiB object_store_memory: 38%|███▊ | 12/32 [00:04<00:05, 3.60it/s]
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 0.12 MiB/4.28 GiB object_store_memory: 84%|████████▍ | 27/32 [00:04<00:00, 9.73it/s]
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 0.11 MiB/4.28 GiB object_store_memory: 97%|█████████▋| 31/32 [00:04<00:00, 9.73it/s]
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.28 GiB object_store_memory: 97%|█████████▋| 31/32 [00:04<00:00, 9.73it/s]
2023-04-18 22:29:07,543 INFO streaming_executor.py:87 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[ReadImage->MapBatches(images_to_bytes)->Write]
2023-04-18 22:29:07,543 INFO streaming_executor.py:88 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=False, actor_locality_enabled=True, verbose_progress=False)
2023-04-18 22:29:07,543 INFO streaming_executor.py:91 -- Tip: To enable per-operator progress reporting, set RAY_DATA_VERBOSE_PROGRESS=1.
Running 0: 0%| | 0/100 [00:00<?, ?it/s]
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.28 GiB object_store_memory: 0%| | 0/100 [00:00<?, ?it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 0%| | 0/100 [00:00<?, ?it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 7.21 MiB/4.28 GiB object_store_memory: 0%| | 0/100 [00:00<?, ?it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 7.21 MiB/4.28 GiB object_store_memory: 1%| | 1/100 [00:00<00:33, 2.98it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 13%|█▎ | 13/100 [00:00<00:29, 2.98it/s]
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 12.02 MiB/4.28 GiB object_store_memory: 13%|█▎ | 13/100 [00:00<00:29, 2.98it/s]
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 12.02 MiB/4.28 GiB object_store_memory: 14%|█▍ | 14/100 [00:00<00:03, 24.86it/s]
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 12.02 MiB/4.28 GiB object_store_memory: 25%|██▌ | 25/100 [00:00<00:01, 42.14it/s]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 33.66 MiB/4.28 GiB object_store_memory: 26%|██▌ | 26/100 [00:00<00:01, 42.14it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 26%|██▌ | 26/100 [00:00<00:01, 42.14it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 32%|███▏ | 32/100 [00:00<00:01, 38.22it/s]
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 12.02 MiB/4.28 GiB object_store_memory: 33%|███▎ | 33/100 [00:00<00:01, 38.22it/s]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 33.66 MiB/4.28 GiB object_store_memory: 37%|███▋ | 37/100 [00:01<00:01, 38.22it/s]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 33.66 MiB/4.28 GiB object_store_memory: 38%|███▊ | 38/100 [00:01<00:01, 41.40it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 39%|███▉ | 39/100 [00:01<00:01, 41.40it/s]
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 12.02 MiB/4.28 GiB object_store_memory: 39%|███▉ | 39/100 [00:01<00:01, 41.40it/s]
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 12.02 MiB/4.28 GiB object_store_memory: 44%|████▍ | 44/100 [00:01<00:01, 36.32it/s]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 33.66 MiB/4.28 GiB object_store_memory: 50%|█████ | 50/100 [00:01<00:01, 36.32it/s]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 33.66 MiB/4.28 GiB object_store_memory: 51%|█████ | 51/100 [00:01<00:01, 41.93it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 52%|█████▏ | 52/100 [00:01<00:01, 41.93it/s]
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 12.02 MiB/4.28 GiB object_store_memory: 52%|█████▏ | 52/100 [00:01<00:01, 41.93it/s]
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 12.02 MiB/4.28 GiB object_store_memory: 56%|█████▌ | 56/100 [00:01<00:01, 35.17it/s]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 33.66 MiB/4.28 GiB object_store_memory: 63%|██████▎ | 63/100 [00:01<00:01, 35.17it/s]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 33.66 MiB/4.28 GiB object_store_memory: 64%|██████▍ | 64/100 [00:01<00:00, 42.87it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 65%|██████▌ | 65/100 [00:01<00:00, 42.87it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=2901) 2023-04-18 22:29:09.421447: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA [repeated 16x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/ray-logging.html#log-deduplication for more options.)
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 65%|██████▌ | 65/100 [00:01<00:00, 42.87it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=2901) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. [repeated 16x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 65%|██████▌ | 65/100 [00:01<00:00, 42.87it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=1581) 2023-04-18 22:27:42.496392: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. [repeated 15x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 65%|██████▌ | 65/100 [00:01<00:00, 42.87it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=1581) 2023-04-18 22:27:43.798676: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 [repeated 30x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 65%|██████▌ | 65/100 [00:01<00:00, 42.87it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=1581) 2023-04-18 22:27:43.798692: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. [repeated 15x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 65%|██████▌ | 65/100 [00:01<00:00, 42.87it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 70%|███████ | 70/100 [00:01<00:00, 37.14it/s]
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 12.02 MiB/4.28 GiB object_store_memory: 72%|███████▏ | 72/100 [00:01<00:00, 37.14it/s]
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 12.02 MiB/4.28 GiB object_store_memory: 77%|███████▋ | 77/100 [00:02<00:00, 42.80it/s]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 33.66 MiB/4.28 GiB object_store_memory: 78%|███████▊ | 78/100 [00:02<00:00, 42.80it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 38.47 MiB/4.28 GiB object_store_memory: 78%|███████▊ | 78/100 [00:02<00:00, 42.80it/s]
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 12.02 MiB/4.28 GiB object_store_memory: 78%|███████▊ | 78/100 [00:02<00:00, 42.80it/s]
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 12.02 MiB/4.28 GiB object_store_memory: 82%|████████▏ | 82/100 [00:02<00:00, 35.20it/s]
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 21.64 MiB/4.28 GiB object_store_memory: 89%|████████▉ | 89/100 [00:02<00:00, 35.20it/s]
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 21.64 MiB/4.28 GiB object_store_memory: 90%|█████████ | 90/100 [00:02<00:00, 43.94it/s]
Running: 8.0/16.0 CPU, 0.0/0.0 GPU, 19.23 MiB/4.28 GiB object_store_memory: 92%|█████████▏| 92/100 [00:02<00:00, 43.94it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 7.21 MiB/4.28 GiB object_store_memory: 92%|█████████▏| 92/100 [00:02<00:00, 43.94it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 7.21 MiB/4.28 GiB object_store_memory: 96%|█████████▌| 96/100 [00:02<00:00, 38.06it/s]
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 2.41 MiB/4.28 GiB object_store_memory: 97%|█████████▋| 97/100 [00:04<00:00, 38.06it/s]
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.28 GiB object_store_memory: 99%|█████████▉| 99/100 [00:04<00:00, 38.06it/s]
2023-04-18 22:29:20,463 INFO streaming_executor.py:87 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[ReadImage->MapBatches(images_to_bytes)->Write]
2023-04-18 22:29:20,463 INFO streaming_executor.py:88 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=False, actor_locality_enabled=True, verbose_progress=False)
2023-04-18 22:29:20,463 INFO streaming_executor.py:91 -- Tip: To enable per-operator progress reporting, set RAY_DATA_VERBOSE_PROGRESS=1.
Running 0: 0%| | 0/51 [00:00<?, ?it/s]
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.28 GiB object_store_memory: 0%| | 0/51 [00:00<?, ?it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.78 MiB/4.28 GiB object_store_memory: 0%| | 0/51 [00:00<?, ?it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.28 GiB object_store_memory: 0%| | 0/51 [00:00<?, ?it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.28 GiB object_store_memory: 2%|▏ | 1/51 [00:00<00:11, 4.44it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 18.37 MiB/4.28 GiB object_store_memory: 25%|██▌ | 13/51 [00:00<00:08, 4.44it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.28 GiB object_store_memory: 25%|██▌ | 13/51 [00:00<00:08, 4.44it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.28 GiB object_store_memory: 27%|██▋ | 14/51 [00:00<00:01, 36.54it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 15.42 MiB/4.28 GiB object_store_memory: 51%|█████ | 26/51 [00:00<00:00, 36.54it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.28 GiB object_store_memory: 51%|█████ | 26/51 [00:00<00:00, 36.54it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.28 GiB object_store_memory: 53%|█████▎ | 27/51 [00:00<00:00, 47.83it/s]
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 10.95 MiB/4.28 GiB object_store_memory: 76%|███████▋ | 39/51 [00:00<00:00, 47.83it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.28 GiB object_store_memory: 76%|███████▋ | 39/51 [00:00<00:00, 47.83it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.28 GiB object_store_memory: 78%|███████▊ | 40/51 [00:00<00:00, 52.87it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=3281) 2023-04-18 22:29:21.867708: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA [repeated 3x across cluster]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.28 GiB object_store_memory: 94%|█████████▍| 48/51 [00:01<00:00, 52.87it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=3281) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. [repeated 3x across cluster]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.28 GiB object_store_memory: 94%|█████████▍| 48/51 [00:01<00:00, 52.87it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=2903) 2023-04-18 22:29:09.735154: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. [repeated 3x across cluster]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.28 GiB object_store_memory: 94%|█████████▍| 48/51 [00:01<00:00, 52.87it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=2903) 2023-04-18 22:29:10.622954: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 [repeated 6x across cluster]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.28 GiB object_store_memory: 94%|█████████▍| 48/51 [00:01<00:00, 52.87it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=2903) 2023-04-18 22:29:10.622968: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. [repeated 3x across cluster]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.33 MiB/4.28 GiB object_store_memory: 94%|█████████▍| 48/51 [00:01<00:00, 52.87it/s]
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 0.91 MiB/4.28 GiB object_store_memory: 94%|█████████▍| 48/51 [00:03<00:00, 52.87it/s]
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 0.91 MiB/4.28 GiB object_store_memory: 96%|█████████▌| 49/51 [00:03<00:00, 9.67it/s]
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.28 GiB object_store_memory: 98%|█████████▊| 50/51 [00:03<00:00, 9.67it/s]
2023-04-18 22:29:24,199 INFO streaming_executor.py:87 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[ReadRange->MapBatches(generate_features)->Write]
2023-04-18 22:29:24,199 INFO streaming_executor.py:88 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=False, actor_locality_enabled=True, verbose_progress=False)
2023-04-18 22:29:24,199 INFO streaming_executor.py:91 -- Tip: To enable per-operator progress reporting, set RAY_DATA_VERBOSE_PROGRESS=1.
Running 0: 0%| | 0/80 [00:00<?, ?it/s]
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:00<?, ?it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:00<?, ?it/s]
(ReadRange->MapBatches(generate_features)->Write pid=3279) 2023-04-18 22:29:55.662969: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA [repeated 3x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:31<?, ?it/s]
(ReadRange->MapBatches(generate_features)->Write pid=3279) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. [repeated 3x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:31<?, ?it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=3280) 2023-04-18 22:29:22.092111: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. [repeated 3x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:31<?, ?it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=3280) 2023-04-18 22:29:22.951696: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 [repeated 6x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:31<?, ?it/s]
(ReadImage->MapBatches(images_to_bytes)->Write pid=3280) 2023-04-18 22:29:22.951704: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. [repeated 3x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:31<?, ?it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [01:01<?, ?it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 1%|▏ | 1/80 [01:01<1:21:17, 61.74s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 1%|▏ | 1/80 [01:01<1:21:17, 61.74s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 1%|▏ | 1/80 [01:02<1:21:17, 61.74s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 2%|▎ | 2/80 [01:02<33:17, 25.61s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 4%|▍ | 3/80 [01:02<32:52, 25.61s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 5%|▌ | 4/80 [01:02<12:08, 9.59s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 8%|▊ | 6/80 [01:02<06:17, 5.11s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 9%|▉ | 7/80 [01:02<06:12, 5.11s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 9%|▉ | 7/80 [01:02<06:12, 5.11s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 10%|█ | 8/80 [01:02<03:41, 3.07s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 11%|█▏ | 9/80 [01:02<03:38, 3.07s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 12%|█▎ | 10/80 [01:02<02:19, 1.99s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 14%|█▍ | 11/80 [01:02<02:17, 1.99s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 15%|█▌ | 12/80 [01:03<01:32, 1.36s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 15%|█▌ | 12/80 [01:03<01:32, 1.36s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 15%|█▌ | 12/80 [01:03<01:32, 1.36s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 15%|█▌ | 12/80 [01:06<01:32, 1.36s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 16%|█▋ | 13/80 [01:06<01:31, 1.36s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 16%|█▋ | 13/80 [01:06<01:31, 1.36s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 18%|█▊ | 14/80 [01:06<01:43, 1.57s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 18%|█▊ | 14/80 [01:07<01:43, 1.57s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 19%|█▉ | 15/80 [01:07<01:29, 1.37s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 19%|█▉ | 15/80 [01:07<01:29, 1.37s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 19%|█▉ | 15/80 [01:07<01:29, 1.37s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 19%|█▉ | 15/80 [01:07<01:29, 1.37s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 20%|██ | 16/80 [01:07<01:12, 1.13s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 20%|██ | 16/80 [01:07<01:12, 1.13s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 20%|██ | 16/80 [02:03<01:12, 1.13s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 21%|██▏ | 17/80 [02:03<14:33, 13.87s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 21%|██▏ | 17/80 [02:03<14:33, 13.87s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 22%|██▎ | 18/80 [02:03<10:52, 10.53s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 22%|██▎ | 18/80 [02:03<10:52, 10.53s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 24%|██▍ | 19/80 [02:04<07:55, 7.79s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 24%|██▍ | 19/80 [02:04<07:55, 7.79s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 24%|██▍ | 19/80 [02:04<07:55, 7.79s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 25%|██▌ | 20/80 [02:04<05:42, 5.72s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 26%|██▋ | 21/80 [02:04<05:37, 5.72s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 26%|██▋ | 21/80 [02:04<05:37, 5.72s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 28%|██▊ | 22/80 [02:04<03:08, 3.26s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 28%|██▊ | 22/80 [02:04<03:08, 3.26s/it]
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.28 GiB object_store_memory: 28%|██▊ | 22/80 [02:04<03:08, 3.26s/it]
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.28 GiB object_store_memory: 29%|██▉ | 23/80 [02:04<02:25, 2.55s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 31%|███▏ | 25/80 [02:04<02:20, 2.55s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 34%|███▍ | 27/80 [02:04<02:15, 2.55s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 34%|███▍ | 27/80 [02:05<02:15, 2.55s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 35%|███▌ | 28/80 [02:05<00:52, 1.01s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 35%|███▌ | 28/80 [02:05<00:52, 1.01s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 35%|███▌ | 28/80 [02:08<00:52, 1.01s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 36%|███▋ | 29/80 [02:08<01:09, 1.37s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 36%|███▋ | 29/80 [02:08<01:09, 1.37s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 36%|███▋ | 29/80 [02:09<01:09, 1.37s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 38%|███▊ | 30/80 [02:09<01:05, 1.32s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 38%|███▊ | 30/80 [02:09<01:05, 1.32s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 38%|███▊ | 30/80 [02:10<01:05, 1.32s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 39%|███▉ | 31/80 [02:10<00:55, 1.14s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 40%|████ | 32/80 [02:10<00:54, 1.14s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 41%|████▏ | 33/80 [03:04<08:27, 10.79s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 41%|████▏ | 33/80 [03:04<08:27, 10.79s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 42%|████▎ | 34/80 [03:05<06:34, 8.57s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 42%|████▎ | 34/80 [03:05<06:34, 8.57s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 42%|████▎ | 34/80 [03:06<06:34, 8.57s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 44%|████▍ | 35/80 [03:06<05:07, 6.84s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 44%|████▍ | 35/80 [03:06<05:07, 6.84s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 44%|████▍ | 35/80 [03:06<05:07, 6.84s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 45%|████▌ | 36/80 [03:06<03:48, 5.20s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 46%|████▋ | 37/80 [03:06<02:46, 3.87s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 48%|████▊ | 38/80 [03:06<02:42, 3.87s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 49%|████▉ | 39/80 [03:06<01:33, 2.29s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 49%|████▉ | 39/80 [03:06<01:33, 2.29s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 49%|████▉ | 39/80 [03:06<01:33, 2.29s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 50%|█████ | 40/80 [03:06<01:11, 1.78s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 51%|█████▏ | 41/80 [03:06<01:09, 1.78s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 52%|█████▎ | 42/80 [03:06<00:41, 1.09s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 52%|█████▎ | 42/80 [03:07<00:41, 1.09s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 52%|█████▎ | 42/80 [03:07<00:41, 1.09s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 55%|█████▌ | 44/80 [03:07<00:26, 1.34it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 55%|█████▌ | 44/80 [03:07<00:26, 1.34it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 56%|█████▋ | 45/80 [03:10<00:45, 1.31s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 56%|█████▋ | 45/80 [03:10<00:45, 1.31s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 56%|█████▋ | 45/80 [03:10<00:45, 1.31s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 56%|█████▋ | 45/80 [03:11<00:45, 1.31s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 57%|█████▊ | 46/80 [03:11<00:37, 1.11s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 57%|█████▊ | 46/80 [03:11<00:37, 1.11s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 57%|█████▊ | 46/80 [03:11<00:37, 1.11s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 59%|█████▉ | 47/80 [03:11<00:30, 1.09it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 59%|█████▉ | 47/80 [03:11<00:30, 1.09it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 59%|█████▉ | 47/80 [03:11<00:30, 1.09it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 60%|██████ | 48/80 [03:11<00:24, 1.32it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 60%|██████ | 48/80 [03:11<00:24, 1.32it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 60%|██████ | 48/80 [04:06<00:24, 1.32it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 61%|██████▏ | 49/80 [04:06<07:57, 15.40s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 61%|██████▏ | 49/80 [04:06<07:57, 15.40s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 61%|██████▏ | 49/80 [04:06<07:57, 15.40s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 62%|██████▎ | 50/80 [04:06<05:35, 11.20s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 62%|██████▎ | 50/80 [04:07<05:35, 11.20s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 62%|██████▎ | 50/80 [04:07<05:35, 11.20s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 64%|██████▍ | 51/80 [04:07<03:59, 8.25s/it]
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.28 GiB object_store_memory: 64%|██████▍ | 51/80 [04:07<03:59, 8.25s/it]
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.28 GiB object_store_memory: 65%|██████▌ | 52/80 [04:07<02:45, 5.90s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 68%|██████▊ | 54/80 [04:08<02:33, 5.90s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 69%|██████▉ | 55/80 [04:08<01:06, 2.65s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 69%|██████▉ | 55/80 [04:08<01:06, 2.65s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 70%|███████ | 56/80 [04:08<01:03, 2.65s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 70%|███████ | 56/80 [04:08<01:03, 2.65s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 70%|███████ | 56/80 [04:08<01:03, 2.65s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 71%|███████▏ | 57/80 [04:08<00:42, 1.85s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 71%|███████▏ | 57/80 [04:08<00:42, 1.85s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 71%|███████▏ | 57/80 [04:09<00:42, 1.85s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 72%|███████▎ | 58/80 [04:09<00:34, 1.55s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 72%|███████▎ | 58/80 [04:09<00:34, 1.55s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 72%|███████▎ | 58/80 [04:09<00:34, 1.55s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 74%|███████▍ | 59/80 [04:09<00:26, 1.25s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 75%|███████▌ | 60/80 [04:09<00:25, 1.25s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 76%|███████▋ | 61/80 [04:12<00:27, 1.46s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 76%|███████▋ | 61/80 [04:12<00:27, 1.46s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 78%|███████▊ | 62/80 [04:12<00:21, 1.17s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 78%|███████▊ | 62/80 [04:13<00:21, 1.17s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 78%|███████▊ | 62/80 [04:13<00:21, 1.17s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 79%|███████▉ | 63/80 [04:13<00:17, 1.05s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 79%|███████▉ | 63/80 [04:13<00:17, 1.05s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 79%|███████▉ | 63/80 [04:13<00:17, 1.05s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 80%|████████ | 64/80 [04:13<00:14, 1.12it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 80%|████████ | 64/80 [04:14<00:14, 1.12it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 80%|████████ | 64/80 [05:08<00:14, 1.12it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 81%|████████▏ | 65/80 [05:08<03:45, 15.00s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 81%|████████▏ | 65/80 [05:08<03:45, 15.00s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 82%|████████▎ | 66/80 [05:08<02:32, 10.91s/it]
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.28 GiB object_store_memory: 82%|████████▎ | 66/80 [05:09<02:32, 10.91s/it]
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.28 GiB object_store_memory: 84%|████████▍ | 67/80 [05:09<01:45, 8.13s/it]
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 12.0 MiB/4.28 GiB object_store_memory: 84%|████████▍ | 67/80 [05:09<01:45, 8.13s/it]
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 12.0 MiB/4.28 GiB object_store_memory: 85%|████████▌ | 68/80 [05:09<01:10, 5.85s/it]
Running: 11.0/16.0 CPU, 0.0/0.0 GPU, 11.0 MiB/4.28 GiB object_store_memory: 85%|████████▌ | 68/80 [05:09<01:10, 5.85s/it]
Running: 11.0/16.0 CPU, 0.0/0.0 GPU, 11.0 MiB/4.28 GiB object_store_memory: 86%|████████▋ | 69/80 [05:09<00:46, 4.21s/it]
Running: 10.0/16.0 CPU, 0.0/0.0 GPU, 10.0 MiB/4.28 GiB object_store_memory: 86%|████████▋ | 69/80 [05:09<00:46, 4.21s/it]
Running: 10.0/16.0 CPU, 0.0/0.0 GPU, 10.0 MiB/4.28 GiB object_store_memory: 88%|████████▊ | 70/80 [05:09<00:30, 3.00s/it]
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 9.0 MiB/4.28 GiB object_store_memory: 88%|████████▊ | 70/80 [05:09<00:30, 3.00s/it]
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 9.0 MiB/4.28 GiB object_store_memory: 89%|████████▉ | 71/80 [05:09<00:19, 2.15s/it]
Running: 8.0/16.0 CPU, 0.0/0.0 GPU, 8.0 MiB/4.28 GiB object_store_memory: 89%|████████▉ | 71/80 [05:10<00:19, 2.15s/it]
Running: 8.0/16.0 CPU, 0.0/0.0 GPU, 8.0 MiB/4.28 GiB object_store_memory: 90%|█████████ | 72/80 [05:10<00:12, 1.54s/it]
Running: 8.0/16.0 CPU, 0.0/0.0 GPU, 8.0 MiB/4.28 GiB object_store_memory: 91%|█████████▏| 73/80 [05:10<00:07, 1.14s/it]
Running: 6.0/16.0 CPU, 0.0/0.0 GPU, 6.0 MiB/4.28 GiB object_store_memory: 92%|█████████▎| 74/80 [05:10<00:06, 1.14s/it]
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 5.0 MiB/4.28 GiB object_store_memory: 92%|█████████▎| 74/80 [05:10<00:06, 1.14s/it]
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 5.0 MiB/4.28 GiB object_store_memory: 94%|█████████▍| 75/80 [05:10<00:03, 1.50it/s]
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 94%|█████████▍| 75/80 [05:10<00:03, 1.50it/s]
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 95%|█████████▌| 76/80 [05:10<00:02, 1.90it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.0 MiB/4.28 GiB object_store_memory: 95%|█████████▌| 76/80 [05:12<00:02, 1.90it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.0 MiB/4.28 GiB object_store_memory: 96%|█████████▋| 77/80 [05:12<00:02, 1.19it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.0 MiB/4.28 GiB object_store_memory: 98%|█████████▊| 78/80 [05:12<00:01, 1.44it/s]
Running: 2.0/16.0 CPU, 0.0/0.0 GPU, 2.0 MiB/4.28 GiB object_store_memory: 98%|█████████▊| 78/80 [05:12<00:01, 1.44it/s]
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 1.0 MiB/4.28 GiB object_store_memory: 98%|█████████▊| 78/80 [05:13<00:01, 1.44it/s]
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 1.0 MiB/4.28 GiB object_store_memory: 99%|█████████▉| 79/80 [05:13<00:00, 1.38it/s]
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.28 GiB object_store_memory: 99%|█████████▉| 79/80 [05:13<00:00, 1.38it/s]
2023-04-18 22:34:37,666 INFO streaming_executor.py:87 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[ReadRange->MapBatches(generate_features)->Write]
2023-04-18 22:34:37,666 INFO streaming_executor.py:88 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=False, actor_locality_enabled=True, verbose_progress=False)
2023-04-18 22:34:37,666 INFO streaming_executor.py:91 -- Tip: To enable per-operator progress reporting, set RAY_DATA_VERBOSE_PROGRESS=1.
Running 0: 0%| | 0/80 [00:00<?, ?it/s]
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:00<?, ?it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:00<?, ?it/s]
(ReadRange->MapBatches(generate_features)->Write pid=7216) 2023-04-18 22:34:49.228824: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA [repeated 4x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:11<?, ?it/s]
(ReadRange->MapBatches(generate_features)->Write pid=7216) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. [repeated 4x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:11<?, ?it/s]
(ReadRange->MapBatches(generate_features)->Write pid=3547) 2023-04-18 22:29:56.934307: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. [repeated 4x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:11<?, ?it/s]
(ReadRange->MapBatches(generate_features)->Write pid=3547) 2023-04-18 22:29:58.277619: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 [repeated 8x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:11<?, ?it/s]
(ReadRange->MapBatches(generate_features)->Write pid=3547) 2023-04-18 22:29:58.277633: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. [repeated 4x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:11<?, ?it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 0%| | 0/80 [00:38<?, ?it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 1%|▏ | 1/80 [00:38<50:45, 38.55s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 1%|▏ | 1/80 [00:38<50:45, 38.55s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 1%|▏ | 1/80 [00:38<50:45, 38.55s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 2%|▎ | 2/80 [00:38<20:52, 16.06s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 4%|▍ | 3/80 [00:38<11:15, 8.77s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 4%|▍ | 3/80 [00:39<11:15, 8.77s/it]
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.28 GiB object_store_memory: 4%|▍ | 3/80 [00:39<11:15, 8.77s/it]
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.28 GiB object_store_memory: 5%|▌ | 4/80 [00:39<06:49, 5.39s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 8%|▊ | 6/80 [00:39<06:38, 5.39s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 9%|▉ | 7/80 [00:39<02:27, 2.03s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 10%|█ | 8/80 [00:39<02:25, 2.03s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 11%|█▏ | 9/80 [00:39<01:31, 1.29s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 11%|█▏ | 9/80 [00:39<01:31, 1.29s/it]
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.28 GiB object_store_memory: 11%|█▏ | 9/80 [00:39<01:31, 1.29s/it]
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.28 GiB object_store_memory: 14%|█▍ | 11/80 [00:39<01:00, 1.14it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 15%|█▌ | 12/80 [00:39<00:59, 1.14it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 15%|█▌ | 12/80 [00:39<00:59, 1.14it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 16%|█▋ | 13/80 [00:39<00:41, 1.61it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 16%|█▋ | 13/80 [00:39<00:41, 1.61it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 16%|█▋ | 13/80 [00:43<00:41, 1.61it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 18%|█▊ | 14/80 [00:43<00:40, 1.61it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 19%|█▉ | 15/80 [00:43<01:10, 1.09s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 20%|██ | 16/80 [00:43<01:09, 1.09s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 20%|██ | 16/80 [00:44<01:09, 1.09s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 20%|██ | 16/80 [01:18<01:09, 1.09s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 21%|██▏ | 17/80 [01:18<06:32, 6.24s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 21%|██▏ | 17/80 [01:18<06:32, 6.24s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 21%|██▏ | 17/80 [01:19<06:32, 6.24s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 22%|██▎ | 18/80 [01:19<05:28, 5.31s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 22%|██▎ | 18/80 [01:20<05:28, 5.31s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 24%|██▍ | 19/80 [01:20<04:18, 4.23s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 25%|██▌ | 20/80 [01:20<04:13, 4.23s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 26%|██▋ | 21/80 [01:20<02:37, 2.67s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 28%|██▊ | 22/80 [01:20<02:35, 2.67s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 28%|██▊ | 22/80 [01:20<02:35, 2.67s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 29%|██▉ | 23/80 [01:20<01:45, 1.85s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 30%|███ | 24/80 [01:20<01:24, 1.51s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 32%|███▎ | 26/80 [01:21<00:53, 1.00it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 32%|███▎ | 26/80 [01:21<00:53, 1.00it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 32%|███▎ | 26/80 [01:21<00:53, 1.00it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 34%|███▍ | 27/80 [01:21<00:47, 1.11it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 35%|███▌ | 28/80 [01:21<00:37, 1.39it/s]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 36%|███▋ | 29/80 [01:21<00:36, 1.39it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 36%|███▋ | 29/80 [01:21<00:36, 1.39it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 36%|███▋ | 29/80 [01:24<00:36, 1.39it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 38%|███▊ | 30/80 [01:24<00:44, 1.12it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 38%|███▊ | 30/80 [01:24<00:44, 1.12it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 39%|███▉ | 31/80 [01:25<00:49, 1.02s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 39%|███▉ | 31/80 [01:25<00:49, 1.02s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 39%|███▉ | 31/80 [01:25<00:49, 1.02s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 39%|███▉ | 31/80 [01:27<00:49, 1.02s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 40%|████ | 32/80 [01:27<01:01, 1.29s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 40%|████ | 32/80 [01:27<01:01, 1.29s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 40%|████ | 32/80 [01:59<01:01, 1.29s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 41%|████▏ | 33/80 [01:59<07:02, 9.00s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 41%|████▏ | 33/80 [01:59<07:02, 9.00s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 41%|████▏ | 33/80 [02:00<07:02, 9.00s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 42%|████▎ | 34/80 [02:00<05:16, 6.89s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 42%|████▎ | 34/80 [02:00<05:16, 6.89s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 42%|████▎ | 34/80 [02:00<05:16, 6.89s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 44%|████▍ | 35/80 [02:00<03:47, 5.05s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 44%|████▍ | 35/80 [02:00<03:47, 5.05s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 44%|████▍ | 35/80 [02:00<03:47, 5.05s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 45%|████▌ | 36/80 [02:00<02:43, 3.71s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 45%|████▌ | 36/80 [02:01<02:43, 3.71s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 46%|████▋ | 37/80 [02:01<01:54, 2.67s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 48%|████▊ | 38/80 [02:01<01:52, 2.67s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 49%|████▉ | 39/80 [02:01<01:02, 1.52s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 49%|████▉ | 39/80 [02:01<01:02, 1.52s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 49%|████▉ | 39/80 [02:01<01:02, 1.52s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 50%|█████ | 40/80 [02:01<00:49, 1.23s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 50%|█████ | 40/80 [02:01<00:49, 1.23s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 51%|█████▏ | 41/80 [02:01<00:36, 1.07it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 52%|█████▎ | 42/80 [02:01<00:27, 1.40it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 52%|█████▎ | 42/80 [02:01<00:27, 1.40it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 52%|█████▎ | 42/80 [02:02<00:27, 1.40it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 54%|█████▍ | 43/80 [02:02<00:21, 1.73it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 54%|█████▍ | 43/80 [02:02<00:21, 1.73it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 54%|█████▍ | 43/80 [02:02<00:21, 1.73it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 55%|█████▌ | 44/80 [02:02<00:20, 1.78it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 55%|█████▌ | 44/80 [02:02<00:20, 1.78it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 55%|█████▌ | 44/80 [02:02<00:20, 1.78it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 56%|█████▋ | 45/80 [02:02<00:17, 2.03it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 56%|█████▋ | 45/80 [02:02<00:17, 2.03it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 56%|█████▋ | 45/80 [02:03<00:17, 2.03it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 57%|█████▊ | 46/80 [02:03<00:22, 1.51it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 57%|█████▊ | 46/80 [02:04<00:22, 1.51it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 59%|█████▉ | 47/80 [02:05<00:30, 1.07it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 59%|█████▉ | 47/80 [02:05<00:30, 1.07it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 59%|█████▉ | 47/80 [02:05<00:30, 1.07it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 59%|█████▉ | 47/80 [02:08<00:30, 1.07it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 60%|██████ | 48/80 [02:08<00:48, 1.52s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 60%|██████ | 48/80 [02:08<00:48, 1.52s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 60%|██████ | 48/80 [02:41<00:48, 1.52s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 61%|██████▏ | 49/80 [02:41<05:36, 10.85s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 61%|██████▏ | 49/80 [02:41<05:36, 10.85s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 61%|██████▏ | 49/80 [02:41<05:36, 10.85s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 62%|██████▎ | 50/80 [02:41<03:52, 7.74s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 62%|██████▎ | 50/80 [02:41<03:52, 7.74s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 62%|██████▎ | 50/80 [02:42<03:52, 7.74s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 64%|██████▍ | 51/80 [02:42<02:40, 5.53s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 64%|██████▍ | 51/80 [02:42<02:40, 5.53s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 64%|██████▍ | 51/80 [02:42<02:40, 5.53s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 65%|██████▌ | 52/80 [02:42<01:51, 3.98s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 66%|██████▋ | 53/80 [02:42<01:16, 2.82s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 68%|██████▊ | 54/80 [02:42<01:13, 2.82s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 68%|██████▊ | 54/80 [02:42<01:13, 2.82s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 69%|██████▉ | 55/80 [02:42<00:39, 1.57s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 69%|██████▉ | 55/80 [02:42<00:39, 1.57s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 69%|██████▉ | 55/80 [02:42<00:39, 1.57s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 69%|██████▉ | 55/80 [02:43<00:39, 1.57s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 70%|███████ | 56/80 [02:43<00:31, 1.32s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 70%|███████ | 56/80 [02:43<00:31, 1.32s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 70%|███████ | 56/80 [02:43<00:31, 1.32s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 71%|███████▏ | 57/80 [02:43<00:24, 1.06s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 71%|███████▏ | 57/80 [02:43<00:24, 1.06s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 72%|███████▎ | 58/80 [02:43<00:17, 1.25it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 74%|███████▍ | 59/80 [02:43<00:16, 1.25it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 74%|███████▍ | 59/80 [02:44<00:16, 1.25it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 75%|███████▌ | 60/80 [02:44<00:12, 1.63it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 75%|███████▌ | 60/80 [02:44<00:12, 1.63it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 75%|███████▌ | 60/80 [02:44<00:12, 1.63it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 76%|███████▋ | 61/80 [02:44<00:10, 1.84it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 76%|███████▋ | 61/80 [02:44<00:10, 1.84it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 76%|███████▋ | 61/80 [02:46<00:10, 1.84it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 78%|███████▊ | 62/80 [02:46<00:14, 1.23it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 78%|███████▊ | 62/80 [02:46<00:14, 1.23it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 79%|███████▉ | 63/80 [02:47<00:15, 1.07it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 79%|███████▉ | 63/80 [02:47<00:15, 1.07it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 79%|███████▉ | 63/80 [02:47<00:15, 1.07it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 79%|███████▉ | 63/80 [02:51<00:15, 1.07it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 80%|████████ | 64/80 [02:51<00:25, 1.61s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 80%|████████ | 64/80 [02:51<00:25, 1.61s/it]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 16.0 MiB/4.28 GiB object_store_memory: 81%|████████▏ | 65/80 [03:23<02:32, 10.17s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 15.0 MiB/4.28 GiB object_store_memory: 81%|████████▏ | 65/80 [03:23<02:32, 10.17s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 81%|████████▏ | 65/80 [03:23<02:32, 10.17s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 14.0 MiB/4.28 GiB object_store_memory: 82%|████████▎ | 66/80 [03:23<01:41, 7.28s/it]
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 13.0 MiB/4.28 GiB object_store_memory: 82%|████████▎ | 66/80 [03:23<01:41, 7.28s/it]
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 12.0 MiB/4.28 GiB object_store_memory: 84%|████████▍ | 67/80 [03:23<01:34, 7.28s/it]
Running: 12.0/16.0 CPU, 0.0/0.0 GPU, 12.0 MiB/4.28 GiB object_store_memory: 85%|████████▌ | 68/80 [03:23<00:48, 4.04s/it]
Running: 11.0/16.0 CPU, 0.0/0.0 GPU, 11.0 MiB/4.28 GiB object_store_memory: 85%|████████▌ | 68/80 [03:23<00:48, 4.04s/it]
Running: 11.0/16.0 CPU, 0.0/0.0 GPU, 11.0 MiB/4.28 GiB object_store_memory: 86%|████████▋ | 69/80 [03:23<00:34, 3.13s/it]
Running: 11.0/16.0 CPU, 0.0/0.0 GPU, 11.0 MiB/4.28 GiB object_store_memory: 88%|████████▊ | 70/80 [03:23<00:23, 2.35s/it]
Running: 10.0/16.0 CPU, 0.0/0.0 GPU, 10.0 MiB/4.28 GiB object_store_memory: 88%|████████▊ | 70/80 [03:23<00:23, 2.35s/it]
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 9.0 MiB/4.28 GiB object_store_memory: 88%|████████▊ | 70/80 [03:24<00:23, 2.35s/it]
Running: 9.0/16.0 CPU, 0.0/0.0 GPU, 9.0 MiB/4.28 GiB object_store_memory: 89%|████████▉ | 71/80 [03:24<00:16, 1.80s/it]
Running: 8.0/16.0 CPU, 0.0/0.0 GPU, 8.0 MiB/4.28 GiB object_store_memory: 89%|████████▉ | 71/80 [03:24<00:16, 1.80s/it]
Running: 8.0/16.0 CPU, 0.0/0.0 GPU, 8.0 MiB/4.28 GiB object_store_memory: 90%|█████████ | 72/80 [03:24<00:11, 1.38s/it]
Running: 7.0/16.0 CPU, 0.0/0.0 GPU, 7.0 MiB/4.28 GiB object_store_memory: 90%|█████████ | 72/80 [03:24<00:11, 1.38s/it]
Running: 7.0/16.0 CPU, 0.0/0.0 GPU, 7.0 MiB/4.28 GiB object_store_memory: 91%|█████████▏| 73/80 [03:24<00:07, 1.05s/it]
Running: 6.0/16.0 CPU, 0.0/0.0 GPU, 6.0 MiB/4.28 GiB object_store_memory: 91%|█████████▏| 73/80 [03:24<00:07, 1.05s/it]
Running: 6.0/16.0 CPU, 0.0/0.0 GPU, 6.0 MiB/4.28 GiB object_store_memory: 92%|█████████▎| 74/80 [03:24<00:04, 1.24it/s]
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 5.0 MiB/4.28 GiB object_store_memory: 92%|█████████▎| 74/80 [03:24<00:04, 1.24it/s]
Running: 5.0/16.0 CPU, 0.0/0.0 GPU, 5.0 MiB/4.28 GiB object_store_memory: 94%|█████████▍| 75/80 [03:24<00:02, 1.67it/s]
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 94%|█████████▍| 75/80 [03:24<00:02, 1.67it/s]
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 95%|█████████▌| 76/80 [03:24<00:01, 2.21it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.0 MiB/4.28 GiB object_store_memory: 95%|█████████▌| 76/80 [03:25<00:01, 2.21it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 3.0 MiB/4.28 GiB object_store_memory: 96%|█████████▋| 77/80 [03:25<00:01, 2.87it/s]
Running: 2.0/16.0 CPU, 0.0/0.0 GPU, 2.0 MiB/4.28 GiB object_store_memory: 96%|█████████▋| 77/80 [03:25<00:01, 2.87it/s]
Running: 2.0/16.0 CPU, 0.0/0.0 GPU, 2.0 MiB/4.28 GiB object_store_memory: 98%|█████████▊| 78/80 [03:25<00:00, 2.34it/s]
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 1.0 MiB/4.28 GiB object_store_memory: 98%|█████████▊| 78/80 [03:27<00:00, 2.34it/s]
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 1.0 MiB/4.28 GiB object_store_memory: 99%|█████████▉| 79/80 [03:27<00:00, 1.39it/s]
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.28 GiB object_store_memory: 99%|█████████▉| 79/80 [03:28<00:00, 1.39it/s]
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.28 GiB object_store_memory: 100%|██████████| 80/80 [03:28<00:00, 1.28it/s]
2023-04-18 22:38:05,765 INFO streaming_executor.py:87 -- Executing DAG InputDataBuffer[Input] -> TaskPoolMapOperator[ReadRange->MapBatches(generate_features)->Write]
2023-04-18 22:38:05,765 INFO streaming_executor.py:88 -- Execution config: ExecutionOptions(resource_limits=ExecutionResources(cpu=None, gpu=None, object_store_memory=None), locality_with_output=False, preserve_order=False, actor_locality_enabled=True, verbose_progress=False)
2023-04-18 22:38:05,765 INFO streaming_executor.py:91 -- Tip: To enable per-operator progress reporting, set RAY_DATA_VERBOSE_PROGRESS=1.
Running 0: 0%| | 0/32 [00:00<?, ?it/s]
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:00<?, ?it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:00<?, ?it/s]
(ReadRange->MapBatches(generate_features)->Write pid=9742) 2023-04-18 22:38:14.549368: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA [repeated 3x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:08<?, ?it/s]
(ReadRange->MapBatches(generate_features)->Write pid=9742) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. [repeated 3x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:08<?, ?it/s]
(ReadRange->MapBatches(generate_features)->Write pid=7218) 2023-04-18 22:34:49.600254: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. [repeated 3x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:08<?, ?it/s]
(ReadRange->MapBatches(generate_features)->Write pid=7218) 2023-04-18 22:34:50.854900: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 [repeated 6x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:08<?, ?it/s]
(ReadRange->MapBatches(generate_features)->Write pid=7218) 2023-04-18 22:34:50.854914: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. [repeated 3x across cluster]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 0%| | 0/32 [00:08<?, ?it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 3%|▎ | 1/32 [00:12<06:26, 12.45s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.28 GiB object_store_memory: 6%|▋ | 2/32 [00:12<06:13, 12.45s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.28 GiB object_store_memory: 9%|▉ | 3/32 [00:12<01:34, 3.27s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.28 GiB object_store_memory: 16%|█▌ | 5/32 [00:12<01:28, 3.27s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.28 GiB object_store_memory: 19%|█▉ | 6/32 [00:12<00:34, 1.31s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.28 GiB object_store_memory: 19%|█▉ | 6/32 [00:12<00:34, 1.31s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.28 GiB object_store_memory: 25%|██▌ | 8/32 [00:12<00:20, 1.18it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 25%|██▌ | 8/32 [00:13<00:20, 1.18it/s]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.28 GiB object_store_memory: 25%|██▌ | 8/32 [00:13<00:20, 1.18it/s]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.28 GiB object_store_memory: 31%|███▏ | 10/32 [00:13<00:13, 1.63it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 31%|███▏ | 10/32 [00:13<00:13, 1.63it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.28 GiB object_store_memory: 31%|███▏ | 10/32 [00:13<00:13, 1.63it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.28 GiB object_store_memory: 38%|███▊ | 12/32 [00:13<00:09, 2.15it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 38%|███▊ | 12/32 [00:13<00:09, 2.15it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 41%|████ | 13/32 [00:13<00:07, 2.41it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.28 GiB object_store_memory: 41%|████ | 13/32 [00:13<00:07, 2.41it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 41%|████ | 13/32 [00:13<00:07, 2.41it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.28 GiB object_store_memory: 41%|████ | 13/32 [00:17<00:07, 2.41it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.28 GiB object_store_memory: 44%|████▍ | 14/32 [00:17<00:20, 1.14s/it]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.28 GiB object_store_memory: 47%|████▋ | 15/32 [00:17<00:15, 1.11it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 47%|████▋ | 15/32 [00:17<00:15, 1.11it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 50%|█████ | 16/32 [00:18<00:13, 1.17it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.28 GiB object_store_memory: 50%|█████ | 16/32 [00:18<00:13, 1.17it/s]
Running: 16.0/16.0 CPU, 0.0/0.0 GPU, 4.0 MiB/4.28 GiB object_store_memory: 50%|█████ | 16/32 [00:18<00:13, 1.17it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.28 GiB object_store_memory: 50%|█████ | 16/32 [00:24<00:13, 1.17it/s]
Running: 15.0/16.0 CPU, 0.0/0.0 GPU, 3.75 MiB/4.28 GiB object_store_memory: 53%|█████▎ | 17/32 [00:24<00:35, 2.34s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.28 GiB object_store_memory: 53%|█████▎ | 17/32 [00:25<00:35, 2.34s/it]
Running: 14.0/16.0 CPU, 0.0/0.0 GPU, 3.5 MiB/4.28 GiB object_store_memory: 56%|█████▋ | 18/32 [00:25<00:24, 1.73s/it]
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 3.25 MiB/4.28 GiB object_store_memory: 56%|█████▋ | 18/32 [00:25<00:24, 1.73s/it]
Running: 13.0/16.0 CPU, 0.0/0.0 GPU, 3.25 MiB/4.28 GiB object_store_memory: 59%|█████▉ | 19/32 [00:25<00:16, 1.27s/it]
Running: 11.0/16.0 CPU, 0.0/0.0 GPU, 2.75 MiB/4.28 GiB object_store_memory: 59%|█████▉ | 19/32 [00:25<00:16, 1.27s/it]
Running: 11.0/16.0 CPU, 0.0/0.0 GPU, 2.75 MiB/4.28 GiB object_store_memory: 62%|██████▎ | 20/32 [00:25<00:11, 1.07it/s]
Running: 8.0/16.0 CPU, 0.0/0.0 GPU, 2.0 MiB/4.28 GiB object_store_memory: 66%|██████▌ | 21/32 [00:25<00:10, 1.07it/s]
Running: 8.0/16.0 CPU, 0.0/0.0 GPU, 2.0 MiB/4.28 GiB object_store_memory: 69%|██████▉ | 22/32 [00:25<00:06, 1.58it/s]
Running: 7.0/16.0 CPU, 0.0/0.0 GPU, 1.75 MiB/4.28 GiB object_store_memory: 75%|███████▌ | 24/32 [00:25<00:05, 1.58it/s]
Running: 7.0/16.0 CPU, 0.0/0.0 GPU, 1.75 MiB/4.28 GiB object_store_memory: 78%|███████▊ | 25/32 [00:25<00:02, 2.96it/s]
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 1.0 MiB/4.28 GiB object_store_memory: 78%|███████▊ | 25/32 [00:25<00:02, 2.96it/s]
Running: 4.0/16.0 CPU, 0.0/0.0 GPU, 1.0 MiB/4.28 GiB object_store_memory: 84%|████████▍ | 27/32 [00:25<00:01, 4.05it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 0.75 MiB/4.28 GiB object_store_memory: 88%|████████▊ | 28/32 [00:26<00:00, 4.05it/s]
Running: 3.0/16.0 CPU, 0.0/0.0 GPU, 0.75 MiB/4.28 GiB object_store_memory: 91%|█████████ | 29/32 [00:26<00:00, 4.97it/s]
Running: 2.0/16.0 CPU, 0.0/0.0 GPU, 0.5 MiB/4.28 GiB object_store_memory: 94%|█████████▍| 30/32 [00:27<00:00, 4.97it/s]
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 0.25 MiB/4.28 GiB object_store_memory: 94%|█████████▍| 30/32 [00:27<00:00, 4.97it/s]
Running: 1.0/16.0 CPU, 0.0/0.0 GPU, 0.25 MiB/4.28 GiB object_store_memory: 97%|█████████▋| 31/32 [00:27<00:00, 2.37it/s]
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.28 GiB object_store_memory: 97%|█████████▋| 31/32 [00:28<00:00, 2.37it/s]
Running: 0.0/16.0 CPU, 0.0/0.0 GPU, 0.0 MiB/4.28 GiB object_store_memory: 100%|██████████| 32/32 [00:28<00:00, 2.65it/s]
Running case: tfrecords-images-100-256
Read progress 0: 0%| | 0/32 [00:00<?, ?it/s]
Result of case tfrecords-images-100-256: {'time': 0.11246879999998782}
Running case: tfrecords-images-100-2048
Read progress 0: 0%| | 0/100 [00:00<?, ?it/s]
Read progress 0: 15%|█▌ | 15/100 [00:00<00:00, 106.12it/s]
Read progress 0: 26%|██▌ | 26/100 [00:00<00:01, 64.49it/s]
Read progress 0: 37%|███▋ | 37/100 [00:00<00:00, 71.89it/s]
Read progress 0: 49%|████▉ | 49/100 [00:00<00:00, 82.14it/s]
Read progress 0: 61%|██████ | 61/100 [00:00<00:00, 85.51it/s]
Read progress 0: 74%|███████▍ | 74/100 [00:00<00:00, 85.95it/s]
Read progress 0: 86%|████████▌ | 86/100 [00:01<00:00, 94.34it/s]
(_execute_read_task_split pid=10303) 2023-04-18 22:38:35.837302: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA [repeated 3x across cluster]
Read progress 0: 97%|█████████▋| 97/100 [00:01<00:00, 94.34it/s]
(_execute_read_task_split pid=10303) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. [repeated 3x across cluster]
Read progress 0: 97%|█████████▋| 97/100 [00:01<00:00, 94.34it/s]
(ReadRange->MapBatches(generate_features)->Write pid=9740) 2023-04-18 22:38:15.385531: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. [repeated 3x across cluster]
Read progress 0: 97%|█████████▋| 97/100 [00:01<00:00, 94.34it/s]
(ReadRange->MapBatches(generate_features)->Write pid=9740) 2023-04-18 22:38:16.881614: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 [repeated 6x across cluster]
Read progress 0: 97%|█████████▋| 97/100 [00:01<00:00, 94.34it/s]
(ReadRange->MapBatches(generate_features)->Write pid=9740) 2023-04-18 22:38:16.881630: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. [repeated 3x across cluster]
Read progress 0: 97%|█████████▋| 97/100 [00:01<00:00, 94.34it/s]
Read progress 0: 98%|█████████▊| 98/100 [00:03<00:00, 14.29it/s]
Result of case tfrecords-images-100-2048: {'time': 3.969540747999986}
Running case: tfrecords-images-1000-mix
Read progress 0: 0%| | 0/32 [00:00<?, ?it/s]
Result of case tfrecords-images-1000-mix: {'time': 0.0678202810000812}
Running case: tfrecords-random-int-1g
Read progress 0: 0%| | 0/80 [00:00<?, ?it/s]
(_execute_read_task_split pid=11572) 2023-04-18 22:38:39.985242: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA [repeated 5x across cluster]
Read progress 0: 0%| | 0/80 [00:03<?, ?it/s]
(_execute_read_task_split pid=11572) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. [repeated 5x across cluster]
Read progress 0: 0%| | 0/80 [00:03<?, ?it/s]
(_execute_read_task_split pid=11572) 2023-04-18 22:38:40.166711: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`. [repeated 6x across cluster]
Read progress 0: 0%| | 0/80 [00:03<?, ?it/s]
(_execute_read_task_split pid=11570) 2023-04-18 22:38:41.374321: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 [repeated 8x across cluster]
Read progress 0: 0%| | 0/80 [00:03<?, ?it/s]
(_execute_read_task_split pid=11570) 2023-04-18 22:38:41.374332: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. [repeated 4x across cluster]
Read progress 0: 0%| | 0/80 [00:03<?, ?it/s]
Read progress 0: 1%|▏ | 1/80 [04:02<5:19:39, 242.78s/it]
Read progress 0: 2%|▎ | 2/80 [04:04<2:11:01, 100.79s/it]
Read progress 0: 6%|▋ | 5/80 [04:04<35:58, 28.78s/it]
Read progress 0: 8%|▊ | 6/80 [04:04<26:31, 21.50s/it]
Read progress 0: 9%|▉ | 7/80 [04:05<19:14, 15.81s/it]
Read progress 0: 10%|█ | 8/80 [04:05<13:58, 11.65s/it]
Read progress 0: 11%|█▏ | 9/80 [04:05<09:56, 8.40s/it]
Read progress 0: 12%|█▎ | 10/80 [04:06<07:01, 6.02s/it]
Read progress 0: 14%|█▍ | 11/80 [04:06<04:56, 4.30s/it]
Read progress 0: 15%|█▌ | 12/80 [04:06<03:34, 3.15s/it]
Read progress 0: 18%|█▊ | 14/80 [04:08<02:15, 2.06s/it]
Read progress 0: 19%|█▉ | 15/80 [04:09<02:00, 1.86s/it]
Read progress 0: 21%|██▏ | 17/80 [08:05<52:18, 49.82s/it]
Read progress 0: 22%|██▎ | 18/80 [08:05<39:50, 38.56s/it]
Read progress 0: 25%|██▌ | 20/80 [08:07<23:40, 23.68s/it]
Read progress 0: 28%|██▊ | 22/80 [08:07<14:42, 15.21s/it]
Read progress 0: 29%|██▉ | 23/80 [08:09<11:45, 12.38s/it]
Read progress 0: 30%|███ | 24/80 [08:09<09:00, 9.66s/it]
Read progress 0: 31%|███▏ | 25/80 [08:09<06:45, 7.38s/it]
Read progress 0: 32%|███▎ | 26/80 [08:10<04:57, 5.51s/it]
Read progress 0: 34%|███▍ | 27/80 [08:10<03:40, 4.16s/it]
Read progress 0: 35%|███▌ | 28/80 [08:11<02:42, 3.12s/it]
Read progress 0: 36%|███▋ | 29/80 [08:11<02:02, 2.40s/it]
Read progress 0: 38%|███▊ | 30/80 [08:11<01:27, 1.76s/it]
Read progress 0: 39%|███▉ | 31/80 [08:12<01:11, 1.46s/it]
Read progress 0: 40%|████ | 32/80 [08:12<00:52, 1.09s/it]
Read progress 0: 41%|████▏ | 33/80 [12:08<55:21, 70.66s/it]
Read progress 0: 42%|████▎ | 34/80 [12:10<38:21, 50.04s/it]
Read progress 0: 44%|████▍ | 35/80 [12:11<26:35, 35.46s/it]
Read progress 0: 45%|████▌ | 36/80 [12:11<18:19, 25.00s/it]
Read progress 0: 46%|████▋ | 37/80 [12:12<12:48, 17.86s/it]
Read progress 0: 48%|████▊ | 38/80 [12:13<08:55, 12.74s/it]
Read progress 0: 49%|████▉ | 39/80 [12:15<06:24, 9.38s/it]
Read progress 0: 50%|█████ | 40/80 [12:15<04:25, 6.64s/it]
Read progress 0: 52%|█████▎ | 42/80 [12:15<02:18, 3.65s/it]
Read progress 0: 54%|█████▍ | 43/80 [12:16<01:52, 3.05s/it]
Read progress 0: 55%|█████▌ | 44/80 [12:17<01:25, 2.37s/it]
Read progress 0: 56%|█████▋ | 45/80 [12:18<01:09, 1.98s/it]
Read progress 0: 57%|█████▊ | 46/80 [12:18<00:50, 1.49s/it]
Read progress 0: 60%|██████ | 48/80 [12:18<00:28, 1.13it/s]
Read progress 0: 61%|██████▏ | 49/80 [16:12<29:23, 56.89s/it]
Read progress 0: 62%|██████▎ | 50/80 [16:14<21:30, 43.03s/it]
Read progress 0: 64%|██████▍ | 51/80 [16:15<15:19, 31.70s/it]
Read progress 0: 65%|██████▌ | 52/80 [16:16<10:49, 23.20s/it]
Read progress 0: 66%|██████▋ | 53/80 [16:18<07:44, 17.19s/it]
Read progress 0: 69%|██████▉ | 55/80 [16:19<04:02, 9.70s/it]
Read progress 0: 70%|███████ | 56/80 [16:20<03:03, 7.65s/it]
Read progress 0: 71%|███████▏ | 57/80 [16:20<02:12, 5.77s/it]
Read progress 0: 72%|███████▎ | 58/80 [16:21<01:35, 4.36s/it]
Read progress 0: 74%|███████▍ | 59/80 [16:22<01:10, 3.36s/it]
Read progress 0: 75%|███████▌ | 60/80 [16:23<00:54, 2.72s/it]
Read progress 0: 78%|███████▊ | 62/80 [16:23<00:28, 1.56s/it]
Read progress 0: 79%|███████▉ | 63/80 [16:24<00:24, 1.45s/it]
Read progress 0: 80%|████████ | 64/80 [16:25<00:19, 1.19s/it]Subprocess return code: 124
[INFO 2023-04-18 22:57:37,602] anyscale_job_wrapper.py: 191 Process 1447 exited with return code 124.
[ERROR 2023-04-18 22:57:37,603] anyscale_job_wrapper.py: 291 Timed out. Time taken: 1800.069069077
[ERROR 2023-04-18 22:57:37,603] anyscale_job_wrapper.py: 68 Couldn't upload to cloud storage: '/tmp/release_test_out.json' does not exist.
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
azure-cli-core 2.40.0 requires packaging<22.0,>=20.9, but you have packaging 23.1 which is incompatible.
mlflow 1.30.0 requires packaging<22, but you have packaging 23.1 which is incompatible.
mosaicml 0.12.1 requires importlib-metadata<7,>=5.0.0, but you have importlib-metadata 4.13.0 which is incompatible.
mosaicml 0.12.1 requires packaging<23,>=21.3.0, but you have packaging 23.1 which is incompatible.
mosaicml 0.12.1 requires pyyaml<7,>=6.0, but you have pyyaml 5.4.1 which is incompatible.
tensorflow 2.11.0 requires protobuf<3.20,>=3.9.2, but you have protobuf 3.20.3 which is incompatible.
Copying file:///tmp/metrics_test_out.json [Content-Type=application/json]...
/ [0 files][ 0.0 B/ 374.0 B]
/ [1 files][ 374.0 B/ 374.0 B]
Operation completed over 1 objects/374.0 B.
Read progress 0: 81%|████████▏ | 65/80 [20:17<15:48, 63.25s/it]Copying file:///tmp/ray/session_2023-04-18_22-27-18_471776_158/runtime_resources/working_dir_files/gs_anyscale-oss-dev-bucket_working_dirs_read_tfrecords_benchmark_single_node_gce_yitigazrhh__anyscale_pkg_aaf02dfc5db28595fbcfe4cd80cdeb4b/output.json [Content-Type=application/json]...
/ [0 files][ 0.0 B/ 247.0 B]
/ [1 files][ 247.0 B/ 247.0 B]
Operation completed over 1 objects/247.0 B.
[INFO 2023-04-18 22:58:57,590] anyscale_job_wrapper.py: 346 ### Finished ###
[INFO 2023-04-18 22:58:57,591] anyscale_job_wrapper.py: 349 ### JSON |{"collected_metrics":true,"last_prepare_time_taken":null,"prepare_return_codes":[],"return_code":124,"total_time_taken":1878.405071267,"uploaded_artifact":false,"uploaded_metrics":true,"uploaded_results":false,"workload_time_taken":1800.069069077}| ###
Read progress 0: 82%|████████▎ | 66/80 [20:20<10:51, 46.55s/it]
Read progress 0: 84%|████████▍ | 67/80 [20:21<07:19, 33.80s/it]
Read progress 0: 86%|████████▋ | 69/80 [20:24<03:30, 19.17s/it]
Read progress 0: 88%|████████▊ | 70/80 [20:24<02:25, 14.57s/it]
Read progress 0: 89%|████████▉ | 71/80 [20:25<01:39, 11.03s/it]
Read progress 0: 90%|█████████ | 72/80 [20:26<01:07, 8.45s/it]
Read progress 0: 91%|█████████▏| 73/80 [20:26<00:43, 6.16s/it]
Read progress 0: 92%|█████████▎| 74/80 [20:27<00:27, 4.57s/it]
Read progress 0: 95%|█████████▌| 76/80 [20:28<00:10, 2.74s/it]
Read progress 0: 98%|█████████▊| 78/80 [20:29<00:03, 1.91s/it]
Read progress 0: 99%|█████████▉| 79/80 [20:30<00:01, 1.59s/it]
Read progress 0: 100%|██████████| 80/80 [20:31<00:00, 1.46s/it]
Result of case tfrecords-random-int-1g: {'time': 1231.250032476}
Running case: tfrecords-random-float-1g
Read progress 0: 0%| | 0/80 [00:00<?, ?it/s]
(_execute_read_task_split pid=25946) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Read progress 0: 0%| | 0/80 [00:01<?, ?it/s]
(_execute_read_task_split pid=25946) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Read progress 0: 0%| | 0/80 [00:01<?, ?it/s]
(_execute_read_task_split pid=11572) 2023-04-18 22:38:41.466254: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 [repeated 4x across cluster]
Read progress 0: 0%| | 0/80 [00:01<?, ?it/s]
(_execute_read_task_split pid=11572) 2023-04-18 22:38:41.466269: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. [repeated 2x across cluster]
Read progress 0: 0%| | 0/80 [00:01<?, ?it/s]
(_execute_read_task_split pid=25947) 2023-04-18 22:59:11.160589: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA
Read progress 0: 0%| | 0/80 [00:01<?, ?it/s]
(_execute_read_task_split pid=25947) To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Read progress 0: 0%| | 0/80 [00:01<?, ?it/s]
(_execute_read_task_split pid=25946) 2023-04-18 22:59:11.328129: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
Read progress 0: 0%| | 0/80 [00:01<?, ?it/s]*** SIGTERM received at time=1681883967 on cpu 15 ***
PC: @ 0x7fb71170dd0b (unknown) unlinkat
@ 0x7fb71195e420 (unknown) (unknown)
[2023-04-18 22:59:27,143 E 1454 1454] logging.cc:361: *** SIGTERM received at time=1681883967 on cpu 15 ***
[2023-04-18 22:59:27,143 E 1454 1454] logging.cc:361: PC: @ 0x7fb71170dd0b (unknown) unlinkat
[2023-04-18 22:59:27,143 E 1454 1454] logging.cc:361: @ 0x7fb71195e420 (unknown) (unknown)
(_execute_read_task_split pid=25946) 2023-04-18 22:59:12.658665: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 [repeated 4x across cluster]
Read progress 0: 0%| | 0/80 [00:17<?, ?it/s]
(_execute_read_task_split pid=25946) 2023-04-18 22:59:12.658679: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly. [repeated 2x across cluster]
Read progress 0: 0%| | 0/80 [00:17<?, ?it/s]
(_execute_read_task_split pid=25947) 2023-04-18 22:59:11.327625: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment