Skip to content

Instantly share code, notes, and snippets.

@novasush

novasush/parallelize_transformation.py

Created March 7, 2020 18:12
Show Gist options
  • Save novasush/907334be75e30ecf443a046306e43b77 to your computer and use it in GitHub Desktop.
Save novasush/907334be75e30ecf443a046306e43b77 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
parallelize_transformation.py
# Get the number of CPU cores.
cores = multiprocessing.cpu_count()
print(cores)
# Parallelize the transformation of the train_dataset by using
# the map operation with the number of parallel calls set to
# the number of CPU cores.
train_dataset = train_dataset.map(read_tfrecord,num_parallel_calls=cores)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment