Skip to content

Instantly share code, notes, and snippets.

@lakshay-arora

lakshay-arora/partition_5.py

Created October 22, 2019 07:19
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save lakshay-arora/37a8c4e640324f4d8100e45a9952cfd6 to your computer and use it in GitHub Desktop.
Save lakshay-arora/37a8c4e640324f4d8100e45a9952cfd6 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
partition_5.py
# create five partitions of the list
my_large_list_with_five_partition = sc.parallelize(my_large_list, numSlices=5)
# filter numbers greater than equal to 200
my_large_list_with_five_partition = my_large_list_with_five_partition.filter(lambda x : x >= 200)
%%time
# count the number of elements in the filtered list
print(my_large_list_with_five_partition.count())
# >> 16162207
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment