Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
# create five partitions of the list
my_large_list_with_five_partition = sc.parallelize(my_large_list, numSlices=5)
# filter numbers greater than equal to 200
my_large_list_with_five_partition = my_large_list_with_five_partition.filter(lambda x : x >= 200)
%%time
# count the number of elements in the filtered list
print(my_large_list_with_five_partition.count())
# >> 16162207
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.