Skip to content

Instantly share code, notes, and snippets.

@alekseyl1992

alekseyl1992/spark_perf_test.py

Created January 20, 2016 12:51
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save alekseyl1992/71f0af8daaeb0e5fea00 to your computer and use it in GitHub Desktop.
Save alekseyl1992/71f0af8daaeb0e5fea00 to your computer and use it in GitHub Desktop.
Download ZIP
Spark Perf test
Raw
spark_perf_test.py
import random
def sample(p):
x, y = random.random(), random.random()
return 1 if x*x + y*y < 1 else 0
def run(sc, num_samples):
count = sc.parallelize(xrange(0, num_samples)).map(sample) \
.reduce(lambda a, b: a + b)
print "Pi is roughly %f" % (4.0 * count / num_samples)
def run_pure(num_samples):
count = reduce(lambda a, b: a + b, map(sample, xrange(0, num_samples)))
print "Pi is roughly %f" % (4.0 * count / num_samples)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment