Skip to content

Instantly share code, notes, and snippets.

@HyukjinKwon
Created July 4, 2017 00:49
Show Gist options
  • Save HyukjinKwon/c3397469c56cb26c2d7dd521ed0bc5a3 to your computer and use it in GitHub Desktop.
Save HyukjinKwon/c3397469c56cb26c2d7dd521ed0bc5a3 to your computer and use it in GitHub Desktop.
[PYSPARK] SPARK-21296 benchmark codes
import time
def timing(f):
def wrap(*args):
time1 = time.time()
ret = f(*args)
time2 = time.time()
print '%s function took %0.3f ms' % (f.func_name, (time2-time1)*1000.0)
return ret
return wrap
data = range(1000000)
for _ in xrange(10):
timing(lambda: spark.createDataFrame(data, "int"))()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment