Skip to content

Instantly share code, notes, and snippets.

@raidery
Created April 24, 2019 06:42
Show Gist options
  • Save raidery/0326544f0994c0c6e2a103ff7c3108e5 to your computer and use it in GitHub Desktop.
Save raidery/0326544f0994c0c6e2a103ff7c3108e5 to your computer and use it in GitHub Desktop.
averageTime block with Scala
def averageTime[R](block: => R, numIter: Int = 10): Unit = {
val t0 = System.nanoTime()
(1 to numIter).foreach( _ => block)
val t1 = System.nanoTime()
val averageTimeTaken = (t1 - t0) / numIter
val timeTakenMs = averageTimeTaken / 1000000
println("Elapsed time: " + timeTakenMs + "ms")
}
val testDf = spark.range(10000000).toDF.cache
val resultSparkNative = testDf.withColumn("addOne", 'id + 1)
scala> averageTime { resultSparkNative.count }
Elapsed time: 36ms
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment