Skip to content

Instantly share code, notes, and snippets.

@retronym

retronym/gist:8c49524b31f4a25d8cee38323c15fd91

Last active May 19, 2020 00:29
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save retronym/8c49524b31f4a25d8cee38323c15fd91 to your computer and use it in GitHub Desktop.
Save retronym/8c49524b31f4a25d8cee38323c15fd91 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
➜ compiler-benchmark git:(topic/case-class-o-rama) time (for v in 2.12.11 2.12.12-bin-8d19363-SNAPSHOT; do echo $v; sbt clean 'set scalaVersion in compilation := "'$v'"' 'hot -psource=scalap -prof gc -wi 3 -i 3 -f 2' ; done)
2.12.11
[info] Loading settings for project global-plugins from idea.sbt,dirtymoney.sbt,metals.sbt,gpg.sbt ...
[info] Loading global plugins from /Users/jz/.sbt/1.0/plugins
[info] Loading settings for project compiler-benchmark-build from plugins.sbt,build.sbt ...
[info] Loading project definition from /Users/jz/code/compiler-benchmark/project
[info] Loading settings for project compiler-benchmark from build.sbt ...
[info] Set current project to compiler-benchmark (in build file:/Users/jz/code/compiler-benchmark/)
[success] Total time: 0 s, completed 19/05/2020 10:21:18 AM
[info] Defining compilation / scalaVersion
[info] The new value will be used by compilation / Compile / compileBinaryFileInputs, compilation / Compile / compileSourceFileInputs and 37 others.
[info] Run `last` for details.
[info] Reapplying settings...
[info] Set current project to compiler-benchmark (in build file:/Users/jz/code/compiler-benchmark/)
[warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
[warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
[info] Compiling 9 Java sources to /Users/jz/code/compiler-benchmark/infrastructure/target/classes ...
[warn] Multiple main classes detected. Run 'show discoveredMainClasses' to see the list
[info] Compiling 4 Scala sources and 3 Java sources to /Users/jz/code/compiler-benchmark/compilation/target/scala-2.12/classes ...
[warn] /Users/jz/code/compiler-benchmark/compilation/src/main/scala/scala/tools/nsc/HotSbtBenchmark.scala:7:36: imported `Mode' is permanently hidden by definition of value Mode in package nsc
[warn] import org.openjdk.jmh.annotations.Mode
[warn] ^
[warn] /Users/jz/code/compiler-benchmark/compilation/src/main/scala/scala/tools/nsc/BenchmarkUtils.scala:50:62: match may not be exhaustive.
[warn] It would fail on the following input: Nil
[warn] val (cond, expr) = trimmedLine.split(" ").toList.filter(_.nonEmpty) match {
[warn] ^
[warn] two warnings found
[info] /Users/jz/code/compiler-benchmark/compilation/src/main/scala/scala/tools/nsc/JavacBenchmark.java: Some input files use unchecked or unsafe operations.
[info] /Users/jz/code/compiler-benchmark/compilation/src/main/scala/scala/tools/nsc/JavacBenchmark.java: Recompile with -Xlint:unchecked for details.
[info] Compiling 1 Scala source to /Users/jz/code/compiler-benchmark/compilation/target/scala-2.12/test-classes ...
Processing 24 classes from /Users/jz/code/compiler-benchmark/compilation/target/scala-2.12/classes with "reflection" generator
Writing out Java source to /Users/jz/code/compiler-benchmark/compilation/target/scala-2.12/src_managed/jmh and resources to /Users/jz/code/compiler-benchmark/compilation/target/scala-2.12/resource_managed/jmh
[info] Compiling 25 Java sources to /Users/jz/code/compiler-benchmark/compilation/target/scala-2.12/classes ...
[info] running (fork) scala.bench.ScalacBenchmarkRunner HotScalacBenchmark -foe true -psource=scalap -prof gc -wi 3 -i 3 -f 2
[info] # JMH version: 1.21
[info] # VM version: JDK 1.8.0_172, Java HotSpot(TM) 64-Bit Server VM, 25.172-b11
[info] # VM invoker: /Users/jz/.jabba/jdk/1.8.172/Contents/Home/jre/bin/java
[info] # VM options: -Xms2G -Xmx2G -Xss2M
[info] # Warmup: 3 iterations, 10 s each
[info] # Measurement: 3 iterations, 10 s each
[info] # Timeout: 10 min per iteration
[info] # Threads: 1 thread, will synchronize iterations
[info] # Benchmark mode: Sampling time
[info] # Benchmark: scala.tools.nsc.HotScalacBenchmark.compile
[info] # Parameters: (corpusPath = ../corpus, corpusVersion = a8c43dc, extraArgs = , resident = false, scalaVersion = 2.12.11, source = scalap)
[info] # Run progress: 0.00% complete, ETA 00:02:00
[info] # Fork: 1 of 2
[info] # Warmup Iteration 1: 5551.161 ms/op
[info] # Warmup Iteration 2: 1347.944 ±(99.9%) 587.480 ms/op
[info] # Warmup Iteration 3: 864.114 ±(99.9%) 103.148 ms/op
[info] Iteration 1: 768.981 ±(99.9%) 47.364 ms/op
[info] compile·p0.00: 707.789 ms/op
[info] compile·p0.50: 756.023 ms/op
[info] compile·p0.90: 828.899 ms/op
[info] compile·p0.95: 831.521 ms/op
[info] compile·p0.99: 831.521 ms/op
[info] compile·p0.999: 831.521 ms/op
[info] compile·p0.9999: 831.521 ms/op
[info] compile·p1.00: 831.521 ms/op
[info] ·gc.alloc.rate: 255.865 MB/sec
[info] ·gc.alloc.rate.norm: 216117365.143 B/op
[info] ·gc.churn.PS_Eden_Space: 290.051 MB/sec
[info] ·gc.churn.PS_Eden_Space.norm: 244992292.571 B/op
[info] ·gc.churn.PS_Survivor_Space: 1.466 MB/sec
[info] ·gc.churn.PS_Survivor_Space.norm: 1237869.143 B/op
[info] ·gc.count: 6.000 counts
[info] ·gc.time: 123.000 ms
[info] Iteration 2: 733.179 ±(99.9%) 44.543 ms/op
[info] compile·p0.00: 691.012 ms/op
[info] compile·p0.50: 727.187 ms/op
[info] compile·p0.90: 804.258 ms/op
[info] compile·p0.95: 824.181 ms/op
[info] compile·p0.99: 824.181 ms/op
[info] compile·p0.999: 824.181 ms/op
[info] compile·p0.9999: 824.181 ms/op
[info] compile·p1.00: 824.181 ms/op
[info] ·gc.alloc.rate: 267.173 MB/sec
[info] ·gc.alloc.rate.norm: 215671142.857 B/op
[info] ·gc.churn.PS_Eden_Space: 253.810 MB/sec
[info] ·gc.churn.PS_Eden_Space.norm: 204884260.571 B/op
[info] ·gc.churn.PS_Survivor_Space: 3.768 MB/sec
[info] ·gc.churn.PS_Survivor_Space.norm: 3041882.286 B/op
[info] ·gc.count: 5.000 counts
[info] ·gc.time: 88.000 ms
[info] Iteration 3: 702.826 ±(99.9%) 35.270 ms/op
[info] compile·p0.00: 667.943 ms/op
[info] compile·p0.50: 689.963 ms/op
[info] compile·p0.90: 762.315 ms/op
[info] compile·p0.95: 765.460 ms/op
[info] compile·p0.99: 765.460 ms/op
[info] compile·p0.999: 765.460 ms/op
[info] compile·p0.9999: 765.460 ms/op
[info] compile·p1.00: 765.460 ms/op
[info] ·gc.alloc.rate: 278.465 MB/sec
[info] ·gc.alloc.rate.norm: 215458264.533 B/op
[info] ·gc.churn.PS_Eden_Space: 295.527 MB/sec
[info] ·gc.churn.PS_Eden_Space.norm: 228659473.067 B/op
[info] ·gc.churn.PS_Survivor_Space: 2.896 MB/sec
[info] ·gc.churn.PS_Survivor_Space.norm: 2240900.267 B/op
[info] ·gc.count: 6.000 counts
[info] ·gc.time: 67.000 ms
[info] # Run progress: 50.00% complete, ETA 00:01:07
[info] # Fork: 2 of 2
[info] # Warmup Iteration 1: 5567.939 ms/op
[info] # Warmup Iteration 2: 1318.584 ±(99.9%) 587.875 ms/op
[info] # Warmup Iteration 3: 877.396 ±(99.9%) 95.319 ms/op
[info] Iteration 1: 773.400 ±(99.9%) 66.202 ms/op
[info] compile·p0.00: 729.809 ms/op
[info] compile·p0.50: 756.023 ms/op
[info] compile·p0.90: 886.047 ms/op
[info] compile·p0.95: 958.398 ms/op
[info] compile·p0.99: 958.398 ms/op
[info] compile·p0.999: 958.398 ms/op
[info] compile·p0.9999: 958.398 ms/op
[info] compile·p1.00: 958.398 ms/op
[info] ·gc.alloc.rate: 255.892 MB/sec
[info] ·gc.alloc.rate.norm: 217247217.143 B/op
[info] ·gc.churn.PS_Eden_Space: 235.331 MB/sec
[info] ·gc.churn.PS_Eden_Space.norm: 199791177.143 B/op
[info] ·gc.churn.PS_Survivor_Space: 2.146 MB/sec
[info] ·gc.churn.PS_Survivor_Space.norm: 1822031.429 B/op
[info] ·gc.count: 5.000 counts
[info] ·gc.time: 99.000 ms
[info] Iteration 2: 731.756 ±(99.9%) 84.927 ms/op
[info] compile·p0.00: 677.380 ms/op
[info] compile·p0.50: 714.080 ms/op
[info] compile·p0.90: 868.745 ms/op
[info] compile·p0.95: 981.467 ms/op
[info] compile·p0.99: 981.467 ms/op
[info] compile·p0.999: 981.467 ms/op
[info] compile·p0.9999: 981.467 ms/op
[info] compile·p1.00: 981.467 ms/op
[info] ·gc.alloc.rate: 269.289 MB/sec
[info] ·gc.alloc.rate.norm: 216925140.571 B/op
[info] ·gc.churn.PS_Eden_Space: 250.529 MB/sec
[info] ·gc.churn.PS_Eden_Space.norm: 201813430.857 B/op
[info] ·gc.churn.PS_Survivor_Space: 3.120 MB/sec
[info] ·gc.churn.PS_Survivor_Space.norm: 2513531.429 B/op
[info] ·gc.count: 5.000 counts
[info] ·gc.time: 77.000 ms
[info] Iteration 3: 696.954 ±(99.9%) 35.801 ms/op
[info] compile·p0.00: 660.603 ms/op
[info] compile·p0.50: 681.574 ms/op
[info] compile·p0.90: 750.780 ms/op
[info] compile·p0.95: 763.363 ms/op
[info] compile·p0.99: 763.363 ms/op
[info] compile·p0.999: 763.363 ms/op
[info] compile·p0.9999: 763.363 ms/op
[info] compile·p1.00: 763.363 ms/op
[info] ·gc.alloc.rate: 282.817 MB/sec
[info] ·gc.alloc.rate.norm: 217312772.267 B/op
[info] ·gc.churn.PS_Eden_Space: 295.582 MB/sec
[info] ·gc.churn.PS_Eden_Space.norm: 227121561.600 B/op
[info] ·gc.churn.PS_Survivor_Space: 2.682 MB/sec
[info] ·gc.churn.PS_Survivor_Space.norm: 2060825.067 B/op
[info] ·gc.count: 6.000 counts
[info] ·gc.time: 64.000 ms
[info] Result "scala.tools.nsc.HotScalacBenchmark.compile":
[info] N = 86
[info] mean = 733.711 ±(99.9%) 20.589 ms/op
[info] Histogram, ms/op:
[info] [ 600.000, 625.000) = 0
[info] [ 625.000, 650.000) = 0
[info] [ 650.000, 675.000) = 7
[info] [ 675.000, 700.000) = 20
[info] [ 700.000, 725.000) = 13
[info] [ 725.000, 750.000) = 20
[info] [ 750.000, 775.000) = 13
[info] [ 775.000, 800.000) = 5
[info] [ 800.000, 825.000) = 4
[info] [ 825.000, 850.000) = 2
[info] [ 850.000, 875.000) = 0
[info] [ 875.000, 900.000) = 0
[info] [ 900.000, 925.000) = 0
[info] [ 925.000, 950.000) = 0
[info] [ 950.000, 975.000) = 1
[info] Percentiles, ms/op:
[info] p(0.0000) = 660.603 ms/op
[info] p(50.0000) = 730.333 ms/op
[info] p(90.0000) = 797.547 ms/op
[info] p(95.0000) = 825.544 ms/op
[info] p(99.0000) = 981.467 ms/op
[info] p(99.9000) = 981.467 ms/op
[info] p(99.9900) = 981.467 ms/op
[info] p(99.9990) = 981.467 ms/op
[info] p(99.9999) = 981.467 ms/op
[info] p(100.0000) = 981.467 ms/op
[info] Secondary result "scala.tools.nsc.HotScalacBenchmark.compile:·gc.alloc.rate":
[info] 268.250 ±(99.9%) 31.349 MB/sec [Average]
[info] (min, avg, max) = (255.865, 268.250, 282.817), stdev = 11.179
[info] CI (99.9%): [236.901, 299.599] (assumes normal distribution)
[info] Secondary result "scala.tools.nsc.HotScalacBenchmark.compile:·gc.alloc.rate.norm":
[info] 216455317.086 ±(99.9%) 2280313.792 B/op [Average]
[info] (min, avg, max) = (215458264.533, 216455317.086, 217312772.267), stdev = 813181.865
[info] CI (99.9%): [214175003.294, 218735630.877] (assumes normal distribution)
[info] Secondary result "scala.tools.nsc.HotScalacBenchmark.compile:·gc.churn.PS_Eden_Space":
[info] 270.138 ±(99.9%) 74.732 MB/sec [Average]
[info] (min, avg, max) = (235.331, 270.138, 295.582), stdev = 26.650
[info] CI (99.9%): [195.406, 344.870] (assumes normal distribution)
[info] Secondary result "scala.tools.nsc.HotScalacBenchmark.compile:·gc.churn.PS_Eden_Space.norm":
[info] 217877032.635 ±(99.9%) 51568212.518 B/op [Average]
[info] (min, avg, max) = (199791177.143, 217877032.635, 244992292.571), stdev = 18389721.345
[info] CI (99.9%): [166308820.117, 269445245.153] (assumes normal distribution)
[info] Secondary result "scala.tools.nsc.HotScalacBenchmark.compile:·gc.churn.PS_Survivor_Space":
[info] 2.680 ±(99.9%) 2.238 MB/sec [Average]
[info] (min, avg, max) = (1.466, 2.680, 3.768), stdev = 0.798
[info] CI (99.9%): [0.442, 4.918] (assumes normal distribution)
[info] Secondary result "scala.tools.nsc.HotScalacBenchmark.compile:·gc.churn.PS_Survivor_Space.norm":
[info] 2152839.937 ±(99.9%) 1721013.185 B/op [Average]
[info] (min, avg, max) = (1237869.143, 2152839.937, 3041882.286), stdev = 613729.881
[info] CI (99.9%): [431826.751, 3873853.122] (assumes normal distribution)
[info] Secondary result "scala.tools.nsc.HotScalacBenchmark.compile:·gc.count":
[info] 33.000 ±(99.9%) 0.001 counts [Sum]
[info] (min, avg, max) = (5.000, 5.500, 6.000), stdev = 0.548
[info] CI (99.9%): [33.000, 33.000] (assumes normal distribution)
[info] Secondary result "scala.tools.nsc.HotScalacBenchmark.compile:·gc.time":
[info] 518.000 ±(99.9%) 0.001 ms [Sum]
[info] (min, avg, max) = (64.000, 86.333, 123.000), stdev = 22.214
[info] CI (99.9%): [518.000, 518.000] (assumes normal distribution)
[info] # Run complete. Total time: 00:02:14
[info] REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
[info] why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
[info] experiments, perform baseline and negative tests that provide experimental control, make sure
[info] the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
[info] Do not assume the numbers tell you what you want them to tell.
[info] Benchmark (corpusPath) (corpusVersion) (extraArgs) (resident) (scalaVersion) (source) Mode Cnt Score Error Units
[info] HotScalacBenchmark.compile ../corpus a8c43dc false 2.12.11 scalap sample 86 733.711 ± 20.589 ms/op
[info] HotScalacBenchmark.compile:compile·p0.00 ../corpus a8c43dc false 2.12.11 scalap sample 660.603 ms/op
[info] HotScalacBenchmark.compile:compile·p0.50 ../corpus a8c43dc false 2.12.11 scalap sample 730.333 ms/op
[info] HotScalacBenchmark.compile:compile·p0.90 ../corpus a8c43dc false 2.12.11 scalap sample 797.547 ms/op
[info] HotScalacBenchmark.compile:compile·p0.95 ../corpus a8c43dc false 2.12.11 scalap sample 825.544 ms/op
[info] HotScalacBenchmark.compile:compile·p0.99 ../corpus a8c43dc false 2.12.11 scalap sample 981.467 ms/op
[info] HotScalacBenchmark.compile:compile·p0.999 ../corpus a8c43dc false 2.12.11 scalap sample 981.467 ms/op
[info] HotScalacBenchmark.compile:compile·p0.9999 ../corpus a8c43dc false 2.12.11 scalap sample 981.467 ms/op
[info] HotScalacBenchmark.compile:compile·p1.00 ../corpus a8c43dc false 2.12.11 scalap sample 981.467 ms/op
[info] HotScalacBenchmark.compile:·gc.alloc.rate ../corpus a8c43dc false 2.12.11 scalap sample 6 268.250 ± 31.349 MB/sec
[info] HotScalacBenchmark.compile:·gc.alloc.rate.norm ../corpus a8c43dc false 2.12.11 scalap sample 6 216455317.086 ± 2280313.792 B/op
[info] HotScalacBenchmark.compile:·gc.churn.PS_Eden_Space ../corpus a8c43dc false 2.12.11 scalap sample 6 270.138 ± 74.732 MB/sec
[info] HotScalacBenchmark.compile:·gc.churn.PS_Eden_Space.norm ../corpus a8c43dc false 2.12.11 scalap sample 6 217877032.635 ± 51568212.518 B/op
[info] HotScalacBenchmark.compile:·gc.churn.PS_Survivor_Space ../corpus a8c43dc false 2.12.11 scalap sample 6 2.680 ± 2.238 MB/sec
[info] HotScalacBenchmark.compile:·gc.churn.PS_Survivor_Space.norm ../corpus a8c43dc false 2.12.11 scalap sample 6 2152839.937 ± 1721013.185 B/op
[info] HotScalacBenchmark.compile:·gc.count ../corpus a8c43dc false 2.12.11 scalap sample 6 33.000 counts
[info] HotScalacBenchmark.compile:·gc.time ../corpus a8c43dc false 2.12.11 scalap sample 6 518.000 ms
[success] Total time: 145 s (02:25), completed 19/05/2020 10:23:44 AM
2.12.12-bin-8d19363-SNAPSHOT
[info] Loading settings for project global-plugins from idea.sbt,dirtymoney.sbt,metals.sbt,gpg.sbt ...
[info] Loading global plugins from /Users/jz/.sbt/1.0/plugins
[info] Loading settings for project compiler-benchmark-build from plugins.sbt,build.sbt ...
[info] Loading project definition from /Users/jz/code/compiler-benchmark/project
[info] Loading settings for project compiler-benchmark from build.sbt ...
[info] Set current project to compiler-benchmark (in build file:/Users/jz/code/compiler-benchmark/)
[success] Total time: 0 s, completed 19/05/2020 10:23:51 AM
[info] Defining compilation / scalaVersion
[info] The new value will be used by compilation / Compile / compileBinaryFileInputs, compilation / Compile / compileSourceFileInputs and 37 others.
[info] Run `last` for details.
[info] Reapplying settings...
[info] Set current project to compiler-benchmark (in build file:/Users/jz/code/compiler-benchmark/)
[warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
[info] Updating
https://scala-ci.typesafe.com/artifactory/scala-pr-validation-snapshots/org/scala-lang/scala-library/2.12.12-bin-8d19363-SNAPSHOT/maven-metadata.xml
100.0% [##########] 1.0 KiB (4.8 KiB / s)
https://scala-ci.typesafe.com/artifactory/scala-pr-validation-snapshots/org/scala-lang/scala-compiler/2.12.12-bin-8d19363-SNAPSHOT/maven-metadata.xml
100.0% [##########] 1.0 KiB (849 B / s)
https://scala-ci.typesafe.com/artifactory/scala-pr-validation-snapshots/org/scala-lang/scala-compiler/2.12.12-bin-8d19363-SNAPSHOT/scala-compiler-2.12.12-bin-8d19363-20200513.09…
100.0% [##########] 2.7 KiB (4.3 KiB / s)
https://scala-ci.typesafe.com/artifactory/scala-pr-validation-snapshots/org/scala-lang/scala-library/2.12.12-bin-8d19363-SNAPSHOT/scala-library-2.12.12-bin-8d19363-20200513.0950…
100.0% [##########] 1.6 KiB (315 B / s)
https://scala-ci.typesafe.com/artifactory/scala-pr-validation-snapshots/org/scala-lang/scala-reflect/2.12.12-bin-8d19363-SNAPSHOT/maven-metadata.xml
100.0% [##########] 1.0 KiB (5.4 KiB / s)
https://scala-ci.typesafe.com/artifactory/scala-pr-validation-snapshots/org/scala-lang/scala-reflect/2.12.12-bin-8d19363-SNAPSHOT/scala-reflect-2.12.12-bin-8d19363-20200513.0950…
100.0% [##########] 1.8 KiB (10.1 KiB / s)
[info] Resolved dependencies
[info] Fetching artifacts of
https://scala-ci.typesafe.com/artifactory/scala-pr-validation-snapshots/org/scala-lang/scala-reflect/2.12.12-bin-8d19363-SNAPSHOT/scala-reflect-2.12.12-bin-8d19363-20200513.0950…
100.0% [##########] 3.5 MiB (962.1 KiB / s)
https://scala-ci.typesafe.com/artifactory/scala-pr-validation-snapshots/org/scala-lang/scala-compiler/2.12.12-bin-8d19363-SNAPSHOT/scala-compiler-2.12.12-bin-8d19363-20200513.09…
100.0% [##########] 10.0 MiB (2.0 MiB / s)
https://scala-ci.typesafe.com/artifactory/scala-pr-validation-snapshots/org/scala-lang/scala-library/2.12.12-bin-8d19363-SNAPSHOT/scala-library-2.12.12-bin-8d19363-20200513.0950…
100.0% [##########] 5.2 MiB (742.2 KiB / s)
[info] Fetched artifacts of
[warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
[info] Compiling 9 Java sources to /Users/jz/code/compiler-benchmark/infrastructure/target/classes ...
[warn] Multiple main classes detected. Run 'show discoveredMainClasses' to see the list
[info] Compiling 4 Scala sources and 3 Java sources to /Users/jz/code/compiler-benchmark/compilation/target/scala-2.12/classes ...
[info] Non-compiled module 'compiler-bridge_2.12' for Scala 2.12.12-bin-8d19363-SNAPSHOT. Compiling...
[info] Compilation completed in 7.336s.
[warn] /Users/jz/code/compiler-benchmark/compilation/src/main/scala/scala/tools/nsc/HotSbtBenchmark.scala:7:36: imported `Mode' is permanently hidden by definition of value Mode in package nsc
[warn] import org.openjdk.jmh.annotations.Mode
[warn] ^
[warn] /Users/jz/code/compiler-benchmark/compilation/src/main/scala/scala/tools/nsc/BenchmarkUtils.scala:50:62: match may not be exhaustive.
[warn] It would fail on the following input: Nil
[warn] val (cond, expr) = trimmedLine.split(" ").toList.filter(_.nonEmpty) match {
[warn] ^
[warn] two warnings found
[info] /Users/jz/code/compiler-benchmark/compilation/src/main/scala/scala/tools/nsc/JavacBenchmark.java: Some input files use unchecked or unsafe operations.
[info] /Users/jz/code/compiler-benchmark/compilation/src/main/scala/scala/tools/nsc/JavacBenchmark.java: Recompile with -Xlint:unchecked for details.
[info] Compiling 1 Scala source to /Users/jz/code/compiler-benchmark/compilation/target/scala-2.12/test-classes ...
Processing 24 classes from /Users/jz/code/compiler-benchmark/compilation/target/scala-2.12/classes with "reflection" generator
Writing out Java source to /Users/jz/code/compiler-benchmark/compilation/target/scala-2.12/src_managed/jmh and resources to /Users/jz/code/compiler-benchmark/compilation/target/scala-2.12/resource_managed/jmh
[info] Compiling 25 Java sources to /Users/jz/code/compiler-benchmark/compilation/target/scala-2.12/classes ...
[info] running (fork) scala.bench.ScalacBenchmarkRunner HotScalacBenchmark -foe true -psource=scalap -prof gc -wi 3 -i 3 -f 2
[info] # JMH version: 1.21
[info] # VM version: JDK 1.8.0_172, Java HotSpot(TM) 64-Bit Server VM, 25.172-b11
[info] # VM invoker: /Users/jz/.jabba/jdk/1.8.172/Contents/Home/jre/bin/java
[info] # VM options: -Xms2G -Xmx2G -Xss2M
[info] # Warmup: 3 iterations, 10 s each
[info] # Measurement: 3 iterations, 10 s each
[info] # Timeout: 10 min per iteration
[info] # Threads: 1 thread, will synchronize iterations
[info] # Benchmark mode: Sampling time
[info] # Benchmark: scala.tools.nsc.HotScalacBenchmark.compile
[info] # Parameters: (corpusPath = ../corpus, corpusVersion = a8c43dc, extraArgs = , resident = false, scalaVersion = 2.12.12-bin-8d19363-SNAPSHOT, source = scalap)
[info] # Run progress: 0.00% complete, ETA 00:02:00
[info] # Fork: 1 of 2
[info] # Warmup Iteration 1: 5664.408 ms/op
[info] # Warmup Iteration 2: 1457.820 ±(99.9%) 663.928 ms/op
[info] # Warmup Iteration 3: 938.571 ±(99.9%) 134.048 ms/op
[info] Iteration 1: 806.355 ±(99.9%) 82.575 ms/op
[info] compile·p0.00: 749.732 ms/op
[info] compile·p0.50: 778.043 ms/op
[info] compile·p0.90: 952.946 ms/op
[info] compile·p0.95: 994.050 ms/op
[info] compile·p0.99: 994.050 ms/op
[info] compile·p0.999: 994.050 ms/op
[info] compile·p0.9999: 994.050 ms/op
[info] compile·p1.00: 994.050 ms/op
[info] ·gc.alloc.rate: 223.002 MB/sec
[info] ·gc.alloc.rate.norm: 197759326.154 B/op
[info] ·gc.churn.PS_Eden_Space: 195.918 MB/sec
[info] ·gc.churn.PS_Eden_Space.norm: 173740977.231 B/op
[info] ·gc.churn.PS_Survivor_Space: 1.578 MB/sec
[info] ·gc.churn.PS_Survivor_Space.norm: 1399284.308 B/op
[info] ·gc.count: 4.000 counts
[info] ·gc.time: 67.000 ms
[info] Iteration 2: 716.177 ±(99.9%) 32.217 ms/op
[info] compile·p0.00: 677.380 ms/op
[info] compile·p0.50: 720.372 ms/op
[info] compile·p0.90: 752.878 ms/op
[info] compile·p0.95: 758.120 ms/op
[info] compile·p0.99: 758.120 ms/op
[info] compile·p0.999: 758.120 ms/op
[info] compile·p0.9999: 758.120 ms/op
[info] compile·p1.00: 758.120 ms/op
[info] ·gc.alloc.rate: 249.957 MB/sec
[info] ·gc.alloc.rate.norm: 197302521.714 B/op
[info] ·gc.churn.PS_Eden_Space: 257.712 MB/sec
[info] ·gc.churn.PS_Eden_Space.norm: 203423744.000 B/op
[info] ·gc.churn.PS_Survivor_Space: 0.696 MB/sec
[info] ·gc.churn.PS_Survivor_Space.norm: 549492.000 B/op
[info] ·gc.count: 5.000 counts
[info] ·gc.time: 109.000 ms
[info] Iteration 3: 697.303 ±(99.9%) 37.851 ms/op
[info] compile·p0.00: 659.554 ms/op
[info] compile·p0.50: 677.380 ms/op
[info] compile·p0.90: 747.635 ms/op
[info] compile·p0.95: 750.780 ms/op
[info] compile·p0.99: 750.780 ms/op
[info] compile·p0.999: 750.780 ms/op
[info] compile·p0.9999: 750.780 ms/op
[info] compile·p1.00: 750.780 ms/op
[info] ·gc.alloc.rate: 256.446 MB/sec
[info] ·gc.alloc.rate.norm: 197025428.267 B/op
[info] ·gc.churn.PS_Eden_Space: 245.303 MB/sec
[info] ·gc.churn.PS_Eden_Space.norm: 188464059.733 B/op
[info] ·gc.churn.PS_Survivor_Space: 2.931 MB/sec
[info] ·gc.churn.PS_Survivor_Space.norm: 2251699.200 B/op
[info] ·gc.count: 5.000 counts
[info] ·gc.time: 62.000 ms
[info] # Run progress: 50.00% complete, ETA 00:01:06
[info] # Fork: 2 of 2
[info] # Warmup Iteration 1: 5234.491 ms/op
[info] # Warmup Iteration 2: 1716.868 ±(99.9%) 2397.516 ms/op
[info] # Warmup Iteration 3: 928.848 ±(99.9%) 169.490 ms/op
[info] Iteration 1: 763.588 ±(99.9%) 25.003 ms/op
[info] compile·p0.00: 720.372 ms/op
[info] compile·p0.50: 762.839 ms/op
[info] compile·p0.90: 799.539 ms/op
[info] compile·p0.95: 810.549 ms/op
[info] compile·p0.99: 810.549 ms/op
[info] compile·p0.999: 810.549 ms/op
[info] compile·p0.9999: 810.549 ms/op
[info] compile·p1.00: 810.549 ms/op
[info] ·gc.alloc.rate: 233.588 MB/sec
[info] ·gc.alloc.rate.norm: 196011016.000 B/op
[info] ·gc.churn.PS_Eden_Space: 210.557 MB/sec
[info] ·gc.churn.PS_Eden_Space.norm: 176685056.000 B/op
[info] ·gc.count: 4.000 counts
[info] ·gc.time: 58.000 ms
[info] Iteration 2: 734.602 ±(99.9%) 25.772 ms/op
[info] compile·p0.00: 705.692 ms/op
[info] compile·p0.50: 734.527 ms/op
[info] compile·p0.90: 772.801 ms/op
[info] compile·p0.95: 794.821 ms/op
[info] compile·p0.99: 794.821 ms/op
[info] compile·p0.999: 794.821 ms/op
[info] compile·p0.9999: 794.821 ms/op
[info] compile·p1.00: 794.821 ms/op
[info] ·gc.alloc.rate: 240.499 MB/sec
[info] ·gc.alloc.rate.norm: 194555621.143 B/op
[info] ·gc.churn.PS_Eden_Space: 275.116 MB/sec
[info] ·gc.churn.PS_Eden_Space.norm: 222560256.000 B/op
[info] ·gc.churn.PS_Survivor_Space: 1.055 MB/sec
[info] ·gc.churn.PS_Survivor_Space.norm: 853826.286 B/op
[info] ·gc.count: 5.000 counts
[info] ·gc.time: 61.000 ms
[info] Iteration 3: 702.196 ±(99.9%) 35.191 ms/op
[info] compile·p0.00: 656.409 ms/op
[info] compile·p0.50: 702.546 ms/op
[info] compile·p0.90: 744.908 ms/op
[info] compile·p0.95: 758.120 ms/op
[info] compile·p0.99: 758.120 ms/op
[info] compile·p0.999: 758.120 ms/op
[info] compile·p0.9999: 758.120 ms/op
[info] compile·p1.00: 758.120 ms/op
[info] ·gc.alloc.rate: 251.097 MB/sec
[info] ·gc.alloc.rate.norm: 194303786.667 B/op
[info] ·gc.churn.PS_Eden_Space: 215.546 MB/sec
[info] ·gc.churn.PS_Eden_Space.norm: 166793489.067 B/op
[info] ·gc.churn.PS_Survivor_Space: 0.988 MB/sec
[info] ·gc.churn.PS_Survivor_Space.norm: 764699.733 B/op
[info] ·gc.count: 4.000 counts
[info] ·gc.time: 47.000 ms
[info] Result "scala.tools.nsc.HotScalacBenchmark.compile":
[info] N = 85
[info] mean = 735.015 ±(99.9%) 19.506 ms/op
[info] Histogram, ms/op:
[info] [ 600.000, 625.000) = 0
[info] [ 625.000, 650.000) = 0
[info] [ 650.000, 675.000) = 10
[info] [ 675.000, 700.000) = 13
[info] [ 700.000, 725.000) = 10
[info] [ 725.000, 750.000) = 25
[info] [ 750.000, 775.000) = 15
[info] [ 775.000, 800.000) = 6
[info] [ 800.000, 825.000) = 3
[info] [ 825.000, 850.000) = 1
[info] [ 850.000, 875.000) = 0
[info] [ 875.000, 900.000) = 1
[info] [ 900.000, 925.000) = 0
[info] [ 925.000, 950.000) = 0
[info] [ 950.000, 975.000) = 0
[info] Percentiles, ms/op:
[info] p(0.0000) = 656.409 ms/op
[info] p(50.0000) = 739.246 ms/op
[info] p(90.0000) = 788.949 ms/op
[info] p(95.0000) = 810.235 ms/op
[info] p(99.0000) = 994.050 ms/op
[info] p(99.9000) = 994.050 ms/op
[info] p(99.9900) = 994.050 ms/op
[info] p(99.9990) = 994.050 ms/op
[info] p(99.9999) = 994.050 ms/op
[info] p(100.0000) = 994.050 ms/op
[info] Secondary result "scala.tools.nsc.HotScalacBenchmark.compile:·gc.alloc.rate":
[info] 242.432 ±(99.9%) 35.194 MB/sec [Average]
[info] (min, avg, max) = (223.002, 242.432, 256.446), stdev = 12.551
[info] CI (99.9%): [207.238, 277.626] (assumes normal distribution)
[info] Secondary result "scala.tools.nsc.HotScalacBenchmark.compile:·gc.alloc.rate.norm":
[info] 196159616.657 ±(99.9%) 4093329.085 B/op [Average]
[info] (min, avg, max) = (194303786.667, 196159616.657, 197759326.154), stdev = 1459720.583
[info] CI (99.9%): [192066287.572, 200252945.743] (assumes normal distribution)
[info] Secondary result "scala.tools.nsc.HotScalacBenchmark.compile:·gc.churn.PS_Eden_Space":
[info] 233.359 ±(99.9%) 86.144 MB/sec [Average]
[info] (min, avg, max) = (195.918, 233.359, 275.116), stdev = 30.720
[info] CI (99.9%): [147.215, 319.502] (assumes normal distribution)
[info] Secondary result "scala.tools.nsc.HotScalacBenchmark.compile:·gc.churn.PS_Eden_Space.norm":
[info] 188611263.672 ±(99.9%) 58972526.370 B/op [Average]
[info] (min, avg, max) = (166793489.067, 188611263.672, 222560256.000), stdev = 21030170.991
[info] CI (99.9%): [129638737.302, 247583790.042] (assumes normal distribution)
[info] Secondary result "scala.tools.nsc.HotScalacBenchmark.compile:·gc.churn.PS_Survivor_Space":
[info] 1.208 ±(99.9%) 2.775 MB/sec [Average]
[info] (min, avg, max) = (≈ 0, 1.208, 2.931), stdev = 0.990
[info] CI (99.9%): [≈ 0, 3.983] (assumes normal distribution)
[info] Secondary result "scala.tools.nsc.HotScalacBenchmark.compile:·gc.churn.PS_Survivor_Space.norm":
[info] 969833.588 ±(99.9%) 2172261.832 B/op [Average]
[info] (min, avg, max) = (≈ 0, 969833.588, 2251699.200), stdev = 774649.495
[info] CI (99.9%): [≈ 0, 3142095.420] (assumes normal distribution)
[info] Secondary result "scala.tools.nsc.HotScalacBenchmark.compile:·gc.count":
[info] 27.000 ±(99.9%) 0.001 counts [Sum]
[info] (min, avg, max) = (4.000, 4.500, 5.000), stdev = 0.548
[info] CI (99.9%): [27.000, 27.000] (assumes normal distribution)
[info] Secondary result "scala.tools.nsc.HotScalacBenchmark.compile:·gc.time":
[info] 404.000 ±(99.9%) 0.001 ms [Sum]
[info] (min, avg, max) = (47.000, 67.333, 109.000), stdev = 21.472
[info] CI (99.9%): [404.000, 404.000] (assumes normal distribution)
[info] # Run complete. Total time: 00:02:12
[info] REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
[info] why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
[info] experiments, perform baseline and negative tests that provide experimental control, make sure
[info] the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts.
[info] Do not assume the numbers tell you what you want them to tell.
[info] Benchmark (corpusPath) (corpusVersion) (extraArgs) (resident) (scalaVersion) (source) Mode Cnt Score Error Units
[info] HotScalacBenchmark.compile ../corpus a8c43dc false 2.12.12-bin-8d19363-SNAPSHOT scalap sample 85 735.015 ± 19.506 ms/op
[info] HotScalacBenchmark.compile:compile·p0.00 ../corpus a8c43dc false 2.12.12-bin-8d19363-SNAPSHOT scalap sample 656.409 ms/op
[info] HotScalacBenchmark.compile:compile·p0.50 ../corpus a8c43dc false 2.12.12-bin-8d19363-SNAPSHOT scalap sample 739.246 ms/op
[info] HotScalacBenchmark.compile:compile·p0.90 ../corpus a8c43dc false 2.12.12-bin-8d19363-SNAPSHOT scalap sample 788.949 ms/op
[info] HotScalacBenchmark.compile:compile·p0.95 ../corpus a8c43dc false 2.12.12-bin-8d19363-SNAPSHOT scalap sample 810.235 ms/op
[info] HotScalacBenchmark.compile:compile·p0.99 ../corpus a8c43dc false 2.12.12-bin-8d19363-SNAPSHOT scalap sample 994.050 ms/op
[info] HotScalacBenchmark.compile:compile·p0.999 ../corpus a8c43dc false 2.12.12-bin-8d19363-SNAPSHOT scalap sample 994.050 ms/op
[info] HotScalacBenchmark.compile:compile·p0.9999 ../corpus a8c43dc false 2.12.12-bin-8d19363-SNAPSHOT scalap sample 994.050 ms/op
[info] HotScalacBenchmark.compile:compile·p1.00 ../corpus a8c43dc false 2.12.12-bin-8d19363-SNAPSHOT scalap sample 994.050 ms/op
[info] HotScalacBenchmark.compile:·gc.alloc.rate ../corpus a8c43dc false 2.12.12-bin-8d19363-SNAPSHOT scalap sample 6 242.432 ± 35.194 MB/sec
[info] HotScalacBenchmark.compile:·gc.alloc.rate.norm ../corpus a8c43dc false 2.12.12-bin-8d19363-SNAPSHOT scalap sample 6 196159616.657 ± 4093329.085 B/op
[info] HotScalacBenchmark.compile:·gc.churn.PS_Eden_Space ../corpus a8c43dc false 2.12.12-bin-8d19363-SNAPSHOT scalap sample 6 233.359 ± 86.144 MB/sec
[info] HotScalacBenchmark.compile:·gc.churn.PS_Eden_Space.norm ../corpus a8c43dc false 2.12.12-bin-8d19363-SNAPSHOT scalap sample 6 188611263.672 ± 58972526.370 B/op
[info] HotScalacBenchmark.compile:·gc.churn.PS_Survivor_Space ../corpus a8c43dc false 2.12.12-bin-8d19363-SNAPSHOT scalap sample 6 1.208 ± 2.775 MB/sec
[info] HotScalacBenchmark.compile:·gc.churn.PS_Survivor_Space.norm ../corpus a8c43dc false 2.12.12-bin-8d19363-SNAPSHOT scalap sample 6 969833.588 ± 2172261.832 B/op
[info] HotScalacBenchmark.compile:·gc.count ../corpus a8c43dc false 2.12.12-bin-8d19363-SNAPSHOT scalap sample 6 27.000 counts
[info] HotScalacBenchmark.compile:·gc.time ../corpus a8c43dc false 2.12.12-bin-8d19363-SNAPSHOT scalap sample 6 404.000 ms
[success] Total time: 168 s (02:48), completed 19/05/2020 10:26:41 AM
( for v in 2.12.11 2.12.12-bin-8d19363-SNAPSHOT; do; echo $v; sbt clean ; ; ) 816.32s user 26.54s system 255% cpu 5:30.51 total
➜ compiler-benchmark git:(topic/case-class-o-rama)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment