Skip to content

Instantly share code, notes, and snippets.

@miguno

miguno/benchmark.md

Last active August 29, 2015 14:09
Show Gist options
  • Save miguno/26a93e6a816e377b011b to your computer and use it in GitHub Desktop.
Save miguno/26a93e6a816e377b011b to your computer and use it in GitHub Desktop.
Download ZIP
@specialized(Int, Long) K benchmark
Raw
benchmark.md

See the discussion in Algebird pull request #354.

> cappi::benchmarkOnly com.twitter.algebird.caliper.CMSBenchmark

Summary

The benchmark results indicate that @specialized(Int, Long) K does not improve the CMS[K] performance for Long.

Before @specialized, for Long:

[info]                               benchmark   eps   us linear runtime
[info]   PlusOfFirstHundredIntegersWithLongCms   0.1  277 ==
[info]   PlusOfFirstHundredIntegersWithLongCms   0.1  272 ==
[info]   PlusOfFirstHundredIntegersWithLongCms   0.1  269 ==
[info]   PlusOfFirstHundredIntegersWithLongCms 0.005  317 ==
[info]   PlusOfFirstHundredIntegersWithLongCms 0.005  319 ==
[info]   PlusOfFirstHundredIntegersWithLongCms 0.005  318 ==

After @specialized, for Long:

[info]   PlusOfFirstHundredIntegersWithLongCms   0.1  274 ==
[info]   PlusOfFirstHundredIntegersWithLongCms   0.1  274 ==
[info]   PlusOfFirstHundredIntegersWithLongCms   0.1  274 ==
[info]   PlusOfFirstHundredIntegersWithLongCms 0.005  325 ==
[info]   PlusOfFirstHundredIntegersWithLongCms 0.005  326 ==
[info]   PlusOfFirstHundredIntegersWithLongCms 0.005  325 ==

Benchmark results

Note: When comparing the results of before/after @specialized, you should pay most of your attention to those tests that run against a Long data type. The tests for BigInt should be unaffected by the @specialized code change.

Results without @specialized(Int, Long) K

3 sequential runs.

[info] Running com.google.caliper.Runner com.twitter.algebird.caliper.CMSBenchmark
[info]  0% Scenario{vm=java, trial=0, benchmark=PlusOfFirstHundredIntegersWithLongCms, delta=0.0000001, eps=0.1, heavyHittersPct=0.2, maxBits=2048, operations=100} 276992.64 ns; σ=3873.83 ns @ 10 trials
[info] 17% Scenario{vm=java, trial=0, benchmark=PlusOfFirstHundredIntegersWithBigIntCms, delta=0.0000001, eps=0.1, heavyHittersPct=0.2, maxBits=2048, operations=100} 825164.73 ns; σ=12521.97 ns @ 10 trials
[info] 33% Scenario{vm=java, trial=0, benchmark=PlusOfRandom2048BitNumbersWithBigIntCms, delta=0.0000001, eps=0.1, heavyHittersPct=0.2, maxBits=2048, operations=100} 3325663.93 ns; σ=50746.06 ns @ 10 trials
[info] 50% Scenario{vm=java, trial=0, benchmark=PlusOfFirstHundredIntegersWithLongCms, delta=0.0000001, eps=0.005, heavyHittersPct=0.2, maxBits=2048, operations=100} 317116.51 ns; σ=4727.77 ns @ 10 trials
[info] 67% Scenario{vm=java, trial=0, benchmark=PlusOfFirstHundredIntegersWithBigIntCms, delta=0.0000001, eps=0.005, heavyHittersPct=0.2, maxBits=2048, operations=100} 892577.86 ns; σ=11302.40 ns @ 10 trials
[info] 83% Scenario{vm=java, trial=0, benchmark=PlusOfRandom2048BitNumbersWithBigIntCms, delta=0.0000001, eps=0.005, heavyHittersPct=0.2, maxBits=2048, operations=100} 3402344.71 ns; σ=38176.90 ns @ 10 trials
[info]
[info]                               benchmark   eps   us linear runtime
[info]   PlusOfFirstHundredIntegersWithLongCms   0.1  277 ==
[info]   PlusOfFirstHundredIntegersWithLongCms 0.005  317 ==
[info] PlusOfFirstHundredIntegersWithBigIntCms   0.1  825 =======
[info] PlusOfFirstHundredIntegersWithBigIntCms 0.005  893 =======
[info] PlusOfRandom2048BitNumbersWithBigIntCms   0.1 3326 =============================
[info] PlusOfRandom2048BitNumbersWithBigIntCms 0.005 3402 ==============================
[info]
[info] vm: java
[info] trial: 0
[info] delta: 0.0000001
[info] heavyHittersPct: 0.2
[info] maxBits: 2048
[info] operations: 100
[success] Total time: 149 s, completed Nov 18, 2014 2:08:17 PM

[info] Running com.google.caliper.Runner com.twitter.algebird.caliper.CMSBenchmark
[info]  0% Scenario{vm=java, trial=0, benchmark=PlusOfFirstHundredIntegersWithLongCms, delta=0.0000001, eps=0.1, heavyHittersPct=0.2, maxBits=2048, operations=100} 272064.60 ns; σ=2249.85 ns @ 3 trials
[info] 17% Scenario{vm=java, trial=0, benchmark=PlusOfFirstHundredIntegersWithBigIntCms, delta=0.0000001, eps=0.1, heavyHittersPct=0.2, maxBits=2048, operations=100} 812068.05 ns; σ=15115.92 ns @ 10 trials
[info] 33% Scenario{vm=java, trial=0, benchmark=PlusOfRandom2048BitNumbersWithBigIntCms, delta=0.0000001, eps=0.1, heavyHittersPct=0.2, maxBits=2048, operations=100} 3335493.24 ns; σ=15680.39 ns @ 3 trials
[info] 50% Scenario{vm=java, trial=0, benchmark=PlusOfFirstHundredIntegersWithLongCms, delta=0.0000001, eps=0.005, heavyHittersPct=0.2, maxBits=2048, operations=100} 318820.63 ns; σ=2289.77 ns @ 3 trials
[info] 67% Scenario{vm=java, trial=0, benchmark=PlusOfFirstHundredIntegersWithBigIntCms, delta=0.0000001, eps=0.005, heavyHittersPct=0.2, maxBits=2048, operations=100} 872806.78 ns; σ=15328.25 ns @ 10 trials
[info] 83% Scenario{vm=java, trial=0, benchmark=PlusOfRandom2048BitNumbersWithBigIntCms, delta=0.0000001, eps=0.005, heavyHittersPct=0.2, maxBits=2048, operations=100} 3375872.20 ns; σ=19879.84 ns @ 3 trials
[info]
[info]                               benchmark   eps   us linear runtime
[info]   PlusOfFirstHundredIntegersWithLongCms   0.1  272 ==
[info]   PlusOfFirstHundredIntegersWithLongCms 0.005  319 ==
[info] PlusOfFirstHundredIntegersWithBigIntCms   0.1  812 =======
[info] PlusOfFirstHundredIntegersWithBigIntCms 0.005  873 =======
[info] PlusOfRandom2048BitNumbersWithBigIntCms   0.1 3335 =============================
[info] PlusOfRandom2048BitNumbersWithBigIntCms 0.005 3376 ==============================
[info]
[info] vm: java
[info] trial: 0
[info] delta: 0.0000001
[info] heavyHittersPct: 0.2
[info] maxBits: 2048
[info] operations: 100
[success] Total time: 67 s, completed Nov 18, 2014 2:15:18 PM

[info] Running com.google.caliper.Runner com.twitter.algebird.caliper.CMSBenchmark
[info]  0% Scenario{vm=java, trial=0, benchmark=PlusOfFirstHundredIntegersWithLongCms, delta=0.0000001, eps=0.1, heavyHittersPct=0.2, maxBits=2048, operations=100} 268911.09 ns; σ=6716.42 ns @ 10 trials
[info] 17% Scenario{vm=java, trial=0, benchmark=PlusOfFirstHundredIntegersWithBigIntCms, delta=0.0000001, eps=0.1, heavyHittersPct=0.2, maxBits=2048, operations=100} 815865.57 ns; σ=4878.40 ns @ 3 trials
[info] 33% Scenario{vm=java, trial=0, benchmark=PlusOfRandom2048BitNumbersWithBigIntCms, delta=0.0000001, eps=0.1, heavyHittersPct=0.2, maxBits=2048, operations=100} 3304103.59 ns; σ=30719.71 ns @ 3 trials
[info] 50% Scenario{vm=java, trial=0, benchmark=PlusOfFirstHundredIntegersWithLongCms, delta=0.0000001, eps=0.005, heavyHittersPct=0.2, maxBits=2048, operations=100} 317790.14 ns; σ=4018.52 ns @ 10 trials
[info] 67% Scenario{vm=java, trial=0, benchmark=PlusOfFirstHundredIntegersWithBigIntCms, delta=0.0000001, eps=0.005, heavyHittersPct=0.2, maxBits=2048, operations=100} 873207.66 ns; σ=4258.53 ns @ 3 trials
[info] 83% Scenario{vm=java, trial=0, benchmark=PlusOfRandom2048BitNumbersWithBigIntCms, delta=0.0000001, eps=0.005, heavyHittersPct=0.2, maxBits=2048, operations=100} 3362244.90 ns; σ=23047.83 ns @ 3 trials
[info]
[info]                               benchmark   eps   us linear runtime
[info]   PlusOfFirstHundredIntegersWithLongCms   0.1  269 ==
[info]   PlusOfFirstHundredIntegersWithLongCms 0.005  318 ==
[info] PlusOfFirstHundredIntegersWithBigIntCms   0.1  816 =======
[info] PlusOfFirstHundredIntegersWithBigIntCms 0.005  873 =======
[info] PlusOfRandom2048BitNumbersWithBigIntCms   0.1 3304 =============================
[info] PlusOfRandom2048BitNumbersWithBigIntCms 0.005 3362 ==============================
[info]
[info] vm: java
[info] trial: 0
[info] delta: 0.0000001
[info] heavyHittersPct: 0.2
[info] maxBits: 2048
[info] operations: 100
[success] Total time: 60 s, completed Nov 18, 2014 2:16:56 PM

Results with @specialized(Int, Long) K

3 sequential runs.

[info] Running com.google.caliper.Runner com.twitter.algebird.caliper.CMSBenchmark
[info]  0% Scenario{vm=java, trial=0, benchmark=PlusOfFirstHundredIntegersWithLongCms, delta=0.0000001, eps=0.1, heavyHittersPct=0.2, maxBits=2048, operations=100} 273572.13 ns; σ=2688.92 ns @ 6 trials
[info] 17% Scenario{vm=java, trial=0, benchmark=PlusOfFirstHundredIntegersWithBigIntCms, delta=0.0000001, eps=0.1, heavyHittersPct=0.2, maxBits=2048, operations=100} 813633.76 ns; σ=13666.78 ns @ 10 trials
[info] 33% Scenario{vm=java, trial=0, benchmark=PlusOfRandom2048BitNumbersWithBigIntCms, delta=0.0000001, eps=0.1, heavyHittersPct=0.2, maxBits=2048, operations=100} 3372615.66 ns; σ=64006.82 ns @ 10 trials
[info] 50% Scenario{vm=java, trial=0, benchmark=PlusOfFirstHundredIntegersWithLongCms, delta=0.0000001, eps=0.005, heavyHittersPct=0.2, maxBits=2048, operations=100} 324773.31 ns; σ=4187.26 ns @ 10 trials
[info] 67% Scenario{vm=java, trial=0, benchmark=PlusOfFirstHundredIntegersWithBigIntCms, delta=0.0000001, eps=0.005, heavyHittersPct=0.2, maxBits=2048, operations=100} 873970.40 ns; σ=8346.37 ns @ 3 trials
[info] 83% Scenario{vm=java, trial=0, benchmark=PlusOfRandom2048BitNumbersWithBigIntCms, delta=0.0000001, eps=0.005, heavyHittersPct=0.2, maxBits=2048, operations=100} 3382273.48 ns; σ=42497.76 ns @ 10 trials
[info]
[info]                               benchmark   eps   us linear runtime
[info]   PlusOfFirstHundredIntegersWithLongCms   0.1  274 ==
[info]   PlusOfFirstHundredIntegersWithLongCms 0.005  325 ==
[info] PlusOfFirstHundredIntegersWithBigIntCms   0.1  814 =======
[info] PlusOfFirstHundredIntegersWithBigIntCms 0.005  874 =======
[info] PlusOfRandom2048BitNumbersWithBigIntCms   0.1 3373 =============================
[info] PlusOfRandom2048BitNumbersWithBigIntCms 0.005 3382 ==============================
[info]
[info] vm: java
[info] trial: 0
[info] delta: 0.0000001
[info] heavyHittersPct: 0.2
[info] maxBits: 2048
[info] operations: 100
[success] Total time: 82 s, completed Nov 18, 2014 2:10:36 PM

[info] Running com.google.caliper.Runner com.twitter.algebird.caliper.CMSBenchmark
[info]  0% Scenario{vm=java, trial=0, benchmark=PlusOfFirstHundredIntegersWithLongCms, delta=0.0000001, eps=0.1, heavyHittersPct=0.2, maxBits=2048, operations=100} 273505.77 ns; σ=6445.85 ns @ 10 trials
[info] 17% Scenario{vm=java, trial=0, benchmark=PlusOfFirstHundredIntegersWithBigIntCms, delta=0.0000001, eps=0.1, heavyHittersPct=0.2, maxBits=2048, operations=100} 834274.95 ns; σ=30604.97 ns @ 10 trials
[info] 33% Scenario{vm=java, trial=0, benchmark=PlusOfRandom2048BitNumbersWithBigIntCms, delta=0.0000001, eps=0.1, heavyHittersPct=0.2, maxBits=2048, operations=100} 3345744.93 ns; σ=31706.78 ns @ 6 trials
[info] 50% Scenario{vm=java, trial=0, benchmark=PlusOfFirstHundredIntegersWithLongCms, delta=0.0000001, eps=0.005, heavyHittersPct=0.2, maxBits=2048, operations=100} 326366.67 ns; σ=1421.73 ns @ 3 trials
[info] 67% Scenario{vm=java, trial=0, benchmark=PlusOfFirstHundredIntegersWithBigIntCms, delta=0.0000001, eps=0.005, heavyHittersPct=0.2, maxBits=2048, operations=100} 862855.15 ns; σ=5957.43 ns @ 3 trials
[info] 83% Scenario{vm=java, trial=0, benchmark=PlusOfRandom2048BitNumbersWithBigIntCms, delta=0.0000001, eps=0.005, heavyHittersPct=0.2, maxBits=2048, operations=100} 3384490.87 ns; σ=30672.12 ns @ 3 trials
[info]
[info]                               benchmark   eps   us linear runtime
[info]   PlusOfFirstHundredIntegersWithLongCms   0.1  274 ==
[info]   PlusOfFirstHundredIntegersWithLongCms 0.005  326 ==
[info] PlusOfFirstHundredIntegersWithBigIntCms   0.1  834 =======
[info] PlusOfFirstHundredIntegersWithBigIntCms 0.005  863 =======
[info] PlusOfRandom2048BitNumbersWithBigIntCms   0.1 3346 =============================
[info] PlusOfRandom2048BitNumbersWithBigIntCms 0.005 3384 ==============================
[info]
[info] vm: java
[info] trial: 0
[info] delta: 0.0000001
[info] heavyHittersPct: 0.2
[info] maxBits: 2048
[info] operations: 100

[info] Running com.google.caliper.Runner com.twitter.algebird.caliper.CMSBenchmark
[info]  0% Scenario{vm=java, trial=0, benchmark=PlusOfFirstHundredIntegersWithLongCms, delta=0.0000001, eps=0.1, heavyHittersPct=0.2, maxBits=2048, operations=100} 274145.67 ns; σ=5527.04 ns @ 10 trials
[info] 17% Scenario{vm=java, trial=0, benchmark=PlusOfFirstHundredIntegersWithBigIntCms, delta=0.0000001, eps=0.1, heavyHittersPct=0.2, maxBits=2048, operations=100} 798378.96 ns; σ=7872.10 ns @ 3 trials
[info] 33% Scenario{vm=java, trial=0, benchmark=PlusOfRandom2048BitNumbersWithBigIntCms, delta=0.0000001, eps=0.1, heavyHittersPct=0.2, maxBits=2048, operations=100} 3297657.89 ns; σ=18137.31 ns @ 3 trials
[info] 50% Scenario{vm=java, trial=0, benchmark=PlusOfFirstHundredIntegersWithLongCms, delta=0.0000001, eps=0.005, heavyHittersPct=0.2, maxBits=2048, operations=100} 324503.06 ns; σ=8508.14 ns @ 10 trials
[info] 67% Scenario{vm=java, trial=0, benchmark=PlusOfFirstHundredIntegersWithBigIntCms, delta=0.0000001, eps=0.005, heavyHittersPct=0.2, maxBits=2048, operations=100} 883753.75 ns; σ=15380.32 ns @ 10 trials
[info] 83% Scenario{vm=java, trial=0, benchmark=PlusOfRandom2048BitNumbersWithBigIntCms, delta=0.0000001, eps=0.005, heavyHittersPct=0.2, maxBits=2048, operations=100} 3458333.33 ns; σ=69079.68 ns @ 10 trials
[info]
[info]                               benchmark   eps   us linear runtime
[info]   PlusOfFirstHundredIntegersWithLongCms   0.1  274 ==
[info]   PlusOfFirstHundredIntegersWithLongCms 0.005  325 ==
[info] PlusOfFirstHundredIntegersWithBigIntCms   0.1  798 ======
[info] PlusOfFirstHundredIntegersWithBigIntCms 0.005  884 =======
[info] PlusOfRandom2048BitNumbersWithBigIntCms   0.1 3298 ============================
[info] PlusOfRandom2048BitNumbersWithBigIntCms 0.005 3458 ==============================
[info]
[info] vm: java
[info] trial: 0
[info] delta: 0.0000001
[info] heavyHittersPct: 0.2
[info] maxBits: 2048
[info] operations: 100
[success] Total time: 75 s, completed Nov 18, 2014 2:13:17 PM
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment