Skip to content

Instantly share code, notes, and snippets.

@meiersi
Created April 24, 2012 19:39
Show Gist options
  • Save meiersi/2483031 to your computer and use it in GitHub Desktop.
Save meiersi/2483031 to your computer and use it in GitHub Desktop.
Benchmark results from comparing cereal, binary, and the new bytestring builder on https://github.com/meiersi/cereal/blob/master/bench/BenchAll.hs
└─╼ ./cabal-dev/bin/cereal-bench
Sanity checks:
lengths of input data: [11,11,11,11,11,11,11]
cereal :/[Int]: 96
binary :/[Int]: 96
new builder :/[Int]: 100
new builder2:/[Int]: 96
cereal :/[String]: 138
binary :/[String]: 138
new builder :/[String]: 121
new builder2:/[String]: 138
warming up
estimating clock resolution...
mean is 1.853680 us (320001 iterations)
found 92892 outliers among 319999 samples (29.0%)
77699 (24.3%) low severe
15193 (4.7%) high severe
estimating cost of a clock call...
mean is 33.14180 ns (17 iterations)
found 2 outliers among 17 samples (11.8%)
1 (5.9%) high mild
1 (5.9%) high severe
benchmarking cereal /[Int] (10)
mean: 1.095145 us, lb 1.080124 us, ub 1.129459 us, ci 0.950
std dev: 110.1799 ns, lb 44.73199 ns, ub 193.3570 ns, ci 0.950
found 12 outliers among 100 samples (12.0%)
7 (7.0%) high mild
5 (5.0%) high severe
variance introduced by outliers: 79.012%
variance is severely inflated by outliers
benchmarking cereal /[String] (10)
mean: 3.912284 us, lb 3.901959 us, ub 3.924826 us, ci 0.950
std dev: 57.95188 ns, lb 48.17431 ns, ub 72.43770 ns, ci 0.950
found 6 outliers among 100 samples (6.0%)
5 (5.0%) high mild
1 (1.0%) high severe
variance introduced by outliers: 7.544%
variance is slightly inflated by outliers
benchmarking binary /[Int] (10)
mean: 1.076071 us, lb 1.072468 us, ub 1.080499 us, ci 0.950
std dev: 20.56727 ns, lb 17.17976 ns, ub 25.12927 ns, ci 0.950
found 5 outliers among 100 samples (5.0%)
5 (5.0%) high mild
variance introduced by outliers: 12.277%
variance is moderately inflated by outliers
benchmarking binary /[String] (10)
mean: 4.386507 us, lb 4.374040 us, ub 4.401063 us, ci 0.950
std dev: 68.97067 ns, lb 58.67120 ns, ub 84.24516 ns, ci 0.950
found 3 outliers among 100 samples (3.0%)
3 (3.0%) high mild
variance introduced by outliers: 8.488%
variance is slightly inflated by outliers
benchmarking new builder/[Int] (10)
mean: 386.2727 ns, lb 385.3292 ns, ub 387.3434 ns, ci 0.950
std dev: 5.154584 ns, lb 4.485224 ns, ub 6.085684 ns, ci 0.950
found 4 outliers among 100 samples (4.0%)
4 (4.0%) high mild
variance introduced by outliers: 6.570%
variance is slightly inflated by outliers
benchmarking new builder/[String] (10)
mean: 821.5591 ns, lb 818.8620 ns, ub 824.6918 ns, ci 0.950
std dev: 14.82703 ns, lb 12.72683 ns, ub 17.81039 ns, ci 0.950
found 5 outliers among 100 samples (5.0%)
5 (5.0%) high mild
variance introduced by outliers: 10.420%
variance is moderately inflated by outliers
benchmarking new builder2/[Int] (10)
mean: 523.7290 ns, lb 517.9199 ns, ub 533.1015 ns, ci 0.950
std dev: 37.02043 ns, lb 25.23230 ns, ub 50.96788 ns, ci 0.950
found 12 outliers among 100 samples (12.0%)
5 (5.0%) high mild
7 (7.0%) high severe
variance introduced by outliers: 65.596%
variance is severely inflated by outliers
benchmarking new builder2/[String] (10)
mean: 1.106214 us, lb 1.098262 us, ub 1.119187 us, ci 0.950
std dev: 51.01611 ns, lb 34.40537 ns, ub 73.63724 ns, ci 0.950
found 7 outliers among 100 samples (7.0%)
6 (6.0%) high severe
variance introduced by outliers: 44.445%
variance is moderately inflated by outliers
└─╼ ./cabal-dev/bin/cereal-bench
Sanity checks:
lengths of input data: [101,101,101,101,101,101,101]
cereal :/[Int]: 816
binary :/[Int]: 816
new builder :/[Int]: 910
new builder2:/[Int]: 816
cereal :/[String]: 1308
binary :/[String]: 1308
new builder :/[String]: 1201
new builder2:/[String]: 1308
warming up
estimating clock resolution...
mean is 1.821400 us (320001 iterations)
found 82596 outliers among 319999 samples (25.8%)
76144 (23.8%) low severe
6452 (2.0%) high severe
estimating cost of a clock call...
mean is 32.56641 ns (18 iterations)
benchmarking cereal /[Int] (100)
mean: 6.325782 us, lb 6.085083 us, ub 6.930631 us, ci 0.950
std dev: 1.725726 us, lb 123.0239 ns, ub 3.192429 us, ci 0.950
found 10 outliers among 100 samples (10.0%)
7 (7.0%) high mild
3 (3.0%) high severe
variance introduced by outliers: 96.802%
variance is severely inflated by outliers
benchmarking cereal /[String] (100)
mean: 35.48248 us, lb 35.18913 us, ub 36.12428 us, ci 0.950
std dev: 2.124618 us, lb 1.139368 us, ub 3.824147 us, ci 0.950
found 6 outliers among 100 samples (6.0%)
3 (3.0%) high mild
3 (3.0%) high severe
variance introduced by outliers: 57.483%
variance is severely inflated by outliers
benchmarking binary /[Int] (100)
mean: 6.969270 us, lb 6.726577 us, ub 7.307411 us, ci 0.950
std dev: 1.458724 us, lb 1.124309 us, ub 1.794970 us, ci 0.950
found 15 outliers among 100 samples (15.0%)
14 (14.0%) high severe
variance introduced by outliers: 94.678%
variance is severely inflated by outliers
benchmarking binary /[String] (100)
mean: 39.44901 us, lb 39.17077 us, ub 39.89830 us, ci 0.950
std dev: 1.783504 us, lb 1.226623 us, ub 2.534307 us, ci 0.950
found 7 outliers among 100 samples (7.0%)
6 (6.0%) high severe
variance introduced by outliers: 43.452%
variance is moderately inflated by outliers
benchmarking new builder/[Int] (100)
mean: 976.3339 ns, lb 963.8796 ns, ub 1.003989 us, ci 0.950
std dev: 89.87930 ns, lb 48.97444 ns, ub 163.7649 ns, ci 0.950
found 5 outliers among 100 samples (5.0%)
5 (5.0%) high severe
variance introduced by outliers: 75.907%
variance is severely inflated by outliers
benchmarking new builder/[String] (100)
mean: 5.368721 us, lb 5.303101 us, ub 5.493326 us, ci 0.950
std dev: 448.4484 ns, lb 270.2259 ns, ub 689.1666 ns, ci 0.950
found 8 outliers among 100 samples (8.0%)
2 (2.0%) high mild
6 (6.0%) high severe
variance introduced by outliers: 72.774%
variance is severely inflated by outliers
benchmarking new builder2/[Int] (100)
mean: 1.925216 us, lb 1.911227 us, ub 1.951544 us, ci 0.950
std dev: 95.52214 ns, lb 57.41560 ns, ub 151.2901 ns, ci 0.950
found 6 outliers among 100 samples (6.0%)
5 (5.0%) high severe
variance introduced by outliers: 47.476%
variance is moderately inflated by outliers
benchmarking new builder2/[String] (100)
mean: 8.936901 us, lb 8.639061 us, ub 9.296029 us, ci 0.950
std dev: 1.677134 us, lb 1.446657 us, ub 1.869163 us, ci 0.950
found 21 outliers among 100 samples (21.0%)
21 (21.0%) high mild
variance introduced by outliers: 93.608%
variance is severely inflated by outliers
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment