pdyraga/benchmark.md

## benchmark.md

      
    Raw
  

              benchmark.md
            
          
    About the benchmarks

The Parity client benchmarks are based on the work of Martin Holst Swende done 21 July 2018 for Constantinopole. The new benchmarks have been done for 2.1.1-beta version released 20 September 2018 as well as for master branch (branch as of 25 Oct - 5a1dc3e) with bn dependency pointing to version before and after the optimalisations by Andre Silva
If we compare pre- and post- benchmarks we see there is a significant performance boost justifying reduction of gas prices for ecmul and ecadd operations. Also, pairing is ~2.82 times faster when we compare Martin's benchmark and version 2.1.1-beta what also suggests that pairing gas cost can be reduced.
Martin's benchmark (21 Jul)


Name
Gascost
Time (ns)
MGas/S
Gascost for 10MGas/S
Gascost for ECDSA eq


ecrecover
3000.0
159077.00
18.8587916544
1590.77
3000.0


Name
Gascost
Time (ns)
MGas/S
Gascost for 10MGas/S
Gascost for ECDSA eq


modexp_nagydani_1_qube
204.0
9388.00
21.7298679165
93.88
206.835773982


alt_bn128_add_cdetrio12
500.0
16894.00
29.596306381
168.94
372.207452668


alt_bn128_mul_cdetrio11
40000.0
748248.00
53.4582117159
7482.48
16485.3487655


alt_bn128_pairing_ten_point_match_1
900000.0
96298039.00
9.34598470899
962980.39
2121631.8097


2.1.1-beta (released 20 Sep)


Name
Gascost
Time (ns)
MGas/S
Gascost for 10MGas/S
Gascost for ECDSA eq


ecrecover
3000.0
116516.00
18.8587916544
1165.16
3000.0


Name
Gascost
Time (ns)
MGas/S
Gascost for 10MGas/S
Gascost for ECDSA eq


modexp_nagydani_1_qube
204.0
8188.00
24.91450904
81.88
210.8208315


alt_bn128_add_cdetrio12
500.0
5551.00
90.07386057
55.51
142.9245769


alt_bn128_mul_cdetrio11
40000.0
229342.00
174.4120135
2293.42
5904.991589


alt_bn128_pairing_ten_point_match_1
900000.0
34076695.00
26.41101198
340766.95
877390.9592


Before bn optimalization


master branch as of 25 Oct (5a1dc3e)
bn pointing at 786c0d5 (before optimalization from paritytech/bn#9)


Name
Gascost
Time (ns)
MGas/S
Gascost for 10MGas/S
Gascost for ECDSA eq


ecrecover
3000.0
113570.00
26.41542661
1135.70
3000.0


Name
Gascost
Time (ns)
MGas/S
Gascost for 10MGas/S
Gascost for ECDSA eq


modexp_nagydani_1_qube
204.0
8196.00
24.89019034
81.96
216.5008365


alt_bn128_add_cdetrio12
500.0
13834.00
36.14283649
138.34
365.4310117


alt_bn128_mul_cdetrio11
40000.0
494407.00
80.90500337
4944.07
13059.97182


alt_bn128_pairing_ten_point_match_1
900000.0
67707776.00
13.29241711
677077.76
1788529.788


After bn optimalization


master branch as of 25 Oct (5a1dc3e)
bn pointing at 2a71dbd (after optimalization from paritytech/bn#9)


Name
Gascost
Time (ns)
MGas/S
Gascost for 10MGas/S
Gascost for ECDSA eq


ecrecover
3000.0
123149.00
24.36073375
1231.49
3000.0


Name
Gascost
Time (ns)
MGas/S
Gascost for 10MGas/S
Gascost for ECDSA eq


modexp_nagydani_1_qube
204.0
9180.00
22.22222222
91.8
223.6315358


alt_bn128_add_cdetrio12
500.0
5179.00
96.54373431
51.79
126.1642401


alt_bn128_mul_cdetrio11
40000.0
214629.00
186.368105
2146.29
5228.519923


alt_bn128_pairing_ten_point_match_1
900000.0
34275714.00
26.2576587
342757.14
834981.5427


## raw-2.1.1-beta.benchmark
$ sysctl -n machdep.cpu.brand_string
Intel(R) Core(TM) i7-7920HQ CPU @ 3.10GHz

test result: ok. 0 passed; 0 failed; 332 ignored; 0 measured; 0 filtered out

     Running `/Users/piotr/git/parity-ethereum/target/release/deps/builtin-1676d3a0791765a3 --bench`

running 57 tests
test alt_bn128_add_cdetrio1              ... bench:         330 ns/iter (+/- 46)
test alt_bn128_add_cdetrio10             ... bench:         630 ns/iter (+/- 71)
test alt_bn128_add_cdetrio11             ... bench:       5,613 ns/iter (+/- 1,000)
test alt_bn128_add_cdetrio12             ... bench:       5,551 ns/iter (+/- 847)
test alt_bn128_add_cdetrio13             ... bench:       5,038 ns/iter (+/- 896)
test alt_bn128_add_cdetrio14             ... bench:       1,311 ns/iter (+/- 129)
test alt_bn128_add_cdetrio2              ... bench:         304 ns/iter (+/- 29)
test alt_bn128_add_cdetrio3              ... bench:         318 ns/iter (+/- 40)
test alt_bn128_add_cdetrio4              ... bench:         310 ns/iter (+/- 26)
test alt_bn128_add_cdetrio5              ... bench:         326 ns/iter (+/- 41)
test alt_bn128_add_cdetrio6              ... bench:         691 ns/iter (+/- 93)
test alt_bn128_add_cdetrio7              ... bench:         650 ns/iter (+/- 89)
test alt_bn128_add_cdetrio8              ... bench:         634 ns/iter (+/- 137)
test alt_bn128_add_cdetrio9              ... bench:         642 ns/iter (+/- 76)
test alt_bn128_add_chfast1               ... bench:       4,743 ns/iter (+/- 303)
test alt_bn128_add_chfast2               ... bench:       4,738 ns/iter (+/- 472)
test alt_bn128_mul_cdetrio1              ... bench:     225,765 ns/iter (+/- 15,520)
test alt_bn128_mul_cdetrio11             ... bench:     229,342 ns/iter (+/- 9,993)
test alt_bn128_mul_cdetrio6              ... bench:     243,032 ns/iter (+/- 34,909)
test alt_bn128_mul_chfast1               ... bench:      50,472 ns/iter (+/- 5,524)
test alt_bn128_mul_chfast2               ... bench:     109,065 ns/iter (+/- 15,625)
test alt_bn128_mul_chfast3               ... bench:     200,649 ns/iter (+/- 15,213)
test alt_bn128_pairing_empty_data        ... bench:          29 ns/iter (+/- 5)
test alt_bn128_pairing_jeff1             ... bench:   7,161,114 ns/iter (+/- 313,825)
test alt_bn128_pairing_jeff2             ... bench:   7,076,581 ns/iter (+/- 310,160)
test alt_bn128_pairing_jeff3             ... bench:   7,139,972 ns/iter (+/- 694,744)
test alt_bn128_pairing_jeff4             ... bench:  10,583,223 ns/iter (+/- 1,630,858)
test alt_bn128_pairing_jeff5             ... bench:  10,379,414 ns/iter (+/- 690,564)
test alt_bn128_pairing_jeff6             ... bench:   7,118,354 ns/iter (+/- 379,069)
test alt_bn128_pairing_one_point         ... bench:   3,851,247 ns/iter (+/- 991,301)
test alt_bn128_pairing_ten_point_match_1 ... bench:  34,076,695 ns/iter (+/- 3,528,604)
test alt_bn128_pairing_ten_point_match_2 ... bench:  36,357,284 ns/iter (+/- 5,783,316)
test alt_bn128_pairing_ten_point_match_3 ... bench:   7,582,345 ns/iter (+/- 1,299,180)
test alt_bn128_pairing_two_point_match_2 ... bench:   7,126,534 ns/iter (+/- 930,637)
test alt_bn128_pairing_two_point_match_3 ... bench:   7,176,366 ns/iter (+/- 507,984)
test alt_bn128_pairing_two_point_match_4 ... bench:   7,011,596 ns/iter (+/- 492,642)
test ecrecover                           ... bench:     116,516 ns/iter (+/- 20,167)
test identity                            ... bench:          79 ns/iter (+/- 14)
test modexp_eip_example1                 ... bench:     566,063 ns/iter (+/- 97,608)
test modexp_eip_example2                 ... bench:         292 ns/iter (+/- 32)
test modexp_nagydani_1_pow0x10001        ... bench:      42,294 ns/iter (+/- 3,057)
test modexp_nagydani_1_qube              ... bench:       8,188 ns/iter (+/- 999)
test modexp_nagydani_1_square            ... bench:       5,597 ns/iter (+/- 349)
test modexp_nagydani_2_pow0x10001        ... bench:      87,109 ns/iter (+/- 5,034)
test modexp_nagydani_2_qube              ... bench:      15,341 ns/iter (+/- 1,500)
test modexp_nagydani_2_square            ... bench:      10,614 ns/iter (+/- 662)
test modexp_nagydani_3_pow0x10001        ... bench:     265,390 ns/iter (+/- 60,985)
test modexp_nagydani_3_qube              ... bench:      44,260 ns/iter (+/- 4,956)
test modexp_nagydani_3_square            ... bench:      29,629 ns/iter (+/- 1,455)
test modexp_nagydani_4_pow0x10001        ... bench:     823,959 ns/iter (+/- 73,310)
test modexp_nagydani_4_qube              ... bench:     145,951 ns/iter (+/- 40,422)
test modexp_nagydani_4_square            ... bench:      97,402 ns/iter (+/- 19,660)
test modexp_nagydani_5_pow0x10001        ... bench:   2,956,937 ns/iter (+/- 120,018)
test modexp_nagydani_5_qube              ... bench:     497,193 ns/iter (+/- 27,340)
test modexp_nagydani_5_square            ... bench:     360,141 ns/iter (+/- 101,174)
test ripemd                              ... bench:         791 ns/iter (+/- 53)
test sha256                              ... bench:         482 ns/iter (+/- 27)

test result: ok. 0 passed; 0 failed; 0 ignored; 57 measured; 0 filtered out

## raw-master-5a1dc3e-bn-2a71dbd.benchmark
$ sysctl -n machdep.cpu.brand_string
Intel(R) Core(TM) i7-7920HQ CPU @ 3.10GHz

test result: ok. 0 passed; 0 failed; 333 ignored; 0 measured; 0 filtered out

     Running `/Users/piotr/git/parity-ethereum/target/release/deps/builtin-a0369717f10190f3 --bench`

running 57 tests
test alt_bn128_add_cdetrio1              ... bench:         305 ns/iter (+/- 9)
test alt_bn128_add_cdetrio10             ... bench:         948 ns/iter (+/- 1,341)
test alt_bn128_add_cdetrio11             ... bench:       5,315 ns/iter (+/- 406)
test alt_bn128_add_cdetrio12             ... bench:       5,179 ns/iter (+/- 594)
test alt_bn128_add_cdetrio13             ... bench:       4,554 ns/iter (+/- 587)
test alt_bn128_add_cdetrio14             ... bench:       1,358 ns/iter (+/- 242)
test alt_bn128_add_cdetrio2              ... bench:         313 ns/iter (+/- 33)
test alt_bn128_add_cdetrio3              ... bench:         313 ns/iter (+/- 69)
test alt_bn128_add_cdetrio4              ... bench:         311 ns/iter (+/- 29)
test alt_bn128_add_cdetrio5              ... bench:         309 ns/iter (+/- 44)
test alt_bn128_add_cdetrio6              ... bench:         641 ns/iter (+/- 124)
test alt_bn128_add_cdetrio7              ... bench:         634 ns/iter (+/- 86)
test alt_bn128_add_cdetrio8              ... bench:         606 ns/iter (+/- 69)
test alt_bn128_add_cdetrio9              ... bench:         593 ns/iter (+/- 58)
test alt_bn128_add_chfast1               ... bench:       4,401 ns/iter (+/- 354)
test alt_bn128_add_chfast2               ... bench:       4,495 ns/iter (+/- 458)
test alt_bn128_mul_cdetrio1              ... bench:     217,850 ns/iter (+/- 15,616)
test alt_bn128_mul_cdetrio11             ... bench:     214,629 ns/iter (+/- 10,053)
test alt_bn128_mul_cdetrio6              ... bench:     217,246 ns/iter (+/- 23,334)
test alt_bn128_mul_chfast1               ... bench:      46,941 ns/iter (+/- 5,343)
test alt_bn128_mul_chfast2               ... bench:     107,822 ns/iter (+/- 13,619)
test alt_bn128_mul_chfast3               ... bench:     198,872 ns/iter (+/- 45,250)
test alt_bn128_pairing_empty_data        ... bench:          29 ns/iter (+/- 9)
test alt_bn128_pairing_jeff1             ... bench:   6,810,682 ns/iter (+/- 312,331)
test alt_bn128_pairing_jeff2             ... bench:   6,867,771 ns/iter (+/- 664,455)
test alt_bn128_pairing_jeff3             ... bench:   6,785,325 ns/iter (+/- 186,119)
test alt_bn128_pairing_jeff4             ... bench:  10,181,887 ns/iter (+/- 598,836)
test alt_bn128_pairing_jeff5             ... bench:  10,209,255 ns/iter (+/- 920,372)
test alt_bn128_pairing_jeff6             ... bench:   6,771,495 ns/iter (+/- 255,136)
test alt_bn128_pairing_one_point         ... bench:   3,402,253 ns/iter (+/- 163,719)
test alt_bn128_pairing_ten_point_match_1 ... bench:  34,275,714 ns/iter (+/- 3,169,219)
test alt_bn128_pairing_ten_point_match_2 ... bench:  33,779,811 ns/iter (+/- 4,012,588)
test alt_bn128_pairing_ten_point_match_3 ... bench:   6,796,607 ns/iter (+/- 224,947)
test alt_bn128_pairing_two_point_match_2 ... bench:   6,809,049 ns/iter (+/- 409,879)
test alt_bn128_pairing_two_point_match_3 ... bench:   6,948,928 ns/iter (+/- 691,229)
test alt_bn128_pairing_two_point_match_4 ... bench:   6,963,919 ns/iter (+/- 744,114)
test ecrecover                           ... bench:     123,149 ns/iter (+/- 21,322)
test identity                            ... bench:          78 ns/iter (+/- 14)
test modexp_eip_example1                 ... bench:     585,479 ns/iter (+/- 101,136)
test modexp_eip_example2                 ... bench:         293 ns/iter (+/- 67)
test modexp_nagydani_1_pow0x10001        ... bench:      48,475 ns/iter (+/- 11,729)
test modexp_nagydani_1_qube              ... bench:       9,180 ns/iter (+/- 2,000)
test modexp_nagydani_1_square            ... bench:       5,932 ns/iter (+/- 382)
test modexp_nagydani_2_pow0x10001        ... bench:      90,050 ns/iter (+/- 9,794)
test modexp_nagydani_2_qube              ... bench:      15,877 ns/iter (+/- 4,157)
test modexp_nagydani_2_square            ... bench:      10,720 ns/iter (+/- 584)
test modexp_nagydani_3_pow0x10001        ... bench:     252,624 ns/iter (+/- 38,900)
test modexp_nagydani_3_qube              ... bench:      44,000 ns/iter (+/- 2,469)
test modexp_nagydani_3_square            ... bench:      30,180 ns/iter (+/- 2,348)
test modexp_nagydani_4_pow0x10001        ... bench:     820,333 ns/iter (+/- 62,901)
test modexp_nagydani_4_qube              ... bench:     143,577 ns/iter (+/- 28,778)
test modexp_nagydani_4_square            ... bench:      99,039 ns/iter (+/- 18,831)
test modexp_nagydani_5_pow0x10001        ... bench:   2,942,966 ns/iter (+/- 286,061)
test modexp_nagydani_5_qube              ... bench:     499,347 ns/iter (+/- 68,674)
test modexp_nagydani_5_square            ... bench:     344,958 ns/iter (+/- 51,048)
test ripemd                              ... bench:         777 ns/iter (+/- 68)
test sha256                              ... bench:         478 ns/iter (+/- 101)

test result: ok. 0 passed; 0 failed; 0 ignored; 57 measured; 0 filtered out

## raw-master-5a1dc3e-bn-786c0d5.benchmark
$ sysctl -n machdep.cpu.brand_string
Intel(R) Core(TM) i7-7920HQ CPU @ 3.10GHz

test result: ok. 0 passed; 0 failed; 333 ignored; 0 measured; 0 filtered out

     Running `/Users/piotr/git/parity-ethereum/target/release/deps/builtin-f8cde5d10d9eff41 --bench`

running 57 tests
test alt_bn128_add_cdetrio1              ... bench:         290 ns/iter (+/- 4)
test alt_bn128_add_cdetrio10             ... bench:         977 ns/iter (+/- 153)
test alt_bn128_add_cdetrio11             ... bench:      13,104 ns/iter (+/- 1,189)
test alt_bn128_add_cdetrio12             ... bench:      13,834 ns/iter (+/- 1,614)
test alt_bn128_add_cdetrio13             ... bench:      11,854 ns/iter (+/- 1,938)
test alt_bn128_add_cdetrio14             ... bench:       2,747 ns/iter (+/- 265)
test alt_bn128_add_cdetrio2              ... bench:         279 ns/iter (+/- 23)
test alt_bn128_add_cdetrio3              ... bench:         291 ns/iter (+/- 29)
test alt_bn128_add_cdetrio4              ... bench:         288 ns/iter (+/- 42)
test alt_bn128_add_cdetrio5              ... bench:         275 ns/iter (+/- 46)
test alt_bn128_add_cdetrio6              ... bench:         943 ns/iter (+/- 146)
test alt_bn128_add_cdetrio7              ... bench:         934 ns/iter (+/- 144)
test alt_bn128_add_cdetrio8              ... bench:         972 ns/iter (+/- 109)
test alt_bn128_add_cdetrio9              ... bench:         919 ns/iter (+/- 61)
test alt_bn128_add_chfast1               ... bench:      11,336 ns/iter (+/- 1,018)
test alt_bn128_add_chfast2               ... bench:      11,905 ns/iter (+/- 1,503)
test alt_bn128_mul_cdetrio1              ... bench:     496,267 ns/iter (+/- 97,125)
test alt_bn128_mul_cdetrio11             ... bench:     494,407 ns/iter (+/- 33,830)
test alt_bn128_mul_cdetrio6              ... bench:     499,397 ns/iter (+/- 97,614)
test alt_bn128_mul_chfast1               ... bench:     111,300 ns/iter (+/- 14,779)
test alt_bn128_mul_chfast2               ... bench:     248,520 ns/iter (+/- 14,387)
test alt_bn128_mul_chfast3               ... bench:     437,082 ns/iter (+/- 27,567)
test alt_bn128_pairing_empty_data        ... bench:          28 ns/iter (+/- 1)
test alt_bn128_pairing_jeff1             ... bench:  13,969,394 ns/iter (+/- 553,386)
test alt_bn128_pairing_jeff2             ... bench:  13,933,523 ns/iter (+/- 562,737)
test alt_bn128_pairing_jeff3             ... bench:  13,961,327 ns/iter (+/- 509,729)
test alt_bn128_pairing_jeff4             ... bench:  21,983,682 ns/iter (+/- 2,389,361)
test alt_bn128_pairing_jeff5             ... bench:  21,978,623 ns/iter (+/- 3,565,608)
test alt_bn128_pairing_jeff6             ... bench:  13,763,837 ns/iter (+/- 554,839)
test alt_bn128_pairing_one_point         ... bench:   6,985,496 ns/iter (+/- 386,617)
test alt_bn128_pairing_ten_point_match_1 ... bench:  67,707,776 ns/iter (+/- 3,632,600)
test alt_bn128_pairing_ten_point_match_2 ... bench:  67,511,217 ns/iter (+/- 3,032,175)
test alt_bn128_pairing_ten_point_match_3 ... bench:  13,710,818 ns/iter (+/- 532,709)
test alt_bn128_pairing_two_point_match_2 ... bench:  13,675,015 ns/iter (+/- 608,874)
test alt_bn128_pairing_two_point_match_3 ... bench:  13,714,703 ns/iter (+/- 641,139)
test alt_bn128_pairing_two_point_match_4 ... bench:  13,769,451 ns/iter (+/- 802,421)
test ecrecover                           ... bench:     113,570 ns/iter (+/- 16,435)
test identity                            ... bench:          77 ns/iter (+/- 11)
test modexp_eip_example1                 ... bench:     569,781 ns/iter (+/- 70,736)
test modexp_eip_example2                 ... bench:         290 ns/iter (+/- 33)
test modexp_nagydani_1_pow0x10001        ... bench:      42,564 ns/iter (+/- 5,066)
test modexp_nagydani_1_qube              ... bench:       8,196 ns/iter (+/- 397)
test modexp_nagydani_1_square            ... bench:       5,760 ns/iter (+/- 459)
test modexp_nagydani_2_pow0x10001        ... bench:      87,110 ns/iter (+/- 10,860)
test modexp_nagydani_2_qube              ... bench:      15,179 ns/iter (+/- 753)
test modexp_nagydani_2_square            ... bench:      10,542 ns/iter (+/- 497)
test modexp_nagydani_3_pow0x10001        ... bench:     253,620 ns/iter (+/- 40,738)
test modexp_nagydani_3_qube              ... bench:      43,602 ns/iter (+/- 2,690)
test modexp_nagydani_3_square            ... bench:      29,735 ns/iter (+/- 2,730)
test modexp_nagydani_4_pow0x10001        ... bench:     855,517 ns/iter (+/- 101,537)
test modexp_nagydani_4_qube              ... bench:     140,814 ns/iter (+/- 23,713)
test modexp_nagydani_4_square            ... bench:      91,803 ns/iter (+/- 7,332)
test modexp_nagydani_5_pow0x10001        ... bench:   2,919,130 ns/iter (+/- 170,181)
test modexp_nagydani_5_qube              ... bench:     499,157 ns/iter (+/- 57,339)
test modexp_nagydani_5_square            ... bench:     332,092 ns/iter (+/- 34,205)
test ripemd                              ... bench:         778 ns/iter (+/- 95)
test sha256                              ... bench:         466 ns/iter (+/- 69)

test result: ok. 0 passed; 0 failed; 0 ignored; 57 measured; 0 filtered out
Name	Gascost	Time (ns)	MGas/S	Gascost for 10MGas/S	Gascost for ECDSA eq
modexp_nagydani_1_qube	204.0	9388.00	21.7298679165	93.88	206.835773982
alt_bn128_add_cdetrio12	500.0	16894.00	29.596306381	168.94	372.207452668
alt_bn128_mul_cdetrio11	40000.0	748248.00	53.4582117159	7482.48	16485.3487655
alt_bn128_pairing_ten_point_match_1	900000.0	96298039.00	9.34598470899	962980.39	2121631.8097
Name	Gascost	Time (ns)	MGas/S	Gascost for 10MGas/S	Gascost for ECDSA eq
modexp_nagydani_1_qube	204.0	8188.00	24.91450904	81.88	210.8208315
alt_bn128_add_cdetrio12	500.0	5551.00	90.07386057	55.51	142.9245769
alt_bn128_mul_cdetrio11	40000.0	229342.00	174.4120135	2293.42	5904.991589
alt_bn128_pairing_ten_point_match_1	900000.0	34076695.00	26.41101198	340766.95	877390.9592
Name	Gascost	Time (ns)	MGas/S	Gascost for 10MGas/S	Gascost for ECDSA eq
modexp_nagydani_1_qube	204.0	8196.00	24.89019034	81.96	216.5008365
alt_bn128_add_cdetrio12	500.0	13834.00	36.14283649	138.34	365.4310117
alt_bn128_mul_cdetrio11	40000.0	494407.00	80.90500337	4944.07	13059.97182
alt_bn128_pairing_ten_point_match_1	900000.0	67707776.00	13.29241711	677077.76	1788529.788
Name	Gascost	Time (ns)	MGas/S	Gascost for 10MGas/S	Gascost for ECDSA eq
modexp_nagydani_1_qube	204.0	9180.00	22.22222222	91.8	223.6315358
alt_bn128_add_cdetrio12	500.0	5179.00	96.54373431	51.79	126.1642401
alt_bn128_mul_cdetrio11	40000.0	214629.00	186.368105	2146.29	5228.519923
alt_bn128_pairing_ten_point_match_1	900000.0	34275714.00	26.2576587	342757.14	834981.5427
	$ sysctl -n machdep.cpu.brand_string
	Intel(R) Core(TM) i7-7920HQ CPU @ 3.10GHz

	test result: ok. 0 passed; 0 failed; 332 ignored; 0 measured; 0 filtered out

	Running `/Users/piotr/git/parity-ethereum/target/release/deps/builtin-1676d3a0791765a3 --bench`

	running 57 tests
	test alt_bn128_add_cdetrio1 ... bench: 330 ns/iter (+/- 46)
	test alt_bn128_add_cdetrio10 ... bench: 630 ns/iter (+/- 71)
	test alt_bn128_add_cdetrio11 ... bench: 5,613 ns/iter (+/- 1,000)
	test alt_bn128_add_cdetrio12 ... bench: 5,551 ns/iter (+/- 847)
	test alt_bn128_add_cdetrio13 ... bench: 5,038 ns/iter (+/- 896)
	test alt_bn128_add_cdetrio14 ... bench: 1,311 ns/iter (+/- 129)
	test alt_bn128_add_cdetrio2 ... bench: 304 ns/iter (+/- 29)
	test alt_bn128_add_cdetrio3 ... bench: 318 ns/iter (+/- 40)
	test alt_bn128_add_cdetrio4 ... bench: 310 ns/iter (+/- 26)
	test alt_bn128_add_cdetrio5 ... bench: 326 ns/iter (+/- 41)
	test alt_bn128_add_cdetrio6 ... bench: 691 ns/iter (+/- 93)
	test alt_bn128_add_cdetrio7 ... bench: 650 ns/iter (+/- 89)
	test alt_bn128_add_cdetrio8 ... bench: 634 ns/iter (+/- 137)
	test alt_bn128_add_cdetrio9 ... bench: 642 ns/iter (+/- 76)
	test alt_bn128_add_chfast1 ... bench: 4,743 ns/iter (+/- 303)
	test alt_bn128_add_chfast2 ... bench: 4,738 ns/iter (+/- 472)
	test alt_bn128_mul_cdetrio1 ... bench: 225,765 ns/iter (+/- 15,520)
	test alt_bn128_mul_cdetrio11 ... bench: 229,342 ns/iter (+/- 9,993)
	test alt_bn128_mul_cdetrio6 ... bench: 243,032 ns/iter (+/- 34,909)
	test alt_bn128_mul_chfast1 ... bench: 50,472 ns/iter (+/- 5,524)
	test alt_bn128_mul_chfast2 ... bench: 109,065 ns/iter (+/- 15,625)
	test alt_bn128_mul_chfast3 ... bench: 200,649 ns/iter (+/- 15,213)
	test alt_bn128_pairing_empty_data ... bench: 29 ns/iter (+/- 5)
	test alt_bn128_pairing_jeff1 ... bench: 7,161,114 ns/iter (+/- 313,825)
	test alt_bn128_pairing_jeff2 ... bench: 7,076,581 ns/iter (+/- 310,160)
	test alt_bn128_pairing_jeff3 ... bench: 7,139,972 ns/iter (+/- 694,744)
	test alt_bn128_pairing_jeff4 ... bench: 10,583,223 ns/iter (+/- 1,630,858)
	test alt_bn128_pairing_jeff5 ... bench: 10,379,414 ns/iter (+/- 690,564)
	test alt_bn128_pairing_jeff6 ... bench: 7,118,354 ns/iter (+/- 379,069)
	test alt_bn128_pairing_one_point ... bench: 3,851,247 ns/iter (+/- 991,301)
	test alt_bn128_pairing_ten_point_match_1 ... bench: 34,076,695 ns/iter (+/- 3,528,604)
	test alt_bn128_pairing_ten_point_match_2 ... bench: 36,357,284 ns/iter (+/- 5,783,316)
	test alt_bn128_pairing_ten_point_match_3 ... bench: 7,582,345 ns/iter (+/- 1,299,180)
	test alt_bn128_pairing_two_point_match_2 ... bench: 7,126,534 ns/iter (+/- 930,637)
	test alt_bn128_pairing_two_point_match_3 ... bench: 7,176,366 ns/iter (+/- 507,984)
	test alt_bn128_pairing_two_point_match_4 ... bench: 7,011,596 ns/iter (+/- 492,642)
	test ecrecover ... bench: 116,516 ns/iter (+/- 20,167)
	test identity ... bench: 79 ns/iter (+/- 14)
	test modexp_eip_example1 ... bench: 566,063 ns/iter (+/- 97,608)
	test modexp_eip_example2 ... bench: 292 ns/iter (+/- 32)
	test modexp_nagydani_1_pow0x10001 ... bench: 42,294 ns/iter (+/- 3,057)
	test modexp_nagydani_1_qube ... bench: 8,188 ns/iter (+/- 999)
	test modexp_nagydani_1_square ... bench: 5,597 ns/iter (+/- 349)
	test modexp_nagydani_2_pow0x10001 ... bench: 87,109 ns/iter (+/- 5,034)
	test modexp_nagydani_2_qube ... bench: 15,341 ns/iter (+/- 1,500)
	test modexp_nagydani_2_square ... bench: 10,614 ns/iter (+/- 662)
	test modexp_nagydani_3_pow0x10001 ... bench: 265,390 ns/iter (+/- 60,985)
	test modexp_nagydani_3_qube ... bench: 44,260 ns/iter (+/- 4,956)
	test modexp_nagydani_3_square ... bench: 29,629 ns/iter (+/- 1,455)
	test modexp_nagydani_4_pow0x10001 ... bench: 823,959 ns/iter (+/- 73,310)
	test modexp_nagydani_4_qube ... bench: 145,951 ns/iter (+/- 40,422)
	test modexp_nagydani_4_square ... bench: 97,402 ns/iter (+/- 19,660)
	test modexp_nagydani_5_pow0x10001 ... bench: 2,956,937 ns/iter (+/- 120,018)
	test modexp_nagydani_5_qube ... bench: 497,193 ns/iter (+/- 27,340)
	test modexp_nagydani_5_square ... bench: 360,141 ns/iter (+/- 101,174)
	test ripemd ... bench: 791 ns/iter (+/- 53)
	test sha256 ... bench: 482 ns/iter (+/- 27)

	test result: ok. 0 passed; 0 failed; 0 ignored; 57 measured; 0 filtered out
	$ sysctl -n machdep.cpu.brand_string
	Intel(R) Core(TM) i7-7920HQ CPU @ 3.10GHz

	test result: ok. 0 passed; 0 failed; 333 ignored; 0 measured; 0 filtered out

	Running `/Users/piotr/git/parity-ethereum/target/release/deps/builtin-a0369717f10190f3 --bench`

	running 57 tests
	test alt_bn128_add_cdetrio1 ... bench: 305 ns/iter (+/- 9)
	test alt_bn128_add_cdetrio10 ... bench: 948 ns/iter (+/- 1,341)
	test alt_bn128_add_cdetrio11 ... bench: 5,315 ns/iter (+/- 406)
	test alt_bn128_add_cdetrio12 ... bench: 5,179 ns/iter (+/- 594)
	test alt_bn128_add_cdetrio13 ... bench: 4,554 ns/iter (+/- 587)
	test alt_bn128_add_cdetrio14 ... bench: 1,358 ns/iter (+/- 242)
	test alt_bn128_add_cdetrio2 ... bench: 313 ns/iter (+/- 33)
	test alt_bn128_add_cdetrio3 ... bench: 313 ns/iter (+/- 69)
	test alt_bn128_add_cdetrio4 ... bench: 311 ns/iter (+/- 29)
	test alt_bn128_add_cdetrio5 ... bench: 309 ns/iter (+/- 44)
	test alt_bn128_add_cdetrio6 ... bench: 641 ns/iter (+/- 124)
	test alt_bn128_add_cdetrio7 ... bench: 634 ns/iter (+/- 86)
	test alt_bn128_add_cdetrio8 ... bench: 606 ns/iter (+/- 69)
	test alt_bn128_add_cdetrio9 ... bench: 593 ns/iter (+/- 58)
	test alt_bn128_add_chfast1 ... bench: 4,401 ns/iter (+/- 354)
	test alt_bn128_add_chfast2 ... bench: 4,495 ns/iter (+/- 458)
	test alt_bn128_mul_cdetrio1 ... bench: 217,850 ns/iter (+/- 15,616)
	test alt_bn128_mul_cdetrio11 ... bench: 214,629 ns/iter (+/- 10,053)
	test alt_bn128_mul_cdetrio6 ... bench: 217,246 ns/iter (+/- 23,334)
	test alt_bn128_mul_chfast1 ... bench: 46,941 ns/iter (+/- 5,343)
	test alt_bn128_mul_chfast2 ... bench: 107,822 ns/iter (+/- 13,619)
	test alt_bn128_mul_chfast3 ... bench: 198,872 ns/iter (+/- 45,250)
	test alt_bn128_pairing_empty_data ... bench: 29 ns/iter (+/- 9)
	test alt_bn128_pairing_jeff1 ... bench: 6,810,682 ns/iter (+/- 312,331)
	test alt_bn128_pairing_jeff2 ... bench: 6,867,771 ns/iter (+/- 664,455)
	test alt_bn128_pairing_jeff3 ... bench: 6,785,325 ns/iter (+/- 186,119)
	test alt_bn128_pairing_jeff4 ... bench: 10,181,887 ns/iter (+/- 598,836)
	test alt_bn128_pairing_jeff5 ... bench: 10,209,255 ns/iter (+/- 920,372)
	test alt_bn128_pairing_jeff6 ... bench: 6,771,495 ns/iter (+/- 255,136)
	test alt_bn128_pairing_one_point ... bench: 3,402,253 ns/iter (+/- 163,719)
	test alt_bn128_pairing_ten_point_match_1 ... bench: 34,275,714 ns/iter (+/- 3,169,219)
	test alt_bn128_pairing_ten_point_match_2 ... bench: 33,779,811 ns/iter (+/- 4,012,588)
	test alt_bn128_pairing_ten_point_match_3 ... bench: 6,796,607 ns/iter (+/- 224,947)
	test alt_bn128_pairing_two_point_match_2 ... bench: 6,809,049 ns/iter (+/- 409,879)
	test alt_bn128_pairing_two_point_match_3 ... bench: 6,948,928 ns/iter (+/- 691,229)
	test alt_bn128_pairing_two_point_match_4 ... bench: 6,963,919 ns/iter (+/- 744,114)
	test ecrecover ... bench: 123,149 ns/iter (+/- 21,322)
	test identity ... bench: 78 ns/iter (+/- 14)
	test modexp_eip_example1 ... bench: 585,479 ns/iter (+/- 101,136)
	test modexp_eip_example2 ... bench: 293 ns/iter (+/- 67)
	test modexp_nagydani_1_pow0x10001 ... bench: 48,475 ns/iter (+/- 11,729)
	test modexp_nagydani_1_qube ... bench: 9,180 ns/iter (+/- 2,000)
	test modexp_nagydani_1_square ... bench: 5,932 ns/iter (+/- 382)
	test modexp_nagydani_2_pow0x10001 ... bench: 90,050 ns/iter (+/- 9,794)
	test modexp_nagydani_2_qube ... bench: 15,877 ns/iter (+/- 4,157)
	test modexp_nagydani_2_square ... bench: 10,720 ns/iter (+/- 584)
	test modexp_nagydani_3_pow0x10001 ... bench: 252,624 ns/iter (+/- 38,900)
	test modexp_nagydani_3_qube ... bench: 44,000 ns/iter (+/- 2,469)
	test modexp_nagydani_3_square ... bench: 30,180 ns/iter (+/- 2,348)
	test modexp_nagydani_4_pow0x10001 ... bench: 820,333 ns/iter (+/- 62,901)
	test modexp_nagydani_4_qube ... bench: 143,577 ns/iter (+/- 28,778)
	test modexp_nagydani_4_square ... bench: 99,039 ns/iter (+/- 18,831)
	test modexp_nagydani_5_pow0x10001 ... bench: 2,942,966 ns/iter (+/- 286,061)
	test modexp_nagydani_5_qube ... bench: 499,347 ns/iter (+/- 68,674)
	test modexp_nagydani_5_square ... bench: 344,958 ns/iter (+/- 51,048)
	test ripemd ... bench: 777 ns/iter (+/- 68)
	test sha256 ... bench: 478 ns/iter (+/- 101)

	test result: ok. 0 passed; 0 failed; 0 ignored; 57 measured; 0 filtered out