Skip to content

Instantly share code, notes, and snippets.

@zac-williamson

zac-williamson/benchmark.md

Last active June 21, 2023 10:44
Show Gist options
  • Star 2 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save zac-williamson/838410a3da179d47d31b25b586c15e53 to your computer and use it in GitHub Desktop.
Save zac-williamson/838410a3da179d47d31b25b586c15e53 to your computer and use it in GitHub Desktop.
Download ZIP
Result of parity benchmarks after `pairing_batch`, `precompute` and `miller_loop` optimizations
Raw
benchmark.md

03 May 2019, Updated Parity Benchmarks

These Parity client benchmarks were run to evaluate the optimizations made to the Parity bn pairing library (link here)

By comparing pre- and post- benchmarks there is a significant improvement to the speed of multiple bilinear pairings that merits a reduction in the gas schedule.

The new benchmarks were run on branches that are currently outstanding pull requests into the Parity client (openethereum/parity-ethereum#10624) and the BN library (paritytech/bn#14).

The previous benchmarks were computed on 25 Oct 2018

2.1.1-beta (release 20 Sep)

  • master branch from 25 Oct (5a1dc3e)
  • bn pointing to 2a71dbd
Name Gascost Time (ns) MGas/S Gascost for 10MGas/S Gascost for ECDSA eq
ecrecover 3000.0 116516.00 18.8587916544 1165.16 3000.0
Name Gascost Time (ns) MGas/S Gascost for 10MGas/S Gascost for ECDSA eq
modexp_nagydani_1_qube 204.0 8188.00 24.91450904 81.88 210.8208315
alt_bn128_add_cdetrio12 500.0 5551.00 90.07386057 55.51 142.9245769
alt_bn128_mul_cdetrio11 40000.0 229342.00 174.4120135 2293.42 5904.991589
alt_bn128_pairing_ten_point_match_1 900000.0 34076695.00 26.41101198 340766.95 877390.9592

After pairing optimizations (05 May 2019)

  • parity-ethereum commit hash f301f365987fd214f2f43d9d5f1f6ad54fbe1c14 (on a fork of parity-ethereum)
  • bn pointing to commit hash 65d150fd3ee3694682060defbfc4dc74754690f0 (add-wnaf branch of a fork of bn)
  • Both forks have outstanding pull requests to the respective Parity repositories
Name Gascost Time (ns) MGas/S Gascost for 10MGas/S Gascost for ECDSA eq
ecrecover 3000.0 116640.00 25.72 1166.40 3000.00
Name Gascost Time (ns) MGas/S Gascost for 10MGas/S Gascost for ECDSA eq
modexp_nagydani_1_qube 204.0 40597.00 5.03 405.97 1044.16
alt_bn128_add_cdetrio12 500.0 6932.50 72.12 69.33 178.31
alt_bn128_mul_cdetrio11 40000.0 232660.00 171.92 2326.60 5984.05
alt_bn128_pairing_one_point 180000.0 3036800.00 59.27 30368.00 78107.00
alt_bn128_pairing_ten_point_match_1 900000.0 14663000.00 61.38 146630.00 377134.77

Computing pairing costs relative to ecrecover

  • ecrecover-equiv-cost of 1 pairing = 78,107
  • ecrecover-equiv-cost of 10 pairings = 377,134
  • set gas-cost = A + (B * number_pairings)
  • A + 10B = 377,134
  • A + B = 78,107
  • B = 33,225
  • A = 44,882
Raw
benchmark_log.txt
$ cargo bench --package ethcore
test result: ok. 0 passed; 0 failed; 315 ignored; 0 measured; 0 filtered out
Running target/release/deps/builtin-63274a173a57e4ec
ecrecover time: [115.64 us 116.64 us 117.84 us]
change: [-15.538% -11.932% -8.4109%] (p = 0.00 < 0.05)
Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
2 (2.00%) high mild
6 (6.00%) high severe
sha256 time: [469.14 ns 470.14 ns 471.26 ns]
change: [-1.4282% -0.8673% -0.3296%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 10 outliers among 100 measurements (10.00%)
4 (4.00%) high mild
6 (6.00%) high severe
ripemd time: [634.48 ns 636.32 ns 638.53 ns]
change: [-1.7531% -1.0084% -0.3099%] (p = 0.01 < 0.05)
Change within noise threshold.
Found 13 outliers among 100 measurements (13.00%)
7 (7.00%) high mild
6 (6.00%) high severe
identity time: [95.023 ns 95.219 ns 95.434 ns]
change: [-0.5492% -0.1407% +0.3080%] (p = 0.54 > 0.05)
No change in performance detected.
Found 13 outliers among 100 measurements (13.00%)
3 (3.00%) low mild
5 (5.00%) high mild
5 (5.00%) high severe
modexp_eip_example1 time: [342.59 us 347.65 us 353.42 us]
change: [+1.2207% +2.5815% +3.9728%] (p = 0.00 < 0.05)
Performance has regressed.
Found 17 outliers among 100 measurements (17.00%)
8 (8.00%) high mild
9 (9.00%) high severe
modexp_eip_example2 time: [209.38 ns 216.23 ns 224.52 ns]
change: [+18.815% +21.650% +24.322%] (p = 0.00 < 0.05)
Performance has regressed.
modexp_nagydani_1_square
time: [3.4935 us 3.5031 us 3.5152 us]
change: [-0.2591% +0.4189% +1.1853%] (p = 0.26 > 0.05)
No change in performance detected.
Found 9 outliers among 100 measurements (9.00%)
4 (4.00%) high mild
5 (5.00%) high severe
modexp_nagydani_1_qube time: [6.7977 us 6.9930 us 7.1784 us]
change: [+6.4921% +8.7898% +11.323%] (p = 0.00 < 0.05)
Performance has regressed.
modexp_nagydani_1_pow0x10001
time: [40.466 us 40.597 us 40.766 us]
change: [-0.6311% -0.0183% +0.7407%] (p = 0.96 > 0.05)
No change in performance detected.
Found 8 outliers among 100 measurements (8.00%)
2 (2.00%) high mild
6 (6.00%) high severe
modexp_nagydani_2_square
time: [6.7436 us 6.8670 us 6.9948 us]
change: [-1.3922% +0.4736% +2.2082%] (p = 0.62 > 0.05)
No change in performance detected.
Found 13 outliers among 100 measurements (13.00%)
2 (2.00%) high mild
11 (11.00%) high severe
modexp_nagydani_2_qube time: [12.013 us 12.222 us 12.496 us]
change: [+14.653% +18.209% +21.687%] (p = 0.00 < 0.05)
Performance has regressed.
modexp_nagydani_2_pow0x10001
time: [88.797 us 89.031 us 89.306 us]
change: [-1.4859% -0.3746% +0.5837%] (p = 0.52 > 0.05)
No change in performance detected.
Found 12 outliers among 100 measurements (12.00%)
6 (6.00%) high mild
6 (6.00%) high severe
modexp_nagydani_3_square
time: [16.818 us 16.866 us 16.924 us]
change: [+1.9029% +2.6082% +3.4648%] (p = 0.00 < 0.05)
Performance has regressed.
Found 14 outliers among 100 measurements (14.00%)
9 (9.00%) high mild
5 (5.00%) high severe
modexp_nagydani_3_qube time: [31.633 us 31.769 us 31.939 us]
change: [+0.7613% +2.5481% +4.6156%] (p = 0.01 < 0.05)
Change within noise threshold.
Found 12 outliers among 100 measurements (12.00%)
2 (2.00%) high mild
10 (10.00%) high severe
modexp_nagydani_3_pow0x10001
time: [258.16 us 261.33 us 265.21 us]
change: [+8.6337% +10.646% +12.779%] (p = 0.00 < 0.05)
Performance has regressed.
modexp_nagydani_4_square
time: [48.384 us 48.510 us 48.657 us]
change: [-0.5903% +0.3551% +1.1182%] (p = 0.46 > 0.05)
No change in performance detected.
Found 13 outliers among 100 measurements (13.00%)
9 (9.00%) high mild
4 (4.00%) high severe
modexp_nagydani_4_qube time: [93.825 us 94.359 us 94.995 us]
change: [+1.0922% +1.7758% +2.4515%] (p = 0.00 < 0.05)
Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) high mild
1 (1.00%) high severe
modexp_nagydani_4_pow0x10001
time: [761.67 us 764.26 us 767.40 us]
change: [-0.9126% -0.2773% +0.3024%] (p = 0.39 > 0.05)
No change in performance detected.
Found 11 outliers among 100 measurements (11.00%)
5 (5.00%) high mild
6 (6.00%) high severe
modexp_nagydani_5_square
time: [162.64 us 163.43 us 164.49 us]
change: [-3.7298% -1.8336% -0.3598%] (p = 0.02 < 0.05)
Change within noise threshold.
Found 10 outliers among 100 measurements (10.00%)
5 (5.00%) high mild
5 (5.00%) high severe
modexp_nagydani_5_qube time: [320.70 us 331.24 us 350.50 us]
change: [+0.6416% +3.1053% +5.9900%] (p = 0.01 < 0.05)
Change within noise threshold.
Found 16 outliers among 100 measurements (16.00%)
2 (2.00%) high mild
14 (14.00%) high severe
modexp_nagydani_5_pow0x10001
time: [2.6686 ms 2.6731 ms 2.6781 ms]
change: [-4.8550% -2.2522% -0.3098%] (p = 0.05 < 0.05)
Change within noise threshold.
Found 9 outliers among 100 measurements (9.00%)
4 (4.00%) high mild
5 (5.00%) high severe
alt_bn128_add_chfast1 time: [5.4961 us 5.5111 us 5.5289 us]
change: [-1.5603% -0.7805% +0.0833%] (p = 0.06 > 0.05)
No change in performance detected.
Found 12 outliers among 100 measurements (12.00%)
3 (3.00%) high mild
9 (9.00%) high severe
alt_bn128_add_chfast2 time: [5.6512 us 5.6725 us 5.7023 us]
change: [-1.5864% -0.9880% -0.3701%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 10 outliers among 100 measurements (10.00%)
2 (2.00%) high mild
8 (8.00%) high severe
alt_bn128_add_cdetrio1 time: [159.34 ns 160.11 ns 161.10 ns]
change: [+2.7805% +4.3479% +6.0450%] (p = 0.00 < 0.05)
Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) high mild
1 (1.00%) high severe
alt_bn128_add_cdetrio2 time: [165.06 ns 165.63 ns 166.32 ns]
change: [+1.5021% +2.1923% +2.8904%] (p = 0.00 < 0.05)
Performance has regressed.
Found 12 outliers among 100 measurements (12.00%)
5 (5.00%) high mild
7 (7.00%) high severe
alt_bn128_add_cdetrio3 time: [168.72 ns 169.31 ns 170.05 ns]
change: [+1.5674% +2.6017% +3.3940%] (p = 0.00 < 0.05)
Performance has regressed.
Found 11 outliers among 100 measurements (11.00%)
6 (6.00%) high mild
5 (5.00%) high severe
alt_bn128_add_cdetrio4 time: [169.06 ns 169.40 ns 169.76 ns]
change: [+2.4895% +3.1333% +3.8183%] (p = 0.00 < 0.05)
Performance has regressed.
Found 7 outliers among 100 measurements (7.00%)
2 (2.00%) high mild
5 (5.00%) high severe
alt_bn128_add_cdetrio5 time: [162.43 ns 163.21 ns 164.34 ns]
change: [-0.2508% +0.4091% +1.2716%] (p = 0.31 > 0.05)
No change in performance detected.
Found 14 outliers among 100 measurements (14.00%)
5 (5.00%) high mild
9 (9.00%) high severe
alt_bn128_add_cdetrio6 time: [486.90 ns 492.22 ns 499.17 ns]
change: [+15.951% +18.859% +21.718%] (p = 0.00 < 0.05)
Performance has regressed.
alt_bn128_add_cdetrio7 time: [476.20 ns 477.32 ns 478.63 ns]
change: [+4.5727% +5.0589% +5.5639%] (p = 0.00 < 0.05)
Performance has regressed.
Found 12 outliers among 100 measurements (12.00%)
6 (6.00%) high mild
6 (6.00%) high severe
alt_bn128_add_cdetrio8 time: [469.91 ns 471.70 ns 473.86 ns]
change: [+3.4846% +4.1954% +4.8508%] (p = 0.00 < 0.05)
Performance has regressed.
Found 10 outliers among 100 measurements (10.00%)
7 (7.00%) high mild
3 (3.00%) high severe
alt_bn128_add_cdetrio9 time: [462.73 ns 463.89 ns 465.18 ns]
change: [+0.7375% +1.7674% +2.6135%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 8 outliers among 100 measurements (8.00%)
4 (4.00%) high mild
4 (4.00%) high severe
alt_bn128_add_cdetrio10 time: [462.89 ns 464.32 ns 466.09 ns]
change: [+1.1816% +1.7718% +2.3790%] (p = 0.00 < 0.05)
Performance has regressed.
Found 12 outliers among 100 measurements (12.00%)
1 (1.00%) high mild
11 (11.00%) high severe
alt_bn128_add_cdetrio11 time: [7.0520 us 7.0686 us 7.0874 us]
change: [+1.4142% +2.3307% +3.2142%] (p = 0.00 < 0.05)
Performance has regressed.
Found 11 outliers among 100 measurements (11.00%)
3 (3.00%) high mild
8 (8.00%) high severe
alt_bn128_add_cdetrio12 time: [6.9087 us 6.9325 us 6.9617 us]
change: [+1.8834% +2.7203% +3.5571%] (p = 0.00 < 0.05)
Performance has regressed.
Found 7 outliers among 100 measurements (7.00%)
4 (4.00%) high mild
3 (3.00%) high severe
alt_bn128_add_cdetrio13 time: [5.7721 us 5.7929 us 5.8161 us]
change: [+1.0361% +1.5660% +2.0925%] (p = 0.00 < 0.05)
Performance has regressed.
Found 13 outliers among 100 measurements (13.00%)
6 (6.00%) high mild
7 (7.00%) high severe
alt_bn128_add_cdetrio14 time: [1.1067 us 1.1110 us 1.1159 us]
change: [+1.5923% +2.1731% +2.7935%] (p = 0.00 < 0.05)
Performance has regressed.
Found 22 outliers among 100 measurements (22.00%)
11 (11.00%) low mild
5 (5.00%) high mild
6 (6.00%) high severe
alt_bn128_mul_chfast1 time: [47.973 us 49.006 us 50.165 us]
change: [+6.9874% +9.2457% +11.364%] (p = 0.00 < 0.05)
Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high severe
alt_bn128_mul_chfast2 time: [104.69 us 105.15 us 105.75 us]
change: [+6.0390% +8.3460% +10.755%] (p = 0.00 < 0.05)
Performance has regressed.
Found 20 outliers among 100 measurements (20.00%)
1 (1.00%) high mild
19 (19.00%) high severe
alt_bn128_mul_chfast3 time: [183.26 us 184.31 us 185.51 us]
change: [-3.6540% -2.0064% -0.5126%] (p = 0.01 < 0.05)
Change within noise threshold.
Found 6 outliers among 100 measurements (6.00%)
3 (3.00%) high mild
3 (3.00%) high severe
alt_bn128_mul_cdetrio1 time: [220.95 us 225.50 us 231.32 us]
change: [+18.269% +21.172% +24.188%] (p = 0.00 < 0.05)
Performance has regressed.
alt_bn128_mul_cdetrio6 time: [210.28 us 210.78 us 211.36 us]
change: [+2.2138% +2.7353% +3.2888%] (p = 0.00 < 0.05)
Performance has regressed.
Found 13 outliers among 100 measurements (13.00%)
7 (7.00%) high mild
6 (6.00%) high severe
alt_bn128_mul_cdetrio11 time: [226.19 us 232.66 us 240.37 us]
change: [+14.899% +17.286% +19.737%] (p = 0.00 < 0.05)
Performance has regressed.
alt_bn128_pairing_jeff1 time: [4.5175 ms 4.5245 ms 4.5326 ms]
change: [+2.0000% +2.6013% +3.1341%] (p = 0.00 < 0.05)
Performance has regressed.
Found 8 outliers among 100 measurements (8.00%)
4 (4.00%) high mild
4 (4.00%) high severe
alt_bn128_pairing_jeff2 time: [4.5477 ms 4.5821 ms 4.6277 ms]
change: [+5.1775% +6.9971% +9.0209%] (p = 0.00 < 0.05)
Performance has regressed.
Found 19 outliers among 100 measurements (19.00%)
5 (5.00%) high mild
14 (14.00%) high severe
alt_bn128_pairing_jeff3 time: [4.5135 ms 4.5196 ms 4.5269 ms]
change: [+1.4982% +2.0518% +2.5441%] (p = 0.00 < 0.05)
Performance has regressed.
Found 7 outliers among 100 measurements (7.00%)
3 (3.00%) high mild
4 (4.00%) high severe
alt_bn128_pairing_jeff4 time: [5.8452 ms 5.8984 ms 5.9696 ms]
change: [+1.4349% +1.9057% +2.4202%] (p = 0.00 < 0.05)
Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
4 (4.00%) high mild
2 (2.00%) high severe
alt_bn128_pairing_jeff5 time: [6.1072 ms 6.1308 ms 6.1626 ms]
change: [+7.2239% +9.3211% +11.486%] (p = 0.00 < 0.05)
Performance has regressed.
Found 17 outliers among 100 measurements (17.00%)
2 (2.00%) high mild
15 (15.00%) high severe
alt_bn128_pairing_jeff6 time: [4.5186 ms 4.5485 ms 4.5872 ms]
change: [+2.2585% +2.6900% +3.2205%] (p = 0.00 < 0.05)
Performance has regressed.
Found 9 outliers among 100 measurements (9.00%)
5 (5.00%) high mild
4 (4.00%) high severe
alt_bn128_pairing_empty_data
time: [29.119 ns 29.368 ns 29.681 ns]
change: [+2.7114% +3.5434% +4.6445%] (p = 0.00 < 0.05)
Performance has regressed.
Found 15 outliers among 100 measurements (15.00%)
5 (5.00%) high mild
10 (10.00%) high severe
alt_bn128_pairing_one_point
time: [3.0260 ms 3.0368 ms 3.0500 ms]
change: [-5.7801% -4.8809% -4.0162%] (p = 0.00 < 0.05)
Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
2 (2.00%) high mild
6 (6.00%) high severe
alt_bn128_pairing_two_point_match_2
time: [3.6699 ms 3.7065 ms 3.7489 ms]
change: [-23.607% -21.653% -19.623%] (p = 0.00 < 0.05)
Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
3 (3.00%) high mild
6 (6.00%) high severe
alt_bn128_pairing_two_point_match_3
time: [4.6246 ms 4.7259 ms 4.8344 ms]
change: [-8.2545% -5.5461% -2.8819%] (p = 0.00 < 0.05)
Performance has improved.
Found 16 outliers among 100 measurements (16.00%)
5 (5.00%) high mild
11 (11.00%) high severe
alt_bn128_pairing_two_point_match_4
time: [4.7411 ms 4.7962 ms 4.8567 ms]
change: [+9.3491% +11.304% +13.421%] (p = 0.00 < 0.05)
Performance has regressed.
Found 21 outliers among 100 measurements (21.00%)
14 (14.00%) high mild
7 (7.00%) high severe
alt_bn128_pairing_ten_point_match_1
time: [14.606 ms 14.663 ms 14.741 ms]
change: [-8.9748% -7.6216% -6.3467%] (p = 0.00 < 0.05)
Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
4 (4.00%) high mild
3 (3.00%) high severe
alt_bn128_pairing_ten_point_match_2
time: [15.554 ms 15.585 ms 15.628 ms]
change: [+0.1624% +0.5436% +0.9161%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 4 outliers among 100 measurements (4.00%)
4 (4.00%) high severe
alt_bn128_pairing_ten_point_match_3
time: [4.4080 ms 4.4138 ms 4.4200 ms]
change: [-0.7576% -0.1348% +0.3910%] (p = 0.66 > 0.05)
No change in performance detected.
Found 4 outliers among 100 measurements (4.00%)
2 (2.00%) high mild
2 (2.00%) high severe
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment