ormolu-fixity-th-bench.md

Comparing three versions of Ormolu:

• Ormolu 0.4.0.0 (only as a comparison, we are mostly interested in the other versions)
• Ormolu from amesgen/issue-826-fixities-th (commit 13cb1d4), such that the fixity information is parsed at compile time via Template Haskell. This is controlled with the new Cabal flag fixity-th, which is enabled by default.
• Like previous, but with the fixity-th flag disabled

As a test file, I used Optics.Operators from optics.

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
ormolu -e src/Optics/Operators.hs	22.5 ± 1.3	20.9	31.9	1.00
./ormolu-fixity src/Optics/Operators.hs	234.9 ± 1.2	232.5	236.5	10.46 ± 0.60
./ormolu-fixity-th src/Optics/Operators.hs	84.4 ± 0.8	82.9	86.0	3.76 ± 0.22

When giving this file multiple (10x and 100x) times to Ormolu, we get

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
ormolu -e src/Optics/Operators.hs ...	158.5 ± 3.6	154.9	166.4	1.00
./ormolu-fixity src/Optics/Operators.hs ...	712.9 ± 6.1	706.6	727.4	4.50 ± 0.11
./ormolu-fixity-th src/Optics/Operators.hs ...	536.5 ± 8.4	529.5	558.7	3.38 ± 0.09

Command	Mean [s]	Min [s]	Max [s]	Relative
ormolu -e src/Optics/Operators.hs ...	1.519 ± 0.031	1.486	1.580	1.00
./ormolu-fixity src/Optics/Operators.hs ...	5.728 ± 0.159	5.556	5.960	3.77 ± 0.13
./ormolu-fixity-th src/Optics/Operators.hs ...	5.342 ± 0.152	5.117	5.526	3.52 ± 0.12

Here, the absolute difference between ./ormolu-fixity and ./ormolu-fixity-th (150ms → 177ms → 439ms) increases at a much slower rate than the total running time.

From these numbers, parsing the fixity at compile time seems like a good idea. Still, the difference might become less relevant if we can cut down the runtime in comparison to Ormolu 0.4.0.0.

Laziness/Overhead of the operator printing algorithm

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
ormolu <<< 1	3.4 ± 0.9	1.0	6.4	1.00 ± 0.37
./ormolu-fixity --no-cabal <<< 1	3.7 ± 1.1	1.6	8.3	1.09 ± 0.41
./ormolu-fixity-th --no-cabal <<< 1	4.0 ± 1.5	1.5	8.4	1.18 ± 0.52
ormolu <<< 1+1	3.4 ± 0.9	1.6	6.5	1.00
./ormolu-fixity --no-cabal <<< 1+1	223.6 ± 1.4	221.3	225.8	65.38 ± 16.35
./ormolu-fixity-th --no-cabal <<< 1+1	67.7 ± 1.2	66.5	73.1	19.80 ± 4.96

How to run these benchmarks

I used hyperfine with this invocation

hyperfine -w 3 "ormolu -e $files" "./ormolu-fixity $files" "./ormolu-fixity-th $files"

where ./ormolu-fixity and ./ormolu-fixity-ty are symlinks to the respective results of nix-build -A ormoluExe (the flag fixity-th can be disabled by adding constraints: ormolu -fixity-th to cabal.project.local).

$files is set to src/Optics/Operators.hs, and it can be repeated using this magic:

files=$(printf 'src/Optics/Operators.hs %.0s' {1..10})

The laziness comparison uses

hyperfine -w 3 -L input 1,1+1 "ormolu <<< {input}" "./ormolu-fixity --no-cabal <<< {input}" "./ormolu-fixity-th --no-cabal <<< {input}"