Skip to content

Instantly share code, notes, and snippets.

@amesgen
Created December 8, 2021 14:50
Show Gist options
  • Save amesgen/64e86711d21fb7e2c4fa5b597438a349 to your computer and use it in GitHub Desktop.
Save amesgen/64e86711d21fb7e2c4fa5b597438a349 to your computer and use it in GitHub Desktop.

Comparing three versions of Ormolu:

  • Ormolu 0.4.0.0 (only as a comparison, we are mostly interested in the other versions)
  • Ormolu from amesgen/issue-826-fixities-th (commit 13cb1d4), such that the fixity information is parsed at compile time via Template Haskell. This is controlled with the new Cabal flag fixity-th, which is enabled by default.
  • Like previous, but with the fixity-th flag disabled

As a test file, I used Optics.Operators from optics.

Command Mean [ms] Min [ms] Max [ms] Relative
ormolu -e src/Optics/Operators.hs 22.5 ± 1.3 20.9 31.9 1.00
./ormolu-fixity src/Optics/Operators.hs 234.9 ± 1.2 232.5 236.5 10.46 ± 0.60
./ormolu-fixity-th src/Optics/Operators.hs 84.4 ± 0.8 82.9 86.0 3.76 ± 0.22

When giving this file multiple (10x and 100x) times to Ormolu, we get

Command Mean [ms] Min [ms] Max [ms] Relative
ormolu -e src/Optics/Operators.hs ... 158.5 ± 3.6 154.9 166.4 1.00
./ormolu-fixity src/Optics/Operators.hs ... 712.9 ± 6.1 706.6 727.4 4.50 ± 0.11
./ormolu-fixity-th src/Optics/Operators.hs ... 536.5 ± 8.4 529.5 558.7 3.38 ± 0.09
Command Mean [s] Min [s] Max [s] Relative
ormolu -e src/Optics/Operators.hs ... 1.519 ± 0.031 1.486 1.580 1.00
./ormolu-fixity src/Optics/Operators.hs ... 5.728 ± 0.159 5.556 5.960 3.77 ± 0.13
./ormolu-fixity-th src/Optics/Operators.hs ... 5.342 ± 0.152 5.117 5.526 3.52 ± 0.12

Here, the absolute difference between ./ormolu-fixity and ./ormolu-fixity-th (150ms → 177ms → 439ms) increases at a much slower rate than the total running time.

From these numbers, parsing the fixity at compile time seems like a good idea. Still, the difference might become less relevant if we can cut down the runtime in comparison to Ormolu 0.4.0.0.

Laziness/Overhead of the operator printing algorithm

Command Mean [ms] Min [ms] Max [ms] Relative
ormolu <<< 1 3.4 ± 0.9 1.0 6.4 1.00 ± 0.37
./ormolu-fixity --no-cabal <<< 1 3.7 ± 1.1 1.6 8.3 1.09 ± 0.41
./ormolu-fixity-th --no-cabal <<< 1 4.0 ± 1.5 1.5 8.4 1.18 ± 0.52
ormolu <<< 1+1 3.4 ± 0.9 1.6 6.5 1.00
./ormolu-fixity --no-cabal <<< 1+1 223.6 ± 1.4 221.3 225.8 65.38 ± 16.35
./ormolu-fixity-th --no-cabal <<< 1+1 67.7 ± 1.2 66.5 73.1 19.80 ± 4.96

How to run these benchmarks

I used hyperfine with this invocation

hyperfine -w 3 "ormolu -e $files" "./ormolu-fixity $files" "./ormolu-fixity-th $files"

where ./ormolu-fixity and ./ormolu-fixity-ty are symlinks to the respective results of nix-build -A ormoluExe (the flag fixity-th can be disabled by adding constraints: ormolu -fixity-th to cabal.project.local).

$files is set to src/Optics/Operators.hs, and it can be repeated using this magic:

files=$(printf 'src/Optics/Operators.hs %.0s' {1..10})

The laziness comparison uses

hyperfine -w 3 -L input 1,1+1 "ormolu <<< {input}" "./ormolu-fixity --no-cabal <<< {input}" "./ormolu-fixity-th --no-cabal <<< {input}"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment