cyber/janet benchmarks

gistfile1.txt

$ hyperfine --warmup 10 --shell=none 'zig-out/cyber/cyber test/bench/fib/fib.cy' 'janet test/bench/fib/fib.janet' 'lua test/bench/fib/fib.lua' 'luajit test/bench/fib/fib.lua'
Benchmark 1: zig-out/cyber/cyber test/bench/fib/fib.cy
  Time (mean ± σ):      54.6 ms ±   1.4 ms    [User: 53.1 ms, System: 1.1 ms]
  Range (min … max):    52.8 ms …  59.7 ms    55 runs

Benchmark 2: janet test/bench/fib/fib.janet
  Time (mean ± σ):     191.3 ms ±   5.4 ms    [User: 188.5 ms, System: 1.6 ms]
  Range (min … max):   184.1 ms … 203.6 ms    16 runs

Benchmark 3: lua test/bench/fib/fib.lua
  Time (mean ± σ):     184.2 ms ±  13.6 ms    [User: 182.5 ms, System: 0.6 ms]
  Range (min … max):   171.5 ms … 225.6 ms    13 runs

Benchmark 4: luajit test/bench/fib/fib.lua
  Time (mean ± σ):      22.5 ms ±   3.7 ms    [User: 21.0 ms, System: 1.1 ms]
  Range (min … max):    15.6 ms …  38.6 ms    113 runs

Summary
  'luajit test/bench/fib/fib.lua' ran
    2.43 ± 0.41 times faster than 'zig-out/cyber/cyber test/bench/fib/fib.cy'
    8.20 ± 1.49 times faster than 'lua test/bench/fib/fib.lua'
    8.52 ± 1.43 times faster than 'janet test/bench/fib/fib.janet'

$ hyperfine --warmup 10 --shell=none 'zig-out/cyber/cyber test/bench/fiber/fiber.cy' 'janet test/bench/fiber/fiber.janet' 'lua test/bench/fiber/fiber.lua' 'luajit test/bench/fiber/fiber.lua'
Benchmark 1: zig-out/cyber/cyber test/bench/fiber/fiber.cy
  Time (mean ± σ):      30.0 ms ±   3.0 ms    [User: 18.5 ms, System: 11.0 ms]
  Range (min … max):    27.6 ms …  47.3 ms    99 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 2: janet test/bench/fiber/fiber.janet
  Time (mean ± σ):     189.7 ms ±   8.9 ms    [User: 154.6 ms, System: 33.2 ms]
  Range (min … max):   180.7 ms … 204.5 ms    16 runs

Benchmark 3: lua test/bench/fiber/fiber.lua
  Time (mean ± σ):     243.8 ms ±  14.7 ms    [User: 185.8 ms, System: 54.2 ms]
  Range (min … max):   228.3 ms … 277.4 ms    12 runs

Benchmark 4: luajit test/bench/fiber/fiber.lua
  Time (mean ± σ):      84.4 ms ±  10.4 ms    [User: 56.6 ms, System: 26.6 ms]
  Range (min … max):    76.3 ms … 132.4 ms    37 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Summary
  'zig-out/cyber/cyber test/bench/fiber/fiber.cy' ran
    2.81 ± 0.45 times faster than 'luajit test/bench/fiber/fiber.lua'
    6.32 ± 0.70 times faster than 'janet test/bench/fiber/fiber.janet'
    8.12 ± 0.95 times faster than 'lua test/bench/fiber/fiber.lua'

$ hyperfine --warmup 10 --shell=none 'zig-out/cyber/cyber test/bench/for/for.cy' 'janet test/bench/for/for.janet' 'lua test/bench/for/for.lua' 'luajit test/bench/for/for.lua'
Benchmark 1: zig-out/cyber/cyber test/bench/for/for.cy
  Time (mean ± σ):      56.4 ms ±   5.8 ms    [User: 45.1 ms, System: 10.0 ms]
  Range (min … max):    51.6 ms …  78.5 ms    55 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 2: janet test/bench/for/for.janet
  Time (mean ± σ):     102.3 ms ±   7.4 ms    [User: 95.3 ms, System: 6.0 ms]
  Range (min … max):    96.5 ms … 137.2 ms    29 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 3: lua test/bench/for/for.lua
  Time (mean ± σ):      88.8 ms ±   4.8 ms    [User: 78.1 ms, System: 9.6 ms]
  Range (min … max):    83.1 ms …  99.5 ms    35 runs

Benchmark 4: luajit test/bench/for/for.lua
  Time (mean ± σ):      13.6 ms ±   1.0 ms    [User: 7.5 ms, System: 5.8 ms]
  Range (min … max):    12.7 ms …  20.0 ms    225 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Summary
  'luajit test/bench/for/for.lua' ran
    4.13 ± 0.52 times faster than 'zig-out/cyber/cyber test/bench/for/for.cy'
    6.51 ± 0.59 times faster than 'lua test/bench/for/for.lua'
    7.50 ± 0.77 times faster than 'janet test/bench/for/for.janet'

$ hyperfine --warmup 10 --shell=none 'zig-out/cyber/cyber test/bench/nop/nop.cy' 'janet test/bench/nop/nop.janet' 'lua test/bench/nop/nop.lua' 'luajit test/bench/nop/nop.lua'
Benchmark 1: zig-out/cyber/cyber test/bench/nop/nop.cy
  Time (mean ± σ):       1.4 ms ±   0.2 ms    [User: 0.5 ms, System: 0.8 ms]
  Range (min … max):     1.1 ms …   6.4 ms    2286 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 2: janet test/bench/nop/nop.janet
  Time (mean ± σ):       2.9 ms ±   0.3 ms    [User: 1.9 ms, System: 0.8 ms]
  Range (min … max):     2.6 ms …   4.7 ms    929 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 3: lua test/bench/nop/nop.lua
  Time (mean ± σ):       1.4 ms ±   0.3 ms    [User: 0.6 ms, System: 0.6 ms]
  Range (min … max):     1.2 ms …  10.0 ms    2181 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 4: luajit test/bench/nop/nop.lua
  Time (mean ± σ):       1.0 ms ±   0.1 ms    [User: 0.4 ms, System: 0.4 ms]
  Range (min … max):     0.8 ms …   1.7 ms    2796 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Summary
  'luajit test/bench/nop/nop.lua' ran
    1.39 ± 0.28 times faster than 'zig-out/cyber/cyber test/bench/nop/nop.cy'
    1.42 ± 0.34 times faster than 'lua test/bench/nop/nop.lua'
    2.91 ± 0.46 times faster than 'janet test/bench/nop/nop.janet'

$ janet -v
1.26.0-release

zig version
0.11.0-dev.1479+97b1a9bb6

Running Cyber built at 1444e71e658f0c297c2a075c49854349fa50e9b2.

I ran the benchmarks on this potato:

Host: KVM/QEMU (Standard PC (i440FX + PIIX, 1996) pc-i
Kernel: 4.14.118
CPU: Virtual a7769a6388d5 (1) @ 2.394GHz
Memory: 270MiB / 993MiB

These are pretty direct except that I wrapped some of the benchmarks in main functions, because mutating environment entries is significantly slower than assigning to local variables:

$ hyperfine --warmup 10 'janet test/bench/for/for.janet' 'janet test/bench/for/for-env.janet'
Benchmark 1: janet test/bench/for/for.janet
  Time (mean ± σ):     102.7 ms ±   4.7 ms    [User: 96.6 ms, System: 5.4 ms]
  Range (min … max):    94.9 ms … 110.0 ms    27 runs

Benchmark 2: janet test/bench/for/for-env.janet
  Time (mean ± σ):     126.9 ms ±   3.8 ms    [User: 121.4 ms, System: 4.8 ms]
  Range (min … max):   122.1 ms … 136.8 ms    23 runs

Summary
  'janet test/bench/for/for.janet' ran
    1.24 ± 0.07 times faster than 'janet test/bench/for/for-env.janet'

$ diff -u test/bench/for/for{,-env}.janet
--- test/bench/for/for.janet
+++ test/bench/for/for-env.janet
@@ -1,4 +1,3 @@
-(defn main [&]
 (def list @[])

 (for i 0 1000000
@@ -9,4 +8,3 @@
   (+= sum i))

 (print sum)
-)

Is this dishonest, showing Janet in a better light than the "obvious" implementations? I dunno. Maybe.