Skip to content

Instantly share code, notes, and snippets.

@cellularmitosis
Last active Dec 12, 2021
Embed
What would you like to do?
Trivial performance comparison of some Scheme interpreters

Blog 2019/11/2

<- previous | index | next ->

Trivial performance comparison of some Scheme interpreters

Let's use a trivial (recursive) Fibonacci benchmark to compare the performance of a few Scheme interpreters.

(define (fib n) (if (< n 2) n (+ (fib (- n 1)) (fib (- n 2)))))
(display (fib $n))
(newline)

Specifically, I was interested in looking at start-up time and overall throughput (actually, function-call overhead).

TL;DR

  • 👍 Startup time: Elk, Chicken
  • 👍 Throughput: Chez, Racket

Results

Overall results:

recursive (fib n)

A close-up of the "knee" of the curve:

recursive (fib n) (1)

Start-up time

If we focus on small values of n, we can compare start-up time. Here, non-JIT'ed interpreters will perform well.

Screen Shot 2019-11-02 at 5 51 07 AM

recursive (fib n) (3)

recursive (fib n) (2)

Throughput

If we focus on large values of n, we get a sense of the overall throughput (actually, the function-call overhead) of the interpreters. Here, the JIT'ed interpreters will perform well.

Screen Shot 2019-11-02 at 5 51 32 AM

recursive (fib n) (5)

#!/bin/bash
# brew install guile gambit-scheme chezscheme mit-scheme chibi-scheme chicken racket scheme48 gerbil-scheme kawa
# note: you'll have to hack this script and execute kawa as "kawa -s".
for scheme in elk guile chez mit-scheme csi chibi-scheme scheme48 racket
do
outfile="results-${scheme}.csv"
echo "(fib n),time" > $outfile
# do a dry-run to ensure vfs cache is warm
$scheme >/dev/null 2>&1 <<EOF
(define (fib n) (if (< n 2) n (+ (fib (- n 1)) (fib (- n 2)))))
(fib 10)
EOF
for n in `seq 1 40`
do
echo "$scheme (fib $n)"
{ time $scheme <<EOF
(define (fib n) (if (< n 2) n (+ (fib (- n 1)) (fib (- n 2)))))
(display (fib $n))
(newline)
EOF
} 2>&1 | grep real | awk '{print $2}' | sed 's/^0m//' | sed 's/s$//' | sed "s/^/$n,/" >> $outfile
done
done
@cellularmitosis
Copy link
Author

cellularmitosis commented May 18, 2020

@mgubi
Copy link

mgubi commented Dec 30, 2020

Why not add also Guile 1.8? As far as I know the interpreter there performs better than in the Guile 2 series. Only Guile 3 seems to be finally on par with that. Also s7 scheme ( https://ccrma.stanford.edu/software/snd/snd/s7.html ) is a popular choice for embedding and the author claims is quite fast. It would be interesting to compare these to chibi, for example.

@mgubi
Copy link

mgubi commented Dec 30, 2020

Update: Just out of curiosity I've tried (fib 40) on my machine (MacBook Air 2019) for Guile 1.8, Guile 2.2.7, Guile 3.0.4 and S7, this are the results:
Guile 1.8.8 : 40.595 sec
Guile 2.2.7 : 8.102 sec
Guile 3.0.4 : 3.864 sec
S7 : 3.297 sec
The results for Guile2/3 are from the compiled code (excluding the compilation time). If we force Guile2/3 to use only the interpreter (with GUILE_AUTO_COMPILE=0 and clean cache) we have
Guile 2.2.7: 120 sec
Guile 3.0.4: 60 sec

@cellularmitosis
Copy link
Author

cellularmitosis commented Dec 30, 2020

@mgubi interesting, thanks for contributing!

@decalek
Copy link

decalek commented Nov 16, 2021

@cellularmitosis, Hi Jason,
I just tested (fib 44) with current chez, janet, fennel [*] and jitterlisp:

  • 0m6.894s: fennel-1.0/luajit-2.1.b3:
  • 0m9.457s: chez-main
  • 0m12.712s: jitterlisp
  • 2m22.527s: janet-master

For you as a Janet contributor, probably the most interesting project will be jitter [1]. Jitter is a toolkit designed for building fast VMs without the "rocket science" needed for advanced compilers like LuaJIT or Chez. Jitterlisp is one of three demo VMs included in the source. I know about this "stealth" project and its genius author - Luca Saiu, only because GNU Poke [2] is implemented as Jittery VM. It would be wonderful, if turns out to be possible Janet VM to receive a significant speed boost only with refactoring its VM C code as a Jittery VM too :-)

[*] janet like dialect, compiles to Lua, initially also started by @bakpakin.
[1] http://git.ageinghacker.net/jitter/tree/example-vms/jitterlisp/examples/fibo.c
[2] https://git.savannah.gnu.org/cgit/poke.git/tree/libpoke/pvm.jitter

@cellularmitosis
Copy link
Author

cellularmitosis commented Dec 12, 2021

@decalek oh jitter sounds very interesting!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment