Jiffy Benchmarks

jiffybench.org

Benchmark Code

#!/usr/bin/env escript
%% -*- erlang -*-
%%! -smp enable -sname jsonbench -pa deps/jiffy/ebin
-mode(compile).

main([Iter]) ->
    BinaryJson = read_data(),
    DecodedJson = jiffy_decode(BinaryJson),
    bench(list_to_integer(Iter), BinaryJson, DecodedJson).

bench(0, _, _) ->
    ok;
bench(Iter, BinaryJson, DecodedJson) ->
    io:format("~p ~p~n", [
                          timeit(BinaryJson, fun jiffy_decode/1),
                          timeit(DecodedJson, fun jiffy_encode/1)
                         ]),
    bench(Iter - 1, BinaryJson, DecodedJson).

diff(End, Start) ->
    tuple_to_micro(End) - tuple_to_micro(Start).

tuple_to_micro({MegaSec, Sec, MicroSec}) ->
    (MegaSec * 1000000) + (Sec * 1000) + (MicroSec div 1000).

read_data() ->
    {ok, Data} = file:read_file("big.json"),
    Data.

timeit(Data, TestFun) ->
    Start = erlang:now(),
    _Res = TestFun(Data),
    End = erlang:now(),
    diff(End, Start).

jiffy_encode(Data) ->
    jiffy:encode(Data).

jiffy_decode(Data) ->
    jiffy:decode(Data).

Benchmark Procedure

Jiffy Version

Two jiffy versions were used. The “new” jiffy is from https://github.com/joedevivo/jiffy at revision 2f405e2b9ae3c2a9cf59ab10179c3262cf4aff03. The “old” version was taken from the rebar.config.lock of the erchef (OSS Chef Server 11) repository: revision 3b6a1327d418f7594ecb23aa8d7874c3986f433e from github.com/davisp/jiffy

Test Data

The test data is a 4.7MB JSON file representing the cookbook manifest returned by the cookbook_versions endpoint. This file was created by uploading 20 cookbooks, each with 600 files and then calling the cookbook_versions endpoing:

for i in {1..20}; do knife cookbook create cookbook${i}; done
for ckbk in /var/chef/cookbooks/cookbook*; do for i in {1..600};do echo $RANDOM-$RANDOM-$RANDOM-$RANDOM > ${ckbk}/recipes/${i}.rb; done; done
knife exec -E 'puts api.post("environments/_default/cookbook_versions", :run_list => ["cookbook1", "cookbook2", "cookbook3", "cookbook4", "cookbook5", "cookbook6", "cookbook7", "cookbook8", "cookbook9", "cookbook10", "cookbook11", "cookbook12", "cookbook13", "cookbook14", "cookbook15", "cookbook16", "cookbook17", "cookbook18", "cookbook19", "cookbook20"]).to_json'

Procedure

1. Edit rebar.lock to point at the desired version of jiffy
2. Get and compile jiffy: rm rf deps/ && rebar get-deps && rebar comile
3. Run 1000 encodes and decodes: ./jsonbench 1000 > output.txt

Results

So far I’ve only looked at simple summary statistics, but the picture seems to be clear, new jiffy isn’t slow and is probably faster.

Old Decode	Old Encode	New Decode	New Encode
Min. :41.00	Min. : 80.00	Min. :18.00	Min. : 57.00
1st Qu.:52.00	1st Qu.: 87.00	1st Qu.:21.00	1st Qu.: 62.00
Median :55.00	Median : 90.00	Median :23.00	Median : 64.00
Mean :56.14	Mean : 90.26	Mean :24.32	Mean : 64.64
3rd Qu.:62.00	3rd Qu.: 93.00	3rd Qu.:25.00	3rd Qu.: 66.00
Max. :76.00	Max. :123.00	Max. :69.00	Max. :180.00

This table is the partial output of:

paste oldjiffy.txt newjiffy.txt | summarize

summarize is an R script that can be found here:

https://gist.github.com/stevendanna/2014462