Python serializer benchmark: json, simplejson, cPickle, messagepack, simpleubjson, marshal, literal_eval

!Python serialization

System
======

Windows 7
Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)]


Small data
==========

marshal          1.02 us
cPickle          2.62 us
msgpack          4.17 us
json             9.37 us
simplejson      10.70 us
literal_eval    26.00 us
simpleubjson    57.40 us


Big data
========

msgpack          4.83 ms
marshal          5.76 ms
cPickle          7.04 ms
simplejson      15.40 ms
json            25.10 ms
literal_eval   118.00 ms
simpleubjson    13.10 s

01_json.py

In [9]: import json

In [10]: data = [1, 2, True, False, 'abcd']

In [11]: %timeit s = json.dumps(data); json.loads(s)
100000 loops, best of 3: 9.37 us per loop

In [12]: data = dict((i, str(i) * 10) for i in xrange(20000))

In [13]: %timeit s = json.dumps(data); json.loads(s)
10 loops, best of 3: 25.1 ms per loop

02_simplejson.py

In [14]: import simplejson  # with C extension

In [15]: data = [1, 2, True, False, 'abcd']

In [16]: %timeit s = simplejson.dumps(data); simplejson.loads(s)
100000 loops, best of 3: 10.7 us per loop

In [17]: data = dict((i, str(i) * 10) for i in xrange(20000))

In [18]: %timeit s = simplejson.dumps(data); simplejson.loads(s)
100 loops, best of 3: 15.4 ms per loop

03_cpickle_binary.py

In [4]: import cPickle

In [5]: data = [1, 2, True, False, 'abcd']

In [6]: %timeit s = cPickle.dumps(data, protocol=-1); cPickle.loads(s)
100000 loops, best of 3: 2.62 us per loop

In [7]: data = dict((i, str(i) * 10) for i in xrange(20000))

In [8]: %timeit s = cPickle.dumps(data, protocol=-1); cPickle.loads(s)
100 loops, best of 3: 7.04 ms per loop

04_messagepack.py

In [1]: import msgpack

In [2]: data = [1, 2, True, False, 'abcd']

In [3]: %timeit s = msgpack.packb(data); msgpack.unpackb(s)
100000 loops, best of 3: 4.17 us per loop

In [4]: data = dict((i, str(i) * 10) for i in xrange(20000))

In [5]: %timeit s = msgpack.packb(data); msgpack.unpackb(s)
100 loops, best of 3: 4.83 ms per loop

05_simpleubjson.py

In [4]: import simpleubjson

In [5]: data = [1, 2, True, False, 'abcd']

In [6]: %timeit s = simpleubjson.encode(data); simpleubjson.decode(s)
10000 loops, best of 3: 57.4 us per loop

In [9]: data = dict((str(i), str(i) * 10) for i in xrange(20000))

In [10]: %timeit s = simpleubjson.encode(data); simpleubjson.decode(s)
1 loops, best of 3: 13.1 s per loop

06_marshal.py

In [19]: import marshal

In [20]: data = [1, 2, True, False, 'abcd']

In [21]: %timeit s = marshal.dumps(data); marshal.loads(s)
1000000 loops, best of 3: 1.02 us per loop

In [22]: data = dict((i, str(i) * 10) for i in xrange(20000))

In [23]: %timeit s = marshal.dumps(data); marshal.loads(s)
100 loops, best of 3: 5.76 ms per loop

07_literal_eval.py

In [22]: import ast

In [23]: data = [1, 2, True, False, 'abcd']

In [24]: %timeit s = repr(data); ast.literal_eval(s)
10000 loops, best of 3: 26 us per loop

In [25]: data = dict((i, str(i) * 10) for i in xrange(20000))

In [26]: %timeit s = repr(data); ast.literal_eval(s)
10 loops, best of 3: 118 ms per loop

Little modified version of this https://gist.github.com/shon/6f04a9821cb178e3de6cb38b7fea75af