Skip to content

Instantly share code, notes, and snippets.

@schlamar
Created July 18, 2012 05:32
Show Gist options
  • Save schlamar/3134391 to your computer and use it in GitHub Desktop.
Save schlamar/3134391 to your computer and use it in GitHub Desktop.
Python serializer benchmark: json, simplejson, cPickle, messagepack, simpleubjson, marshal, literal_eval
System
======
Windows 7
Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)]
Small data
==========
marshal 1.02 us
cPickle 2.62 us
msgpack 4.17 us
json 9.37 us
simplejson 10.70 us
literal_eval 26.00 us
simpleubjson 57.40 us
Big data
========
msgpack 4.83 ms
marshal 5.76 ms
cPickle 7.04 ms
simplejson 15.40 ms
json 25.10 ms
literal_eval 118.00 ms
simpleubjson 13.10 s
In [9]: import json
In [10]: data = [1, 2, True, False, 'abcd']
In [11]: %timeit s = json.dumps(data); json.loads(s)
100000 loops, best of 3: 9.37 us per loop
In [12]: data = dict((i, str(i) * 10) for i in xrange(20000))
In [13]: %timeit s = json.dumps(data); json.loads(s)
10 loops, best of 3: 25.1 ms per loop
In [14]: import simplejson # with C extension
In [15]: data = [1, 2, True, False, 'abcd']
In [16]: %timeit s = simplejson.dumps(data); simplejson.loads(s)
100000 loops, best of 3: 10.7 us per loop
In [17]: data = dict((i, str(i) * 10) for i in xrange(20000))
In [18]: %timeit s = simplejson.dumps(data); simplejson.loads(s)
100 loops, best of 3: 15.4 ms per loop
In [4]: import cPickle
In [5]: data = [1, 2, True, False, 'abcd']
In [6]: %timeit s = cPickle.dumps(data, protocol=-1); cPickle.loads(s)
100000 loops, best of 3: 2.62 us per loop
In [7]: data = dict((i, str(i) * 10) for i in xrange(20000))
In [8]: %timeit s = cPickle.dumps(data, protocol=-1); cPickle.loads(s)
100 loops, best of 3: 7.04 ms per loop
In [1]: import msgpack
In [2]: data = [1, 2, True, False, 'abcd']
In [3]: %timeit s = msgpack.packb(data); msgpack.unpackb(s)
100000 loops, best of 3: 4.17 us per loop
In [4]: data = dict((i, str(i) * 10) for i in xrange(20000))
In [5]: %timeit s = msgpack.packb(data); msgpack.unpackb(s)
100 loops, best of 3: 4.83 ms per loop
In [4]: import simpleubjson
In [5]: data = [1, 2, True, False, 'abcd']
In [6]: %timeit s = simpleubjson.encode(data); simpleubjson.decode(s)
10000 loops, best of 3: 57.4 us per loop
In [9]: data = dict((str(i), str(i) * 10) for i in xrange(20000))
In [10]: %timeit s = simpleubjson.encode(data); simpleubjson.decode(s)
1 loops, best of 3: 13.1 s per loop
In [19]: import marshal
In [20]: data = [1, 2, True, False, 'abcd']
In [21]: %timeit s = marshal.dumps(data); marshal.loads(s)
1000000 loops, best of 3: 1.02 us per loop
In [22]: data = dict((i, str(i) * 10) for i in xrange(20000))
In [23]: %timeit s = marshal.dumps(data); marshal.loads(s)
100 loops, best of 3: 5.76 ms per loop
In [22]: import ast
In [23]: data = [1, 2, True, False, 'abcd']
In [24]: %timeit s = repr(data); ast.literal_eval(s)
10000 loops, best of 3: 26 us per loop
In [25]: data = dict((i, str(i) * 10) for i in xrange(20000))
In [26]: %timeit s = repr(data); ast.literal_eval(s)
10 loops, best of 3: 118 ms per loop
@shon
Copy link

shon commented Oct 20, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment