Benchmark of msgspec, orjson, pydantic, ... taken from Python discord

benchmark.py

# This is a modified version of `orig_benchmark.py`, using different data to
# highlight performance differences.
import json
import random
import string
import timeit
from statistics import mean, stdev

import orjson
import simdjson
import msgspec
import pydantic


def describe_json(buf: bytes) -> None:
    """Describe the type of values found in a JSON message"""
    json_types = [
        ("objects", dict),
        ("arrays", list),
        ("strs", str),
        ("ints", int),
        ("floats", float),
        ("bools", bool),
        ("nulls", type(None)),
    ]
    counts = dict.fromkeys([v for _, v in json_types], 0)

    def inner(obj):
        typ = type(obj)
        counts[typ] += 1
        if typ is list:
            for i in obj:
                inner(i)
        elif typ is dict:
            for k, v in obj.items():
                inner(k)
                inner(v)

    inner(msgspec.json.decode(buf))

    total = sum(counts.values())
    print("JSON Types:")

    results = [(k, counts[v]) for k, v in json_types if counts[v]]
    results.sort(key=lambda row: row[1], reverse=True)
    for kind, count in results:
        print(f"- {kind}: {count} ({count/total:.2f})")


random.seed(42)


def randstr():
    return "".join(random.choices(string.printable, k=10))


class ItemStruct(msgspec.Struct):
    name: str
    value: int


class UserStruct(msgspec.Struct):
    username: str
    exp: float
    level: float
    items: list[ItemStruct]


class ItemPydantic(pydantic.BaseModel):
    name: str
    value: int


class UserPydantic(pydantic.BaseModel):
    username: str
    exp: float
    level: float
    items: list[ItemPydantic]


N = 10000
msg = msgspec.json.encode(
    [
        UserStruct(
            randstr(),
            random.random(),
            random.uniform(0, 100),
            [
                ItemStruct(randstr(), random.randint(0, 100))
                for _ in range(random.randrange(10, 20))
            ],
        )
        for _ in range(N)
    ]
)

BENCHMARKS = [
    ("stdlib json", json.loads),
    ("orjson", orjson.loads),
    ("simdjson", simdjson.loads),
    ("msgspec-dict", msgspec.json.decode),
    ("msgspec-struct", msgspec.json.Decoder(list[UserStruct]).decode),
    ("pydantic-v2", pydantic.TypeAdapter(list[UserPydantic]).validate_json),
]

describe_json(msg)

print("")
print("Benchmarks:")
results = {}
for name, fun in BENCHMARKS:
    results[name] = times = timeit.repeat(
        "loads(msg)",
        repeat=20,
        number=10,
        globals={"loads": fun, "msg": msg},
    )
    print(f"- {name}: {mean(times):.2f} ± {stdev(times):.2f}")

orig_benchmark.py

# This is a cleaned up version of the original benchmark. Semantically it's the
# same, just a bit easier to read. The performance numbers also match those
# seen on discord.
import json
import random
import string
import timeit
from statistics import mean, stdev

import orjson
import simdjson
import msgspec
import pydantic


def describe_json(buf: bytes) -> None:
    """Describe the type of values found in a JSON message"""
    json_types = [
        ("objects", dict),
        ("arrays", list),
        ("strs", str),
        ("ints", int),
        ("floats", float),
        ("bools", bool),
        ("nulls", type(None)),
    ]
    counts = dict.fromkeys([v for _, v in json_types], 0)

    def inner(obj):
        typ = type(obj)
        counts[typ] += 1
        if typ is list:
            for i in obj:
                inner(i)
        elif typ is dict:
            for k, v in obj.items():
                inner(k)
                inner(v)

    inner(msgspec.json.decode(buf))

    total = sum(counts.values())
    print("JSON Types:")

    results = [(k, counts[v]) for k, v in json_types if counts[v]]
    results.sort(key=lambda row: row[1], reverse=True)
    for kind, count in results:
        print(f"- {kind}: {count} ({count/total:.2f})")


random.seed(42)


def randstr():
    return "".join(random.choices(string.printable, k=10))


class UserStruct(msgspec.Struct):
    username: str
    exp: float
    level: float
    last_values: list[float]


class UserPydantic(pydantic.BaseModel):
    username: str
    exp: float
    level: float
    last_values: list[float]


N = 10000
msg = msgspec.json.encode(
    [
        UserStruct(
            randstr(),
            random.random(),
            random.uniform(0, 100),
            [random.uniform(0, 100) for _ in range(random.randrange(10, 20))],
        )
        for _ in range(N)
    ]
)

BENCHMARKS = [
    ("stdlib json", json.loads),
    ("orjson", orjson.loads),
    ("simdjson", simdjson.loads),
    ("msgspec-dict", msgspec.json.decode),
    ("msgspec-struct", msgspec.json.Decoder(list[UserStruct]).decode),
    ("pydantic-v2", pydantic.TypeAdapter(list[UserPydantic]).validate_json),
]

describe_json(msg)

print("")
print("Benchmarks:")
results = {}
for name, fun in BENCHMARKS:
    results[name] = times = timeit.repeat(
        "loads(msg)",
        repeat=20,
        number=10,
        globals={"loads": fun, "msg": msg},
    )
    print(f"- {name}: {mean(times):.2f} ± {stdev(times):.2f}")

On the python discord someone posted a benchmark comparing msgspec, orjson, pydantic, simdjson, ... This original benchmark shows msgspec decoding and validating JSON to be ~the same performance (or a bit slower) as orjson decoding it alone.

While nice (msgspec is doing more for the user in the same amount of time), this didn't match performance numbers seen on the msgspec website here. I was curious about the differences - what about the message structure here was leading to different results?

After asking for the benchmark (which was output from an .ipynb), I spent some time cleaning it up but changing none of the semantic details. This is orig_benchmark.py. I then played around a bit with inspecting the message structure, and arbitrarily changing it to see how that affected the results. A similar message with a different structure shown in benchmark.py.

Results

Running these on my machine:

• orig_benchmark.py

$ python orig_benchmark.py 
JSON Types:
- floats: 165390 (0.70)
- strs: 50000 (0.21)
- arrays: 10001 (0.04)
- objects: 10000 (0.04)

Benchmarks:
- stdlib json: 0.46 ± 0.01
- orjson: 0.10 ± 0.01
- simdjson: 0.12 ± 0.00
- msgspec-dict: 0.11 ± 0.00
- msgspec-struct: 0.10 ± 0.00
- pydantic-v2: 0.37 ± 0.01

• benchmark.py

$ python benchmark.py 
JSON Types:
- strs: 485663 (0.60)
- objects: 155221 (0.19)
- ints: 145221 (0.18)
- floats: 20000 (0.02)
- arrays: 10001 (0.01)

Benchmarks:
- stdlib json: 0.82 ± 0.00
- orjson: 0.45 ± 0.00
- simdjson: 0.70 ± 0.01
- msgspec-dict: 0.47 ± 0.02
- msgspec-struct: 0.25 ± 0.00
- pydantic-v2: 2.02 ± 0.01

Analysis

The two benchmarks above show different results for the same parsers. The first one shows msgspec performing ~the same as orjson, the second one shows it performing ~2x faster than orjson. What's going on?

There are two main differences in the message structures here:

1. Fewer floats.

JSON is composed of 7 core types (object, array, str, int, float, bool, null). When benchmarking individual types for the core parsing routines, msgspec's float parser is known to be a bit slower (~15% slower) than orjson's, while the other core type parsing routines are approximately equivalent (we're slightly faster at ints for some reason).

The message used in the original benchmark is 70% floats, which in my experience is much higher than average for JSON data. Most JSON apis I've interacted with are composed of mostly object, int, and str types, so I haven't spent a ton of time optimizing float performance. If float parsing performance is a user priority we could spend more time on this to close the gap.

The second benchmark adjusts the ratios a bit to skew more towards strings/ints. This makes a small difference, but isn't the main reason for the different results.

2. More objects

This is the biggest difference. Decoding into a msgspec.Struct type is significantly cheaper than decoding into a dict. Messages that have a higher ratio of object types will hit this hot path more frequently, resulting in a larger relative speedup. In this case we've replaced the list of ints with a list of small objects.

Conversely, pydantic does much worse here, since allocating a pydantic BaseModel is relatively more expensive than allocating a dict.

Closing thoughts

Benchmarks are hard. As we've seen here, the structure of your messages matters a ton to determine how well a certain JSON library will handle it. When benchmarking it's important to ensure the structure of the messages you're using matches those in your real world use case.