Skip to content

Instantly share code, notes, and snippets.

View jcrist's full-sized avatar

Jim Crist-Harif jcrist

View GitHub Profile
@jcrist
jcrist / bench.py
Created November 28, 2023 21:47
A quick benchmark of msgspec vs mashumaro to clear up misconceptions in a flyte issue
import sys
import importlib.metadata
import timeit
from dataclasses import dataclass
import msgspec
import orjson
from mashumaro.codecs.json import JSONEncoder, JSONDecoder
from mashumaro.codecs.orjson import ORJSONEncoder, ORJSONDecoder
@jcrist
jcrist / pypi.ipynb
Created August 2, 2023 20:34
Analyzing PyPI data with Ibis
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@jcrist
jcrist / altair_and_ibis.ipynb
Created August 1, 2023 16:58
A quick notebook demoing plotting in altair with ibis
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@jcrist
jcrist / benchmark.py
Last active July 11, 2023 16:05
Benchmark of msgspec, orjson, pydantic, ... taken from Python discord
# This is a modified version of `orig_benchmark.py`, using different data to
# highlight performance differences.
import json
import random
import string
import timeit
from statistics import mean, stdev
import orjson
import simdjson
@jcrist
jcrist / example_msgspec.py
Created February 17, 2023 22:09
An example of using `msgspec`, mirroring the examples at https://github.com/ArjanCodes/2023-attrs
from datetime import date
from enum import StrEnum, auto
from typing import Annotated
from msgspec import Struct, Meta
class OrderStatus(StrEnum):
OPEN = auto()
CLOSED = auto()
@jcrist
jcrist / bench_init.py
Created January 23, 2023 17:14
A benchmark comparing init performance of various dataclass-like libraries
"""A quick benchmark comparing how quickly `__init__` with default values runs
for various dataclass-like libraries.
We also compare against the time it takes to initialize a `dict` or `tuple`
with the same data, as a "low-bar" for pure-python implementations.
"""
import timeit
import attrs
@jcrist
jcrist / kv.py
Created September 19, 2022 01:48
An example TCP Key-Value store written using msgspec and asyncio
"""An example key-value store server and client implementation using msgspec
and asyncio.
Requests are serialized using the MessagePack protocol, as implemented by
msgspec. Additionally, all messages are length-prefix framed using a 32 bit
big-endian integer.
Note that this encoding isn't tied to either asyncio or msgspec - this could
just as easily be implemented using sockets and a different serialization
protocol. Length-prefix framing is useful in that respect - it separates the IO
@jcrist
jcrist / scotrail_ibis.py
Created August 25, 2022 13:21
Clips from a tweet about ibis-datasette analyzing the scotrail datasette
import ibis
from ibis import _
con = ibis.datasette.connect("https://scotrail.datasette.io/scotrail")
t = con.tables.announcements
def random(category):
return (
t.filter(_.Category == category)
.sort_by(ibis.random())
@jcrist
jcrist / Scotrail.ipynb
Created August 24, 2022 19:54
A Jupyter Notebook using ibis-datasette to analyze the ScotRail datasette
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@jcrist
jcrist / snippet1.ipy
Created August 18, 2022 14:29
Snippets used in a twitter thread about ibis-datasette
In [1]: import ibis
In [2]: ibis.options.interactive = True # enable interactive mode
In [3]: con = ibis.sqlite.connect("legislators.db") # connect to a database
In [4]: legislators = con.tables["legislators"] # access tables
In [5]: legislators.groupby("bio_gender").count() # query using a dataframe-like API
Out[5]: