ml-serve is a small model server I wrote to learn production ML infrastructure: FastAPI, DistilBERT, CPU inference, dynamic request batching, a Redis cache, Prometheus metrics. I published two numbers in the README. Batching gave 3.8x throughput at 64 concurrent clients, and the cache gave 6.3x on a repeat-heavy workload.
Then I read On Evaluating Performance of LLM Inference Systems (Agrawal et al., 2025), which catalogs eight ways inference benchmarks go wrong, and went back through my own.