ivanium

#!/usr/bin/env bash
# Benchmark CPU KV cache offloading overhead.
# Launches a vllm server, runs `vllm bench serve` against it, then
# repeats with offloading enabled. Uses random (unique) prompts so
# no offloaded prefix is ever hit — isolating the pure store overhead.
#
# Usage:
#   bash benchmarks/benchmark_cpu_offloading.sh [MODEL] [INPUT_LEN] [OUTPUT_LEN] [NUM_PROMPTS]
#
# Environment variables:

ivanium

/**
 * @file bench_bw.cpp
 * @compile nvcc -o bench_bw bench_bw.cpp -O2 -g -std=c++14
 */

#include <cuda_runtime.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

ivanium

# This isn't supposed to run as a bash script, i named it with ".sh" for syntax highlighting.

# https://developer.nvidia.com/nsight-systems
# https://docs.nvidia.com/nsight-systems/profiling/index.html

# My preferred nsys (command line executable used to create profiles) commands
#
# In your script, write
# torch.cuda.nvtx.range_push("region name")
# ...

ivanium

Keybase proof

I hereby claim:

• I am ivanium on github.
• I am ivanium97 (https://keybase.io/ivanium97) on keybase.
• I have a public key ASDT37crHhceBToAmdPE9F09wd6Kc601D63x-GSRdQzBRQo

To claim this, I am signing this object: