This is the setup, the run prompt and the scoring I used for Qwen3.8-Flash-Next vs Claude Opus 5.5, made generic so you can run it with your own models, harnesses and hardware. Everything here is public. My 12 PRs are in deepu105/llamastash-b4r2 if you want to compare.
- Repo: llamastash/llamastash, latest
main. Write down the SHA you start from and use the same one for every run. The feature isn't in LlamaStash yet, it's the openTODO.mditem "Keep a model's prompt cache across an unload". I used623c28ca11c7f10969208ff4a81392e795f08a31if you want the exact same starting point. - A scratch GitHub repo for the PRs, with only
mainat your start SHA. Turn off its scheduled workflows. All runs open their PRs against it. - llama-server: a recent llama.cpp build. The feature uses
--slot-save-pathandPOST /slots/{id}?action=save|restore. A GPU build makes the agents' own tests a lot faster. - Test model:
Llama-3.2-1B-Instruct-Q4_K_M.gguf(bartowski). Small, so the end-to-end test is quick. - Rust and cargo-nextest, and the repo's pre-push hook (cargo-husky installs it on the first
cargo test).
The task asks for save and restore numbers at 100k tokens. Measuring that inside every run cost me 48 to 53 minutes per run on a CPU, and it doesn't change the code quality. So measure it once yourself and paste the numbers into the prompt.
- Start llama-server with the test model,
-cas large as fits (100k if it fits, otherwise the largest that does), all layers on the GPU, and--slot-save-path <dir>. - Send a prompt that nearly fills the context. Note
timings.prompt_msfrom the response. That is the reprocess time. POST /slots/0?action=savewith{"filename": "spike.bin"}. Notetimings.save_msand the file size.- Restart llama-server,
POST /slots/0?action=restorewith the same filename. Notetimings.restore_ms. Do it again after dropping the page cache if you want the cold number. - Send the same prompt plus a short follow-up. Note how many prompt tokens came from the cache (
tokens_cachedortimings.cache_n, depending on the build).
Run each step 2 to 4 times and keep the range.
- One fresh clone per run at your start SHA, with an empty
target/. One run at a time on the same machine, with nothing else heavy running. - The same prompt for every run. Only
<BRANCH>and<SCRATCH>change. - 3 hour limit. If the agent stops before it is done, send one
continue. A second stop counts as did not finish. - Time runs from sending the prompt to the PR being open and the final summary posted. CI runs afterwards and is not in the time.
- No other human input. Set the harness up so it doesn't stop on permission dialogs: for example Claude Code with
--permission-mode auto --add-dir <SCRATCH>, and Pi with any guard extension off. Note which harness sends its permission checks to the model, since that costs time and tokens. - The same env for every run. I used
NEXTEST_RETRIES=2(the repo's CI profile also retries twice),NEXTEST_TEST_THREADS=8on a 16-thread CPU, andTMPDIRpointed at<SCRATCH>. - Record the versions of the harness, the model (and quant, if local), llama.cpp, Rust and nextest.
- Do 2 or more runs per combination if you can. I did one per combination per prompt, and that is my biggest caveat.
Per run, record: time, number of turns and continues, input / cached / output tokens, and cost at API prices.
Fill in the placeholders. <SPIKE_RESULTS> is what you measured in step 2.
You are in a fresh clone of llamastash at commit <SHA>, on branch <BRANCH>. Do all the work here.
Task (TODO.md and docs/plans/2026-10-05-001-feat-r14-batches-plan.md, Batch 4):
"Keep a model's prompt cache across an unload. When make-room or the idle sweep stops a llama.cpp launch mid-conversation, the next request reprocesses the whole prompt. llama-server can write a slot's KV cache to disk and read it back: --slot-save-path PATH plus POST /slots/{id_slot}?action=save|restore (present on local build 11390 and in upstream tools/server/README.md, checked 2026-10-05). Nothing in src/ uses it. Spike first: measure save time, restore time and file size at a large context (100k tokens) against plain reprocessing. Then decide when to save (before an eviction only), where the files live, how they are capped and cleaned up, and how a restored slot is matched to the returning conversation."
The spike is already done. Do not repeat it, and keep every prompt you send to the test model under 20,000 tokens. Results with the test model on this machine:
<SPIKE_RESULTS: context size, hardware and llama.cpp build, then reprocess time, save time, restore time (page cache warm and dropped), file size, and cached tokens on the follow-up after a restore>
Rules
- Work from scratch. Do not look at any other branch, pull request, clone or worktree.
- Do not ask me anything and do not stop to check in. If something is unclear, pick the reasonable option, note it for the PR, and continue until done.
- Do not use child agents. Do all the work in the main session.
Environment
- llama-server: <LLAMA_SERVER>
- Test model: <TEST_MODEL>
- Run llama-server <on the GPU with all layers offloaded | on the CPU with N threads>, and one server at a time.
- Put temporary files, slot save files and daemon directories under <SCRATCH>, not /tmp.
- Any llamastash daemon you start gets its own LLAMASTASH_STATE_DIR, LLAMASTASH_CONFIG_DIR and LLAMASTASH_CACHE_DIR and non-default ports. Do not send test requests to the model endpoint you are running on.
- While coding, run narrow tests only: cargo nextest run --features test-fixtures --lib <filter>, or --test <file>. The full suite runs once, at the end, in the pre-push hook.
Done means
1. The spike numbers above are in a doc under docs/spikes/.
2. End-to-end proof on an isolated daemon with the test model: send a prompt, let the idle sweep or make-room stop the launch, send the follow-up, and show that the prompt came from the restored cache (cached token count and time, before and after).
3. Unit tests cover the new code. Docs, config example, CHANGELOG and TODO.md are updated as AGENTS.md requires.
4. The pre-push hook passes. If it fails on tests your change does not touch, run it once more; if it then passes, note the flake. Do not use --no-verify.
5. A PR against main is open, titled <BRANCH>, with these sections: Spike numbers, Design decisions (when to save, where files live, cap and cleanup, slot matching), End-to-end proof, Tests, Known limits.
Then stop. Do not wait for CI or poll it; I check CI myself. End with a summary of at most 10 lines.
If you'd rather have each agent run the spike itself, drop the "The spike is already done" paragraph and change item 1 to "Spike numbers at 100k tokens are in a doc under docs/spikes/: reprocess time, save time, restore time, file size." Expect about an hour more per run on a CPU.
Run this by hand or with a script, the same way for every PR:
- Build the PR. Start a daemon with its own state, config and cache dirs and non-default ports,
proxy.idle_ttl_secs: 20, ctx 32768, all layers on the GPU, and the PR's own on switch underbackend.llamacpp(each PR names it differently, read itsconfig.example.yaml). - Send a prompt of about 15,000 tokens to the test model.
- Wait until the idle sweep stops llama-server (about 35 s with a 20 s TTL).
- Send a follow-up that repeats the conversation plus a short new turn.
- Pass if the launch stopped, a new llama-server process answered, and the follow-up's cached tokens are at least 90% of the first prompt.
Controls: the same PR with its switch off, and the start commit. Both should show 0 cached tokens. For reference, all 12 of my PRs passed: 15,000 cached tokens, follow-up in 1.4 s against 5.2 to 6.1 s without the feature.
Prepare each PR so the reviewer can't tell which run made it:
git diff <SHA>..<PR head>per PR, with the branch name replaced byNEUTRALRUNeverywhere.- Give each diff a random id (A, B, C and so on) and keep the id-to-run map to yourself.
- Check out each PR head in its own directory so the reviewer can read unchanged code around the diff.
Then give this prompt to at least two different reviewer models, each in its own session:
You are reviewing <N> diffs blind, ids <A..>. Each diff is at <DIFF_DIR>/<id>.diff, against commit <SHA> of llamastash, and the full tree at the PR head is at <TREE_DIR>/<id>/. Do not try to find out which model or harness wrote a diff. Do not build or run tests. Read code only.
The task each author was given is in <TASK_FILE>. Read it first.
For each id, answer the 7 questions below. For each question, list every problem you find with file:line evidence, give each problem a severity, and give the answer the grade of its worst problem. Several problems of the same severity do not add up. A missing protection is graded by what it lets happen, not as 0.
| Grade | Severity | Problem |
|---|---|---|
| 1 | none | Nothing a user could hit |
| 0.75 | low | An edge case llama-server refuses anyway, an untested edge path, a comment naming the backend, a missing README or SECURITY row, a plan or spike doc that overstates, a small unrelated edit |
| 0.5 | medium | A wrong cache restored only in a rare setup (LoRA, rope, a model file replaced at the same path), a save lost in a rare race, disk that is never freed, untested core save/restore paths, a user doc (README, usage, architecture, config example) that states wrong behaviour, generic code reading a backend's config |
| 0.25 | high | A user can get a cut-off or wrong answer in a case the code is meant to handle |
Q1. Restore safety. Is a restore into another model, a replaced model file at the same path, or another launch config (ctx, KV cache type, LoRA, rope, extra args, -np, draft model) refused or keyed away? Read the key or fingerprint function and the restore path. Things llama-server itself refuses (a KV layout or ctx that does not fit) are safe.
Q2. Caps and cleanup. Are save files capped and cleaned up? Look for a total cap, an age limit, delete after restore, partial files after a failed save, and cleanup when the feature is turned off or at daemon start.
Q3. Backend neutrality. AGENTS.md (section "Adding a backend") says no backend id or name may appear, in code or comments, outside src/backend/<id>/, src/backend/mod.rs and the backend's own config struct. Check with:
git diff -U0 <SHA>..HEAD -- src ':!src/backend/llama_cpp' ':!src/backend/mod.rs' | rg '^\+.*(?i:llama[._ -]?cpp|llamacpp|llama-server)'
A re-export of the backend's config from crate::config is allowed.
Q4. Tests. Would the tests fail if the new logic were wrong? Unit tests on helpers only, against integration tests that run save and restore through the eviction path on the fake llama-server (tests/fixtures/fake_llama_server.rs). Are the PR's own safety checks tested?
Q5. Docs. Check every file in the AGENTS.md "Docs ship with code" drift list, including the README config key table and SECURITY.md (the feature writes conversation tokens to disk and turns on llama-server's slot endpoints). Check each behaviour claim in the user docs against the code. A doc that describes something the code does not do is medium.
Q6. Scope. List changes the task did not ask for, with lines added. A change the feature needs is not a problem.
Q7. Eviction safety. Read src/proxy/eviction.rs around the save hook and any claim on the managed model. (a) After the save, does it check for in-flight requests again before stopping, and keep the launch running if one came in? (b) Can a second sweep tick or make-room save and stop the same launch while the first eviction is still saving? Grade what each missing guard lets happen. A request cut off during a save that takes under a second is a rare race: medium.
Write one file per id: the 7 grades, each problem with severity and file:line, and a one-line verdict (merge as-is, merge after fixes, reject). Write each file as soon as that id is done.
<TASK_FILE> is the run prompt from section 4 with the branch name replaced by NEUTRALRUN.
Read each PR's checks once CI is done. A failure counts only if the PR caused it: check that the failing test is new or changed in the diff.
| Grade | CI result |
|---|---|
| 1 | All checks pass. A flaky failure in a test the PR doesn't touch, or a failure also seen on main, counts as a pass |
| 0.75 | A test the PR added fails or hangs (the test is wrong, or the cause is not known) |
| 0.5 | The code is shown to be wrong on one platform |
| 0.25 | The code is wrong on every platform, or the build, lint or format check fails |
- Per answer, average the reviewers' grades.
- Where two reviewers disagree, check the code. If one of them got a fact wrong or missed one, use the other's grade. If they agree on the facts and only judge the severity differently, keep the average.
- Per PR, score = Q1 to Q7 plus CI, out of 8. The % is that over 8. The floor is 0.25 per item, so the lowest possible score is 2 (25%).
- Per combination, average the PRs.
Report time, tokens and cost next to the score, not in it. I didn't score acceptance (all passed), diff size, or prompt rule breaks.
The % depends on the scheme, so publish the per-question grades with the totals. On my data, Opus / Claude Code + Flash-Next / Pi + Flash-Next came out at 82% / 68% / 66% with the grades above, 82% / 51% / 46% with a 0 / 0.5 / 1 scale, and 83% / 43% / 40% with Q1 and Q7 weighted 3x and Q2 and Q4 2x (the last two on Q1 to Q7 only, without CI).