These notes support the article “What I learned trying to reduce my LLM costs” (September 8, 2026).
WhatIsThatBook helps people identify a book from what they remember about it. The question behind these experiments was straightforward: could I spend less on AI without making the app worse at finding books?
The result: cheaper models and shorter instructions often lost correct answers. Keeping the existing model and instructions, but using discounted Flex service, saved an estimated 53–59% on the tested AI generation calls. That is not a measured saving on the whole app bill, and the tests did not prove identical quality.
These are historical results for this application, not current prices or general model rankings. This summary contains aggregate findings only. It does not include reader queries, model responses, credentials, account identifiers, private repository links, source code, or internal tooling.
- First result: the correct book was the model's first suggestion.
- Within five: the correct book appeared among its first five suggestions.
- Raw answers: the suggested titles and authors were compared with expected answers.
- Catalog matches: the suggestions also passed through the app's lookup step, which connects them to actual book records. This is a different measure from raw answers.
- Flex: a discounted service option for the same model. An extra attempt or a switch to standard service can add cost.
A test set is a fixed collection of book-identification questions with expected answers. Good results on that set do not guarantee the same results for every reader.
Each model answered the same 100 test questions with the original instructions. Results went through the app's catalog lookup. Cost was estimated from token usage and the prices observed on August 18, excluding catalog lookup costs.
| Model | Correct first result | Correct within five | Estimated cost for 100 searches |
|---|---|---|---|
| Gemini 3.7 Flash | 69% | 71% | $0.1682 |
| GPT-5.6 Terra | 64% | 66% | $0.6893 |
| Qwen 3.7 Plus | 56% | 60% | $0.1099 |
| Nemotron 3.5 Lightning | 47% | 50% | $0.0223 |
The cheapest model lost 21 percentage points on the within-five measure compared with Gemini. The more expensive Terra model also found fewer correct books in this test.
Shortening Gemini's instructions reduced estimated cost from $0.1682 to $0.1118 per 100 searches, a 33.5% saving. First-result accuracy stayed at 69%; within-five accuracy rose from 71% to 73%. That small change came from one run and is not proof of improved quality.
The later test used 171 questions: the original 100 and an additional set of 71. Five original questions were labeled unanswerable; correctly declining to suggest a book counted as a success. The additional set had been used before, so it was not a fresh independent test.
A much shorter prompt combined with Flex cost about $0.13 versus $0.44 for the baseline: a 70.4% saving. But the quality result changed depending on the measurement:
| Within-five successes | Existing instructions, standard service | Shortest instructions, Flex |
|---|---|---|
| Raw answers, all 171 questions | 107 | 114 |
| Catalog matches, all 171 questions | 123 | 122 |
| Catalog matches, additional 71 questions | 48 | 45 |
Read across each row to compare the approaches. The rows use different scoring methods or question sets.
The shorter instructions looked better when comparing titles and authors alone. Once the app tried to find actual catalog records, they lost matches. The additional set exposed three lost answers that the overall score made easy to overlook. I rejected this version.
Revising the shorter instructions improved some checks but still lost those additional-set matches. That version was rejected too.
The selected change kept the existing model, instructions, and reasoning setting. It preferred Flex, allowed provider fallback, and added one standard-service attempt if generation returned no usable output. A valid “no matches” answer did not trigger that extra attempt.
I ran the same 171 questions twice with separately generated answers. These were 342 observations of 171 questions, not 342 independent questions.
Across both runs:
- Standard-service generation cost about $0.89.
- Flex generation cost $0.37–$0.42, depending on how two calls with unknown bills were accounted for.
- That produced the 53.2–58.5% estimated saving, before extra application attempts and fallback costs.
- Among the 332 observations where a book answer was expected, standard service found 234 within-five catalog matches; Flex found 231.
The same question produced no usable Flex output in each run. The cost range uses known charges at one end and a conservative allowance for those two unknown bills at the other. It does not assume fallback would recover the failures.
Small score differences also appeared when repeating the standard-service test. These results supported trying the more conservative change, but they did not establish equal quality or long-term reliability. The 70% saving with preserved quality was not achieved.
- Asking two cheap models to agree covered few questions and increased cost by about 15% once uncertain cases went to the original model.
- Asking for a second opinion on every “no matches” response added 29 calls for one extra correct match.
- A cheaper model for extracting titles from search snippets showed potential, but needed more validation and was left out of the change.
The article's calculator estimates:
Monthly saving = monthly bill × share eligible for the change × discount.
For a $100 bill, if 60% of spending is eligible and that portion gets a 55% discount, the saving is $33, leaving a $67 bill.
That is an illustration, not a prediction. Other AI operations, evaluation traffic, extra attempts, and standard-service fallback can reduce the saving on the total bill. The experiment itself cost about $3.61 against a $5 evaluation budget.
The article compares three evening windows, each from 17:00 to midnight UTC. September 7 spending was about 47% lower than September 6, with nearly the same number of identification outcomes. August 31 provides a same-weekday reference.
The billing export covered several projects. It was filtered to the three API keys associated with WhatIsThatBook; their names and identifiers are omitted here. Filtering by application name alone would miss requests with that field blank. Historical evaluation traffic also means the keys' full histories cannot be treated as production searches.
An afternoon model-failure period on September 7, mostly before the merge, would distort a whole-day comparison. The evening comparison uses completed hours, and latency includes only identification calls that returned usable responses. Fast failures therefore do not count as faster successful answers.
This is an observation, not a controlled experiment. The merge time is known, but the exact time the deployment reached users was not verified. Queries differed between evenings, individual charges could not be matched to searches, and every provider fallback was not visible. Slightly fewer searches returned results; their correctness still needs checking. Other AI stages also contribute to the bill. One evening cannot establish a lasting monthly saving or preserved match quality.