You cut your LLM bill 30% in the first two weeks. Then the curve went flat. The obvious fixes are done, the bill stopped shrinking, and you're not sure if there's more to find or if 30% is just where this ends.
It isn't. Across public production data, academic benchmarks, and the aggregate patterns visible in Preto's own traffic, the same five fixes recur in almost every team that gets from "we're overspending" to a durable 40-60% reduction — and they tend to happen in the same order, on the same timeline, for the same reasons.
1. The first three fixes — model routing, provider-native prompt caching, duplicate-request caching — account for most of the reduction and ship in under two weeks.
2. The last two — system prompt audits and batch processing — add another 10-20 points but take longer to find and require ongoing discipline.
3. Savings that aren't protected by budget enforcement don't stay saved. The most common way a 50% reduction becomes a 20% reduction six months later is a single unmonitored feature regressing.
The Shape of Every Cost Reduction Curve
Before the individual patterns, the shape matters more than any single number. Nearly every team's cost curve — whether it's a five-person indie SaaS or a Series A engineering org — follows the same three stages.
The Five Patterns
Here's what fills in that curve, in the order teams typically find and fix them.
Almost every codebase that adds an LLM call starts with one model hardcoded everywhere — usually the most capable one available at the time, because that's the path of least resistance during a demo or MVP. It stays that way until someone looks at the bill by endpoint.
Berkeley's RouteLLM benchmark is the anchor number here: a matrix-factorization router sending only 26% of calls to the expensive model preserves 95% of GPT-4-class quality on MT-Bench at roughly 85% lower cost. The pattern in production traffic mirrors this closely — classification, extraction, and short Q&A almost never need the frontier model, and those categories are usually 40-60% of total request volume.
This is consistently the fastest-shipping fix and the largest single line-item drop, which is why it's almost always pattern one.
Both OpenAI and Anthropic ship prompt caching that most teams never explicitly enable or architect for. OpenAI's is automatic on prompts ≥1,024 tokens; Anthropic's requires explicit cache_control markers but returns cache hits at 10% of base input price per Anthropic's own pricing documentation. The catch is prefix stability — a timestamp or user ID placed at the top of a system prompt invalidates the entire cache on every call.
The clearest public case of this pattern: engineer Du'An Lightfoot documented going from $720/month to $72/month — a 90% cut — after restructuring a stable system prompt to sit at the front of the request and moving dynamic content to the end. That's the upper bound; most production traffic with mixed prompt patterns lands at 20-40% blended reduction once caching is architected correctly.
This is distinct from provider-native caching — it's catching identical or near-identical requests your own app is sending. Helicone's open dataset of 1.5B+ requests shows this pattern at scale, and the MeanCache research paper's analysis of 27,000 real queries found roughly 31% of a user's queries are similar enough to a prior query to safely cache-hit. Exact-match caching (hash the prompt, return the stored response) catches the easy half; semantic caching (embedding similarity) catches the rest, at the cost of needing a quality gate to prevent false-positive hits.
The typical production hit rate once both layers are running is 20-30%, rising to 90%+ on narrow, highly repetitive workloads like FAQ bots — a range corroborated by Helicone's caching documentation and an AWS-published benchmark showing 86% cost reduction on repetitive query patterns.
This one is slower to find because it's not a single expensive call — it's a small tax paid on every single request. Few-shot examples that no longer move quality, formatting instructions the model already follows by default, politeness boilerplate that never shows up in output: all of it gets paid for, every time, indefinitely.
Microsoft Research's LLMLingua benchmark demonstrates up to 20x compression with roughly 1.5% performance loss on GSM8K — the research ceiling. In practice, a manual audit against your own eval set (not an automated compressor) typically finds 25-50% of a production system prompt is removable without measurable quality change, because most of what accumulates in a system prompt over six months of iteration was added defensively and never removed.
OpenAI's Batch API and Anthropic's Message Batches API both offer a flat 50% discount on input and output tokens in exchange for a 24-hour completion window. The pattern here isn't technical difficulty — it's that teams underestimate how much of their traffic doesn't actually need to return in under five seconds. Nightly enrichment jobs, eval suites, bulk classification backfills, and weekly digest generation all commonly run through the real-time endpoint by default, simply because that's how the first version was built.
Quora is publicly on record using Anthropic's Batches API for summarization and highlight extraction, describing it as far simpler than "dealing with the complexity of running many parallel live queries." A documented customer-support nightly pipeline moved from $3,750 to $1,875 per month on identical volume — a flat 50% — by making this single switch.
Want to see where your own curve stands?
Preto tags your traffic by feature and model, so you can see which of these five patterns is your biggest remaining opportunity — with a dollar estimate per fix.
Get the Before/After Dashboard TemplateTrack your own cost reduction. Plug in your numbers. See where you stand.
What the Stacked Curve Looks Like on a Real Bill
Applying realistic mid-range impact for each pattern against a starting bill, in the order teams typically ship them:
| Stage | Change | Running Total |
|---|---|---|
| Starting bill | — | $12,000/mo |
| Model routing (35% cut on routable ~50% of traffic) | −17.5% | $9,900 |
| Provider-native caching (25% on cache-friendly share) | −14% | $8,514 |
| Duplicate-request caching (22% hit rate) | −12% | $7,492 |
| System prompt audit (30% cut on top prompts) | −6% | $7,042 |
| Batch API on eligible non-realtime share | −5% | $6,690 |
| Final bill | −44% | $6,690 |
Deliberately conservative — real workloads have overlap and diminishing returns between fixes, so the effects don't simply add. The 40-60% headline range spans teams that stop after fix three (low end) and teams that carry all five through and keep measuring (high end).
Where Teams Get Stuck
The plateau at month two isn't usually because the easy wins ran out — it's because measurement stopped at the same time the easy wins did.
The regression nobody catches. A cache hit rate that quietly drops from 30% to 12% over six weeks, because a new feature started injecting a timestamp into the cached prefix, looks identical to "normal variance" on a dashboard nobody's checking daily. It shows up as an unexplained bill increase two months later, disconnected from the change that caused it.
The second failure mode is sharper: a single unmonitored incident erasing months of disciplined savings in hours. Documented cases include a Stanford lab burning $9,200 of GPT-4o in 12 hours on a forgotten notebook token, and a multi-agent production loop running 11 days and $47,000 before anyone noticed. Neither team lacked the will to control costs — they lacked a hard floor. Per-key and per-feature budget enforcement at the proxy layer, with alerting before the hard cutoff, is what turns a one-time 40-60% cut into a number that stays cut.
The teams that hold their savings share one habit: cost-per-feature tracking doesn't stop being reviewed once the big fixes ship. It becomes a standing line item in the same weekly review as error rates and latency — not a one-time audit, but a permanent instrument.
That's the difference between a 40-60% cut and a 40-60% cut that's still 40-60% six months later. Preto keeps the five-pattern audit running continuously against live traffic, so the regression that would otherwise take two months to notice shows up the week it starts.
For the full breakdown of each fix with implementation gotchas, see the seven-tactic cost audit. For the caching pattern specifically, semantic caching architecture and hit rates and duplicate detection by prompt hash go deeper. And none of these five patterns are visible without cost-per-request attribution in place first.
Frequently Asked Questions
What's a realistic LLM cost reduction target for a production AI team?
Which LLM cost fix delivers the fastest payback?
How long does it take to see LLM cost savings after starting?
Do LLM cost cuts always mean quality tradeoffs?
Why do LLM cost-cutting efforts stall after the first month?
See which of the five patterns is costing you the most.
Preto tags your live traffic by feature and model, ranks your waste sources by dollar impact, and keeps watching after the first fixes ship — so the plateau doesn't turn into a regression six months from now.
Get the Before/After Dashboard TemplateFree forever up to 10K requests. No credit card required.
