What's the difference between exact-match and semantic caching for LLM responses?

Exact-match caching hashes the full prompt and returns a stored response only when a new request hashes identically — safe, simple, but only catches literal duplicates. Semantic caching uses embedding similarity to catch requests that are worded differently but mean the same thing, catching more traffic but requiring a similarity threshold and a quality gate to prevent false-positive hits returning wrong answers.

How do I invalidate a cached LLM response when the underlying data changes?

Attach a version key or data-freshness timestamp to the cache key itself, not just the prompt hash. Any request depending on data that changes (a user's account state, a document that gets edited) should include a version identifier in what gets hashed, so a data change automatically produces a cache miss instead of serving stale content.

A Step by Step Guide to Implementing LLM Response Caching

Q: Should I build my own caching layer or use provider-native caching?

Both, for different things. Provider-native prompt caching (OpenAI, Anthropic) is essentially free to enable and handles the repeated-prefix case automatically. Your own exact-match and semantic caching layer catches duplicate or near-duplicate full requests across users and sessions, which provider-native caching doesn't do. They're complementary, not competing.

Somewhere between 15% and 30% of production LLM traffic is duplicate or near-duplicate requests — the same question, or close enough, asked by a different user or the same user twice. None of that traffic needs a fresh model call. Here's how to build the caching layer that catches it, step by step, without shipping wrong answers along the way.

TL;DR

1. Three caching layers exist and they're complementary: provider-native prompt caching, your own exact-match caching, and semantic caching.
2. Exact-match is safe and simple to ship first — a prompt hash and a Redis lookup. Semantic caching catches more but needs a quality gate.
3. Cache invalidation is the part most implementations get wrong — build a version key into the cache key from day one, not as an afterthought.

The Three Layers, and Where Each One Fits

Layer	Catches	Setup Effort	Risk
Provider-native prompt caching	Repeated prefix across calls (stable system prompts)	Minimal — automatic (OpenAI) or one parameter (Anthropic)	None — provider-managed
Exact-match caching	Identical full requests, any time, any user	Low — hash + key-value store	Very low — only exact duplicates hit
Semantic caching	Similar-but-not-identical requests	Medium — embeddings + vector search	Moderate — false hits without a quality gate

Ship them in that order. Provider-native caching is close to free and should already be on. Exact-match is the highest-value-per-effort addition for most teams. Semantic caching is worth adding once exact-match is running and you can measure how much traffic it's not catching.

Step 1: Exact-Match Caching With Redis

Hash the request, not just the prompt string

Include everything that affects the response in the hash — the prompt, the model name, and any parameters that change output (temperature, top_p). Two identical prompts sent to different models are not cache-equivalent.

import hashlib
import json

def cache_key(prompt, model, params):
    payload = json.dumps({
        "prompt": prompt,
        "model": model,
        "temperature": params.get("temperature", 0),
        "version": params.get("data_version", "v1"),
    }, sort_keys=True)
    return "llmcache:" + hashlib.sha256(payload.encode()).hexdigest()

The version field is the invalidation hook — more on that below. Sorting keys before hashing ensures identical logical requests produce identical hashes regardless of dict ordering.

Check the cache before calling the model

import redis
r = redis.Redis(host="localhost", port=6379, db=0)

def get_cached_response(prompt, model, params):
    key = cache_key(prompt, model, params)
    cached = r.get(key)
    if cached:
        return json.loads(cached), True  # hit
    return None, False  # miss

def store_response(prompt, model, params, response, ttl_seconds=3600):
    key = cache_key(prompt, model, params)
    r.set(key, json.dumps(response), ex=ttl_seconds)

Set a TTL that matches how long a cached answer stays valid for your use case — an hour for anything that might reference changing data, days or weeks for genuinely static content like documentation Q&A.

Want to see your own cache hit potential first?

Preto identifies which of your requests are cacheable automatically, before you write a line of caching code.

See Your Cache Hit Potential — Free

Preto identifies cacheable requests automatically. See yours now.

Step 2: Adding Semantic Caching

Embed the query, search for a near match

Semantic caching catches the traffic exact-match misses — "summarize this" and "give me a summary of this" are different strings but the same request. Generate an embedding for each query and search a vector store for anything above a similarity threshold.

def get_semantic_match(prompt, embedding_client, vector_store, threshold=0.95):
    query_embedding = embedding_client.embed(prompt)
    matches = vector_store.search(query_embedding, top_k=1)
    if matches and matches[0].score >= threshold:
        return matches[0].cached_response, matches[0].score
    return None, None

Embedding cost is negligible relative to the LLM call it might replace — well under 1% of a typical generation call's cost — so the economics favor checking on every request, not sampling.

The quality gate — don't skip this

False positive hits ship wrong answers, which is a worse failure mode than a cache miss. Sample a percentage of hits (1-5% is typical) to a shadow uncached path and diff the outputs. If the diff rate exceeds your tolerance, raise the similarity threshold. Never enable semantic caching on clinical, financial, or legal workloads without this gate running continuously, not just during initial testing.

Step 3: Cache Invalidation — Where Implementations Break

The hardest part of caching isn't storing the response — it's knowing when a cached answer is no longer correct. Two patterns handle most cases:

Version keys for changing data. If a request's correct answer depends on data that changes (a user's account balance, a document that gets edited), include a version or last-modified timestamp for that data in the cache key itself — the version field in the hash function above. When the underlying data changes, bump the version, and the old cache key naturally stops matching. No explicit deletion required.

TTL for time-sensitive content. For anything where "eventually stale" is an acceptable risk (general knowledge Q&A, static reference content), a TTL is simpler than version tracking and good enough. Set it based on how quickly the underlying reality changes, not an arbitrary default.

The mistake to avoid: caching without either mechanism, relying on "it'll probably still be fine" — this is how caches end up serving month-old answers to questions about current state, and it's the single most common reason teams distrust caching enough to rip it out entirely.

What Realistic Hit Rates Look Like

Production hit rates for the combined exact-match plus semantic caching layer typically land at 20-30% blended across mixed traffic, rising to 90%+ on narrow, highly repetitive workloads like FAQ bots or support deflection. Even the low end of that range — 20% — is a meaningful, permanent reduction to a bill that would otherwise scale linearly with request volume.

The caching layer pays for its implementation effort almost immediately for any team running meaningful volume: the embedding and storage costs are a rounding error against the LLM calls it prevents.

For a deeper look at the semantic caching architecture specifically — thresholds, embedding models, and production hit-rate benchmarks — see the full semantic caching deep dive. And if you want to know exactly how much of your traffic is duplicate before building anything, prompt hashing and duplicate detection covers the detection side in detail.

Frequently Asked Questions

What's the difference between exact-match and semantic caching?

Exact-match caching hashes the full prompt and returns a stored response only on an identical hash match. Semantic caching uses embedding similarity to catch requests worded differently but meaning the same thing — catches more traffic, but needs a similarity threshold and quality gate to prevent false hits.

What's a safe similarity threshold for semantic caching?

There's no universal number — it's workload-dependent. Start conservative (0.95+ cosine similarity) and lower it gradually while monitoring a shadow-path diff rate. If the diff rate exceeds your tolerance, raise the threshold back up.

How do I invalidate a cached LLM response when data changes?

Attach a version key or data-freshness timestamp to the cache key itself. Any request depending on changing data should include a version identifier in what gets hashed, so a data change automatically produces a cache miss.

Should I build my own caching layer or use provider-native caching?

Both, for different things. Provider-native prompt caching is essentially free and handles repeated prefixes automatically. Your own exact-match and semantic caching layer catches duplicate or near-duplicate full requests across users and sessions, which provider-native caching doesn't do.

See how much of your traffic is cacheable before you build anything.

Preto identifies exact and near-duplicate requests automatically across your live traffic — so you know the expected hit rate before writing a line of caching code.

See Your Cache Hit Potential — Free

Free forever up to 10K requests. No credit card required.

Gaurav Dagade

Founder of Preto.ai. 11 years engineering leadership. Previously Engineering Manager at Bynry. Building the cost intelligence layer for AI infrastructure.

LinkedIn · Twitter