Every LLM application pays for the same work over and over: the same system prompt processed thousands of times a day, the same tool definitions re-read on every turn, the same question answered for the hundredth user who phrased it the same way. Caching is the discipline of not paying twice — and in LLM systems it is stranger, deeper, and more consequential than the caching you already know.
The most expensive no-op in software
Classical caching lore says there are two hard problems in computer science: cache invalidation and naming things. LLM applications add a third: knowing what a cache hit even means when the function you are caching is a probabilistic model. But before the philosophy, consider the raw waste. A typical agent sends its model a context that begins with several thousand tokens of system prompt, tool definitions, and instructions. Those tokens are identical on every single request. Without caching, the model re-processes them from scratch each time — same tensors, same arithmetic, same result — and you are billed for the privilege. Multiply by every turn of every conversation of every user, and the largest line item on your inference bill is often work whose output was fully known before it started.
This is the defining economic fact of production LLM systems in 2026: most tokens a serious application processes are tokens it has processed before. The teams that internalize this build systems that are three to ten times cheaper and noticeably faster than the teams that do not, using the same models, the same prompts, and the same infrastructure. Caching is not an optimization pass you schedule for later. It is a design axis, and it deserves the same attention you give your prompts.
What a cache means when the function is a model
In a conventional system, a cache memoizes a pure function: same input, same output, store the pair, skip the work. LLM applications complicate every clause of that sentence. The input is a context window that shifts slightly on every call. The output is sampled, so the same input can legitimately produce different outputs. And the work has internal structure — a prefill phase that processes the prompt and a decode phase that generates tokens — which means there are things worth caching inside a single call, not just across calls.
It helps to name the layers precisely. At the bottom is the KV cache, an inference-time structure that lives in accelerator memory and makes token generation possible at all. Above it sits prompt caching, which reuses prefill computation across requests that share a prefix. Above that, response caching returns a previously generated answer without invoking the model, either on exact matches or on semantic similarity. And around the model sit the ordinary caches of any distributed system — tool results, retrieval indexes, embeddings — which become newly interesting because an autonomous agent, not a programmer, decides when to call them. Each layer has its own hit semantics, its own invalidation story, and its own ways of failing. Conflating them is the most common source of confusion in design discussions, so we will take them one at a time.
The KV cache: where inference gets its speed
Start at the bottom, because everything above inherits its shape. A transformer generating token number one thousand needs to attend to the 999 tokens before it. Naively, that means recomputing the attention keys and values for the entire prefix on every new token — quadratic waste of the most expensive compute on earth. The KV cache fixes this: as each token is processed, its key and value tensors for every layer are stored in GPU memory, so generating the next token only requires computing attention against stored tensors rather than recomputing them. This is not an optional optimization; without it, autoregressive generation at modern context lengths would be economically absurd.
The KV cache is why inference has two distinct phases with two distinct costs. Prefill — processing your prompt — is compute-bound and parallel, chewing through thousands of tokens at once to populate the cache. Decode — generating the answer — is memory-bound and serial, producing one token at a time while reading the entire cache at every step. The practical consequence: long prompts cost you at prefill, and the KV cache they build occupies scarce accelerator memory for the life of the request. Serving stacks fight over this memory with paging schemes, quantized caches, and eviction policies, because every gigabyte of cached keys and values is a gigabyte unavailable for serving another user. When a provider prices input tokens differently from output tokens, this two-phase structure is what they are pricing.
Prompt caching: renting the KV cache between requests
The insight behind prompt caching is simple: if two requests share an identical prefix, the KV cache built for the first can be reused by the second, skipping prefill for the shared portion entirely. Providers operationalize this by checkpointing the KV state at designated points and holding it for minutes — sometimes hours — keyed by an exact hash of the prefix. When your next request arrives with the same opening tokens, the model resumes from the checkpoint rather than starting cold. Cached input tokens are typically billed at a tenth or less of the normal rate, and time-to-first-token drops dramatically because the prefill work simply does not happen.
The operative word is exact. Prompt caches match byte-for-byte prefixes. One changed character in your system prompt — a timestamp, a user name, a reordered tool definition — and every token after the change misses the cache. This gives prompt caching a peculiar grammar: it rewards prompts organized like sediment, with the immutable layers at the bottom and the volatile layers at the top. Teams that learn this grammar routinely see cache hit rates above ninety percent on agentic workloads. Teams that do not learn it wonder why their identical-looking requests bill at full price, and the answer is almost always a dynamic value interpolated somewhere near the top of the prompt, silently invalidating everything beneath it.
Cache-friendly context layout
Once you know the rule — longest common prefix wins — context layout becomes an engineering discipline. Put the system prompt first and freeze it. Tool definitions next, in a canonical order that never varies between requests; alphabetize them if you must, but never let serialization order drift with dictionary iteration. Reference material and few-shot examples after that. Only then the conversation history, and only at the very end the things that genuinely change every turn. Anything volatile that feels like it belongs early — the current date, the user's name, feature flags — should be moved late or passed through a tool result rather than baked into the preamble.
The same rule dictates how you handle history. An agent conversation grows by appending: each turn adds messages to the end, which means each request's context is a strict extension of the previous one, which means the entire prior context is a cache hit. This is the happy path, and it is worth protecting. The moment you rewrite history — summarize old turns in place, delete a failed tool call, reorder messages — you break the prefix and pay full prefill on everything after the edit. Sometimes the rewrite is worth it; compaction that halves your context can beat a cache hit on the full one. But it should be a deliberate trade, made with the numbers in front of you, not a side effect of a helper function that tidies messages because tidiness felt virtuous.
Why agents break caches by default
Agentic workloads are simultaneously the best and worst case for prompt caching. Best, because an agent loop is naturally append-only: think, call a tool, append the result, think again. Each iteration reuses everything before it, and a twenty-step agent run can hit the cache on well over ninety percent of its input tokens. Worst, because almost everything a framework does casually will sabotage this. Injecting the current timestamp into the system prompt on every turn. Re-rendering tool schemas from an unordered map. Rotating which examples appear in the preamble for freshness. Truncating history from the front — the single most cache-hostile operation available, since trimming the oldest message shifts every remaining byte and misses the entire cache.
Multi-agent architectures compound the problem. Every subagent with its own distinct system prompt builds its own cache lineage; a supervisor that rewrites context before delegating guarantees cold prefills at every hop. None of this means agents cannot cache well — it means cache behavior must be designed, not hoped for. The test is simple: log the cached-token count on every model call, and trace any turn where it drops unexpectedly. The culprit is nearly always an innocent-looking line of code that touched the prefix. Treat such regressions the way you treat performance regressions — with a metric on a dashboard and an alert — because that is what they are.
The economics, plainly
Run the arithmetic on a concrete agent, because the numbers make the argument better than prose can. Suppose a coding assistant carries a 6,000-token system preamble, accumulates 40,000 tokens of context over a session, and makes thirty model calls as it works. Uncached, that session prefills something like 700,000 input tokens across its calls. With disciplined prefix caching, the vast majority are billed at the cached rate — often a ninety percent discount — and the session's input cost falls by a factor of five or more. At scale this is not marginal. For an application spending fifty thousand dollars a month on inference, cache discipline is frequently the difference between a business that works and one that does not.
Latency follows the same curve. Prefill for tens of thousands of tokens takes seconds; resuming from a warm cache takes a fraction of that. Users experience this as the difference between an assistant that answers instantly and one that visibly winds up before every response. There are second-order effects, too: faster prefill means shorter GPU occupancy per request, which means more throughput from the same hardware, which shows up in your provider's willingness to price cached tokens down. The entire serving economy is organized around the fact that reused prefixes are nearly free while cold ones are not. Your job is to be on the right side of that fact.
Semantic caching: matching meaning, not bytes
Prompt caching accelerates the model; semantic caching tries to skip it entirely. The idea: embed each incoming query, compare it against a vector store of previously answered queries, and if a stored one is similar enough, return its answer without invoking the model at all. "What's your refund policy?" and "how do refunds work" land within a whisker of each other in embedding space; answering the second from the first's cached response costs an embedding lookup — fractions of a millisecond and fractions of a cent — instead of a full generation. For high-traffic applications with repetitive query distributions, support bots being the canonical case, semantic caches can absorb thirty to sixty percent of traffic before a model is ever touched.
The machinery is straightforward: an embedding model, a vector index, a similarity threshold, and a TTL. The judgment lives entirely in the threshold. Set it loose and you will serve wrong answers with total confidence — the user who asked about "canceling my subscription" gets the answer about "canceling my order," and the cache has manufactured a hallucination out of two correct answers. Set it tight and the hit rate collapses to exact duplicates, at which point a hash map would have done the job cheaper. Production systems converge on conservative thresholds, per-category tuning, and a verification pass for anything borderline — and they log every hit with its similarity score, because the failure mode is silent and only the logs will confess it.
When semantic caching goes wrong
The failure modes of semantic caching deserve their own section because they are subtle enough to survive code review. The classic is polarity blindness: "how do I enable notifications" and "how do I disable notifications" are nearly identical in embedding space, differing by one high-stakes word. Negations, antonyms, and entity swaps — "reset my password" versus "reset my email" — all produce embeddings far closer than their answers are. A cache that cannot distinguish them will cheerfully cross-wire questions and answers, and because each individual answer looks fluent and plausible, users may not notice they were answered wrongly. That is worse than an error message; it is misinformation with good manners.
Context-dependence is the second trap. The same question means different things from different users — "what's my plan's limit?" depends on who is asking — so any query whose answer depends on identity, time, or session state must either carry that context in its cache key or be excluded from caching entirely. The third trap is staleness with confidence: the cached answer was right in March and is wrong after Tuesday's pricing change, and unlike a stale web page, a stale LLM answer arrives in fresh, confident prose with no timestamp attached. The mitigations are unglamorous — scope keys by user segment, keep TTLs short for volatile domains, purge aggressively on content updates, and route anything personalized straight past the cache — but they are the difference between a cost optimization and an incident report.
Exact-match response caching
Before reaching for embeddings, do not skip the boring version: hash the normalized request — model, full prompt, temperature, tool schemas — and return the stored response on an exact match. It sounds too naive to matter until you measure real traffic. Deployments discover that surprising fractions of their calls are literal duplicates: the same classification run twice by a retry, the same summarization requested by two services that do not know about each other, the same onboarding question asked by every new user in identical words because the UI put the words in their mouth. Exact-match caching has none of semantic caching's risks — the input is identical, so serving the stored output is safe for any deterministic-enough use case — and it costs a hash map, not a vector database.
The nuance is sampling. If you run at nonzero temperature, identical inputs are supposed to produce varied outputs, and a cache freezes that variety into repetition. For classification, extraction, and structured tasks run at temperature zero, this is irrelevant; cache freely. For creative and conversational surfaces, exact-match caching changes product behavior, and users do notice when the "regenerate" button returns the same answer twice. The rule of thumb: cache where determinism is a feature, bypass where variety is the feature, and make the decision explicitly per call site rather than globally — a single flag on your inference client is enough to encode it.
Caching tool results
An agent's model calls get the attention, but its tool calls are often where the real latency and money hide. Agents are enthusiastic re-askers: the same file read four times in one session, the same API queried on every turn because the result never made it into context durably, the same expensive database aggregation rerun because the model forgot it already knew the answer. A tool-result cache — keyed on tool name and canonicalized arguments, with per-tool TTLs — converts this repetition from a cost into a no-op. Reads of source files can cache until the file's mtime changes; a weather lookup is good for an hour; an exchange-rate query for a minute; a compliance database, perhaps not at all.
Tool caching has an agentic twist that classical systems lack: the cache can be visible to the model. Injecting a cached result with a note that it was retrieved earlier lets the model decide whether freshness matters enough to re-fetch — a judgment call that a TTL cannot make but a model often can. The inverse pattern also earns its keep: idempotency keys on write tools. When an agent retries a failed-looking payment or resend, the idempotency key ensures the side effect happens once, no matter how many times the loop stutters. Strictly speaking that is deduplication rather than caching, but it is the same instinct — recognize repeated work and refuse to do it twice — applied to actions instead of answers, where the stakes are considerably higher.
Caching the retrieval layer
RAG pipelines are cache stacks wearing a trench coat. Embedding computation is a pure function of text and model version — cache it forever, keyed on a content hash, and re-embedding your corpus becomes an incremental operation instead of a quarterly cost spike. Search results for common queries follow a steep power law; caching the top of that distribution for even a few minutes absorbs a remarkable share of retrieval traffic. Reranker scores for repeated query-document pairs, chunk expansions, document parses — every stage upstream of the model is deterministic or nearly so, and every one of them is cheaper to store than to recompute.
The subtlety is index freshness. A cached search result is a claim about what the index contained at query time, and indexes change as documents arrive. The clean pattern is event-driven invalidation: when a document is updated, purge or version the cache entries whose results included it — which requires recording, per cached query, which documents contributed. Where that bookkeeping is too heavy, short TTLs are the honest fallback. And note the elegant interaction with prompt caching: retrieved chunks injected into the context are part of the prefix, so stable retrieval — same query, same chunks, same order — preserves downstream prompt-cache hits, while jittery retrieval that reorders chunks on every call quietly destroys them. Determinism in the retrieval layer pays twice.
Invalidation: the second hard thing
Phil Karlton's joke survives because invalidation really is where caches go to fail, and LLM systems raise the difficulty. A conventional cache invalidates when data changes. An LLM response depends on data, but also on the prompt template, the model version, the tool schemas, the retrieval index, and the sampling parameters — change any one and every cached response derived from it is suspect. The workable discipline is to make the dependencies explicit: build cache keys that include a version stamp for each ingredient — prompt hash, model ID, index generation, schema fingerprint — so that deploying a new prompt or model naturally strands the old entries rather than serving them. Stranded entries cost storage; stale entries cost trust.
Then decide, per cache, which staleness you can tolerate. KV and prompt caches need no semantic invalidation at all — they cache computation over exact bytes, so a changed prompt simply misses; the provider's TTL handles the rest. Semantic and response caches need real invalidation tied to content changes, because their entries assert facts. Tool caches need per-tool policies, because "how stale is acceptable" is a property of the tool's domain, not of the caching layer. Write the policy down next to each cache. The failure pattern in postmortems is rarely a wrong policy — it is a cache nobody remembered existed, doing exactly what it was configured to do long after the configuration stopped being true.
Freshness, TTLs, and the lifetime of a fact
Time-to-live is the bluntest invalidation instrument, and for many LLM caches the only practical one, so it is worth choosing with intent. The question a TTL answers is not "how long can I store this?" but "what is the half-life of this fact?" A company's founding date has a half-life of forever. Its pricing page, months — until the week it changes. Stock quotes, seconds. The same application often serves all three from the same cache with the same TTL, which means the TTL is wrong for at least two of them. Segmenting caches by volatility class — evergreen, slow-changing, live — costs little and removes a whole genre of staleness bugs.
Provider-side prompt caches have their own clock. Entries typically expire after minutes of disuse, refreshed on every hit, and some providers offer extended TTLs at a premium. This creates a mild scheduling game: an agent that will resume in ninety seconds benefits from arriving before the cache cools; a nightly batch job never will, and should be optimized for throughput instead. Some teams ping long-lived sessions to keep caches warm — worthwhile only when the resume is near-certain and the preamble is enormous; otherwise it is paying rent on an apartment you may never revisit. As with all freshness engineering, the goal is not maximal caching. It is knowing, for every cached byte, who will be wrong if it outlives its truth — and by how much.
Privacy: the cache as a leak
Any cache shared across users is a potential channel between them, and LLM caches carry unusually sensitive cargo. The obvious hazard is response leakage: a semantic cache that stores one user's answer — containing their account details, their document excerpts, their medical question — and serves it to a different user whose query landed nearby in embedding space. The fix is structural, not procedural: partition cache keyspaces by tenant, by user where the data is personal, and never let personalized generations into a shared pool. Shared caching is for shared truths — public documentation, common product questions — and the classifier that decides "is this answer personal?" should fail closed.
Prompt caches have a subtler cousin: timing. Whether a prefix was warm or cold is observable through latency, which means an attacker who can measure time-to-first-token can, in principle, probe whether someone else recently submitted a given prefix. Providers mitigate by scoping caches per organization, which you should verify rather than assume, and the residual risk within an organization is usually acceptable — but "usually" deserves a moment's thought in multi-tenant products built atop a single org's API keys, where per-tenant salts in the prompt can deliberately fragment the cache at a known cost. Privacy reviews that scrutinize logs and analytics routinely walk straight past the caching layer. Walk back.
Determinism and the illusion of reuse
A recurring confusion deserves a clean answer: does caching make models deterministic? No — and the layers differ in what they promise. KV and prompt caches are mathematically transparent: resuming from cached prefill produces the same distributions as recomputing, so they change cost and latency, never behavior. (In practice, floating-point nondeterminism in batched inference means even identical uncached calls can drift; the cache neither causes nor cures this.) Response caches are the opposite: they impose determinism, freezing one sampled draw and replaying it. That can be a gift — support answers become consistent across users, screenshots match documentation — or a liability, when a mediocre generation gets fossilized and served ten thousand times.
The fossilization risk earns a countermeasure: cache with review. High-traffic cached answers are, by construction, your most-served content, which makes them the highest-leverage place for human curation. Some teams promote hot cache entries into an edited knowledge layer — a person polishes the answer once and the cache serves the improved version thereafter — quietly converting the cache from a cost optimization into a content pipeline. Others periodically resample: regenerate a small fraction of hits, compare against the stored answer, and refresh when the new model or new prompt does better. Both patterns treat the cache as what it actually is: a publication surface, with an editor's responsibilities attached.
Measuring hit rates that matter
The vanity metric of caching is the hit rate; the real metrics are dollars and milliseconds saved, weighted by correctness. A ninety percent hit rate on a cache of cheap calls saves less than a forty percent hit rate on your most expensive prompt family, and a hit that serves a wrong answer has negative value that no ratio captures. Instrument accordingly. For prompt caches, track cached versus uncached input tokens per request, sliced by agent and by turn number — the shape tells you instantly when a code change breaks prefix stability, because turn-over-turn cached fractions should climb toward the high nineties and a regression shows up as a cliff. For response and semantic caches, track hit rate alongside a sampled correctness audit: pull a fraction of hits, regenerate fresh, and diff.
Attribute savings honestly. Cached tokens are discounted, not free; semantic cache lookups cost embeddings and index queries; storage and invalidation machinery cost engineering time that could have gone elsewhere. The dashboard worth building shows, per cache layer: traffic absorbed, marginal cost per hit versus per miss, estimated monthly savings, and staleness incidents. That last column keeps everyone honest. A caching layer whose savings column is large and whose incident column is unknown is not a success story — it is a story that has not reached its ending yet, and the ending has a way of arriving during a pricing change or a product launch.
Warming, prefetching, and speculative work
Caches reward those who arrive before the user does. Prompt caches can be warmed deliberately: fire a cheap one-token request against tomorrow's system preamble when you deploy it, and the first real user of the day resumes from a warm checkpoint instead of eating the cold prefill. Session handoffs benefit from the same trick — when a user opens a conversation, warming the context before they finish typing their first message converts prefill time into time the user never sees. Semantic caches can be pre-seeded from your FAQ, your documentation, and last month's query logs: generate the answers once, review them, and launch with a cache that starts at a forty percent hit rate instead of zero.
Speculation goes a step further: doing work that might be needed before it is requested. An agent waiting on a slow tool call can speculatively begin drafting its response under each of the two most likely results, keeping whichever branch reality confirms — spending compute to buy latency. A retrieval layer can prefetch the documents a session's trajectory suggests will be asked about next. These techniques sit at the aggressive end of the caching spectrum and their arithmetic must be checked — speculative work that is wrong ninety percent of the time is usually a subsidy to your provider — but for latency-critical surfaces like voice, where hundreds of milliseconds decide whether the product feels alive, speculation is often the only remaining lever once the caches are warm.
Caching in multi-agent systems
Multi-agent architectures multiply both the opportunity and the difficulty. The opportunity: subagents with stable, specialized system prompts are ideal cache citizens — a research agent whose preamble never varies hits its prefix cache on every invocation across every user, and a fleet of identical workers shares warm state beautifully. The difficulty: every context handoff is a potential cache break. A supervisor that summarizes findings before passing them along creates a fresh prefix for the recipient; an orchestrator that rewrites task descriptions per worker guarantees cold starts. Sometimes the rewrite carries real value — smaller contexts, cleaner separation — but teams should notice they are trading cache efficiency for it, because the bill will.
Design patterns help. Give each agent role a frozen preamble and pass all variability through the final user message, keeping the cacheable prefix maximal. Prefer appending delegation records to rewriting them. Where a shared blackboard or memory store carries inter-agent state, cache its rendered form so that agents reading the same state render the same bytes — identical rendering is what turns shared state into shared cache hits. And measure per role: a fleet dashboard showing cached-token fraction by agent type will reveal, with embarrassing clarity, which orchestration choices are burning money. In one common finding, the supervisor — the smallest fraction of calls — accounts for most uncached tokens, because its context is assembled fresh each time from pieces no two of which arrive in the same order.
Streaming, latency budgets, and where the time goes
Caching decisions ultimately serve a latency budget, so it pays to know where the time actually goes. A user-facing turn spends time in queue, in prefill, in decode, and in your application code between model and tool calls. Prompt caching attacks prefill, which dominates the start of long-context requests — this is why warm agents feel snappy and cold ones feel like they are inhaling before speaking. Response and semantic caches attack everything at once, collapsing the whole pipeline into a lookup for the requests they absorb. Decode time, meanwhile, yields to none of them; only shorter outputs, faster models, or speculative decoding touch it. If your product's pain is a slow first token, invest in prefix discipline. If it is slow full answers, caching is not your bottleneck.
Streaming reshapes perception rather than arithmetic. A cached-prefix request that begins streaming in three hundred milliseconds feels instant even if the full answer takes ten seconds; an uncached one that sits silent for four seconds feels broken even if it finishes sooner overall. This is why cache regressions surface first as user complaints about "slowness" that percentile dashboards of total latency fail to show — the total barely moved, but time-to-first-token doubled. Put TTFT on the dashboard next to the cached-token fraction. The two lines move together, and together they are the truest picture of what your users are feeling.
Provider caches versus building your own
A reasonable question at this point: how much of this do you build? The answer has converged in practice. KV caching is the provider's problem — you never touch it directly. Prompt caching is the provider's mechanism but your responsibility: they store the checkpoints, you supply the prefix discipline that makes checkpoints hittable, and a few providers ask you to mark cache breakpoints explicitly while others detect prefixes automatically. Everything above that line — response caches, semantic caches, tool and retrieval caches — is yours, built from ordinary infrastructure: a key-value store, a vector index, a hash function, and policy.
Resist the temptation to buy a "semantic caching platform" before you have measured your traffic. The right first moves are embarrassingly small: turn on provider prompt caching and fix your prefix order; add an exact-match cache in front of your top three prompt families; cache your embeddings. Each is a day of work and each pays immediately. The vector-similarity layer is a second-quarter project, justified only when logs show a large mass of near-duplicate queries that exact matching cannot reach. Caching architecture is like any architecture: the versions that survive are the ones grown from observed load, not the ones installed in anticipation of it.
A reference architecture
Assembled, the stack looks like this. Requests enter through a gateway that computes a normalized request hash and checks the exact-match cache first — microseconds, zero risk. Misses proceed to the semantic layer, where queries eligible for shared answers are embedded and checked against a curated, tenant-scoped index with a conservative threshold; hits are logged with similarity scores and sampled for audit. Requests that reach the model do so through a context assembler that enforces prefix discipline — frozen preamble, canonical tool order, append-only history — so that provider prompt caching does its work invisibly. Around the loop, tool calls pass through a result cache with per-tool TTLs and idempotency keys on writes; the retrieval layer caches embeddings permanently and search results briefly, with event-driven purges on document updates.
Two cross-cutting services complete it. A version registry stamps every cache key with the current prompt hash, model ID, schema fingerprint, and index generation, so deployments strand stale entries automatically and rollbacks resurrect the matching ones. And an observability plane records, per layer, the traffic absorbed, the tokens and dollars saved, TTFT, and the audit results — with alerts on cached-fraction cliffs, which almost always mean someone touched the prefix. Nothing in this architecture is exotic. Its value is in the joints: each layer catches what the previous one could not, and each failure mode has a metric watching it.
Common failure modes
The recurring accidents are worth naming so you can recognize them early. The timestamp in the preamble: a helpful engineer adds the current date to the system prompt for the model's benefit and silently invalidates the prompt cache on every request; the fix is moving volatile facts to the end of context or into a tool result. The unordered serialization: tool schemas rendered from a map iterate in different orders across processes, so identical logical requests never share bytes. The front-truncation: history trimmed from the oldest end to fit a context limit, shifting every byte and missing the entire cache — compaction at checkpointed boundaries, preserving the prefix, costs the same tokens and keeps the discount. The over-eager semantic hit: a loose threshold cross-wires two questions that embeddings consider siblings and users consider opposites.
Add the organizational failures: the cache nobody owns, discovered during an incident by an engineer who did not know it existed; the savings estimate presented without a staleness audit; the shared cache that quietly outlived the privacy review that approved it back when all its content was public. And the subtle one — the cache that works so well it hides a regression: response quality degrades after a model swap, but cached answers from the previous model keep serving, so the degradation surfaces weeks later, mysteriously, as the cache ages out. Every one of these has appeared in a real postmortem. The common thread is invisibility, and the common cure is the dashboard described above, plus a name in the on-call rotation next to every cache that can lie.
When not to cache
Caching has genuine contraindications, and knowing them saves you from optimizing your way into trouble. Do not response-cache anything personalized, legally sensitive, or safety-critical — medical guidance, financial advice tailored to a situation, content moderation verdicts — where serving user A's answer to user B is not a bug but a breach. Do not cache where variety is the product: creative writing surfaces, brainstorming tools, anything with a regenerate button that users expect to do something. Be wary of caching in adversarial settings — fraud analysis, security triage — where an attacker who learns that answers replay can probe the cache to map your defenses, and where yesterday's verdict on similar-looking input is precisely what they hope you will serve.
And do not let caching calcify iteration. Early in a product's life, prompts change daily, models swap weekly, and the traffic distribution is a rumor; heavyweight caching machinery built then will invalidate constantly, mask experiments, and slow the very iteration that finds product-market fit. Prompt caching costs nothing to keep on and should be. But the layered response-caching apparatus belongs to the scaling phase, after behavior stabilizes, when the same questions arrive at volume and the answers have settled. Premature caching is premature optimization's more expensive cousin, because it does not merely waste effort — it serves stale versions of a product you are still trying to change.
Getting started without overbuilding
The pragmatic on-ramp fits in a sprint. First, turn on provider prompt caching and audit your prefix: freeze the preamble, canonicalize tool order, move volatility to the end, and confirm on the provider's usage dashboard that cached input tokens climb past eighty percent of total on multi-turn workloads. Second, log every model call with its cache metrics, and put cached-fraction and TTFT on the dashboard you actually look at. Third, add exact-match response caching to your highest-volume deterministic call sites — classification, extraction, routing — where it is pure savings with no behavioral change. Fourth, hash-cache your embeddings. Stop there and measure for a month.
What the month of data tells you determines the second phase. If logs show heavy near-duplicate query traffic, build the semantic layer, starting with a curated seed set and conservative thresholds. If tool latency dominates traces, add the tool-result cache with per-tool TTLs. If neither shows up, you are done, and you have avoided building the infrastructure your traffic never asked for. Throughout, hold to the two disciplines that make everything else safe: version-stamped keys, so deployments invalidate automatically, and sampled audits of every cache that serves stored answers, so staleness is a number you track rather than a surprise you receive. Caching rewards exactly this kind of unglamorous rigor, and punishes every shortcut eventually.
The bottom line
LLM applications re-buy the same computation at a scale no previous software category has managed, and caching is the discipline that stops the bleeding. It operates at every altitude: KV caches make generation feasible, prompt caches make long contexts affordable, response and semantic caches make repeated questions nearly free, and the humble caches around tools and retrieval quietly carry the rest. The through-line is the same at every layer — recognize work you have already done, and decline to do it again — but the implementations differ enough that each layer needs its own keys, its own invalidation story, and its own place on the dashboard.
The craft is in the restraint. Byte-exact prefix discipline costs nothing and belongs in every system from day one. Stored-answer caches are powerful and dangerous in proportion to each other, and earn their place only with version stamps, tenant scoping, and audits attached. Staleness, leakage, and fossilized mediocrity are not edge cases; they are the default outcomes of caching applied carelessly to systems that generate confident prose. Build the layers your traffic justifies, watch the two numbers that matter — cached fraction and time-to-first-token — and treat every cache that can speak to a user as the publication surface it is. The reward for getting this right is not subtle: the same product, several times cheaper and visibly faster, built from work you were already doing and simply refused to pay for twice.