Prompts are the part everyone talks about. Context is the part that decides whether your application actually works. This is a deep, practical tour of deciding what enters the model context window, in what form, and why.
From prompting to context engineering
Prompt engineering asks how to phrase an instruction. Context engineering asks a larger and harder question: of everything the model could possibly know at this moment, what should actually be inside the context window, and in what form? For any application beyond a toy, this is where most of the quality, cost, and latency live. The prompt is a few hundred tokens you write once and reuse. The context is thousands of tokens assembled fresh on every single request, drawn from documents, databases, prior conversation, tool outputs, and user input. Getting it right is a systems and engineering problem, not a writing problem, and it is the part of the stack that most reliably separates demos that impress from products that hold up.
The shift in mindset is the whole point. Stop thinking of the model as something you talk to and start thinking of it as something you assemble inputs for. On every request, your system constructs a context: it retrieves relevant material, ranks and trims it, summarizes what is too long, formats everything into a legible structure, and hands the result to the model. That construction pipeline is the product. When people say the model is not good enough, they are very often describing a context problem in disguise: the right information was not present, or the wrong information was, or the right information was buried where the model could not use it.
It is worth saying plainly that the model is the commodity in this picture and the context pipeline is the differentiator. Everyone has access to roughly the same frontier models. What they do not have is your data, your retrieval logic, your chunking choices, your ranking, and your memory design. That is where the defensible quality of an application comes from, and it is why context engineering deserves at least as much of your attention as the prompt that sits on top of it.
A working model of the context window
The context window is the fixed-size span of tokens the model can attend to on a given call. Everything the model uses to produce its answer, including your instructions, the conversation so far, retrieved documents, and the user question, has to fit inside it, and the model has no access to anything outside it. Modern windows are large, sometimes hundreds of thousands of tokens, and that abundance is a trap. The instinct it encourages, to simply pour everything in and let the model sort it out, is exactly the instinct that produces slow, expensive, and surprisingly inaccurate systems.
Treat the window as a budget you spend deliberately rather than a bucket you fill to the brim. Every token you place in it competes for the model attention and costs money and time on every call. The discipline of context engineering is spending that budget on the most relevant information, in the most useful position, in the most compact faithful form. Almost everything that follows is a technique for doing exactly that, and almost every context-related failure is a failure to spend the budget well.
Context engineering is a retrieval and assembly pipeline: the right tokens, ranked and trimmed, placed where the model can actually use them.
The context window is a budget, not a bucket
There are three concrete reasons restraint beats abundance, and each one has bitten teams that assumed a bigger window solved their problem. The first is cost and latency: both scale with the number of tokens, so every irrelevant document you include is paid for on every call, in money and in the user waiting. At scale this is not a rounding error; it is often the dominant line in the bill and the main source of sluggishness, and it grows linearly with every marginal document you decided was safer to include than to leave out.
The second reason is that attention degrades with length. Models reliably lose track of information buried in the middle of a long context, a pattern documented well enough to have a name, the lost-in-the-middle effect. Information at the very start and the very end of the context is used far more reliably than information in the middle. So padding the window with marginal material does not just cost money; it can actively push the relevant content into the dead zone where the model half-ignores it. More context can mean worse recall of the part that mattered, which is the opposite of what the person stuffing the window intended.
The third reason is that distractors cause errors. Plausible but irrelevant information does not sit there harmlessly; it pulls the answer off course. A retrieved passage that is topically similar but factually irrelevant to the question can lead the model to a confidently wrong conclusion. Once you have enough relevant context to answer, additional marginal context tends to hurt rather than help. Precision beats recall past a certain point, and recognizing where that point is for your task is much of the craft. The hard discipline is being willing to leave out material that is related but not load-bearing.
Retrieval: getting the right tokens in
Retrieval-augmented generation, or RAG, is the workhorse of context engineering, and for good reason. Instead of relying on what the model memorized during training, you fetch relevant information at query time from a source you control, and you place it in the context so the model answers from current, specific, authoritative material. This is how you get a model to answer questions about your private documents, your latest data, or facts that postdate its training, and how you make its answers attributable to sources rather than to an opaque blend of training data.
The naive version is easy to stand up and easy to outgrow. You split your documents into chunks, embed each chunk into a vector, store the vectors, embed the incoming query, retrieve the handful of chunks whose vectors are nearest, and paste them into the prompt. This works well enough to build a convincing demo in an afternoon, which is precisely why so many teams ship it and then hit a wall when real queries expose its weaknesses. The improvements that matter are not exotic; they are a series of targeted fixes to the places the naive pipeline leaks quality, and the rest of this section walks through them in the order they usually pay off.
Chunking: the unglamorous foundation
How you split documents into chunks quietly determines how well retrieval can ever work, because the chunk is the unit you retrieve. Splitting purely by character count, the default in many quick tutorials, routinely cuts sentences in half, separates a claim from its qualification, and severs a table from its header, destroying the meaning that made the passage worth retrieving. The result is chunks that match a query on surface terms but no longer contain a coherent, self-sufficient answer, and no amount of clever ranking downstream can repair information that chunking already threw away.
Better chunking respects the structure of the document. Split on semantic and structural boundaries such as sections, paragraphs, and list items, and size chunks so that each one can stand alone as a meaningful unit a reader could understand without its neighbors. A common and effective refinement is overlap: let consecutive chunks share a little text at their edges so that an idea spanning a boundary survives in at least one chunk intact. For structured or hierarchical documents, attaching a short breadcrumb of context to each chunk, such as the document title and the section heading it came from, helps both retrieval and the model understanding of where the passage sits. Chunking is tedious and it is where a great deal of RAG quality is silently won or lost.
There is no universal chunk size. The right granularity depends on your documents and your queries: dense reference material rewards smaller, precise chunks, while narrative or argumentative text often needs larger ones to keep an idea whole. Treat chunk size and overlap as parameters you tune against a real evaluation set, not constants you copy from a tutorial, because the wrong choice silently caps the ceiling of everything built on top of it.
Embeddings and vector search
Vector search is the engine that lets you retrieve by meaning rather than by exact words. An embedding model maps text into a high-dimensional vector such that passages with similar meaning land near each other, so a query about canine nutrition can retrieve a passage about feeding dogs even with no shared keywords. The choice of embedding model matters; they differ in how well they capture nuance, how they handle long inputs, and which domains they were trained on. A model that excels on general web text may underperform on legal, medical, or code retrieval, where a domain-tuned embedding can be markedly better.
Vector search has a characteristic blind spot, though, and ignoring it is a common cause of frustrating misses. Because it matches on meaning, it can fail on exactly the cases where the precise term is the point: a product code, an error string, a person name, a specific identifier. Two strings that are semantically unimportant but lexically exact, like a part number, may not sit close in embedding space even though an exact-match search would find them instantly. This is the gap that the next technique exists to close, and it is the single most common reason a vector-only system feels mysteriously unreliable on precisely the queries users care most about.
Vector search retrieves by meaning, which is powerful but blind to exact terms like codes and names. Hybrid search closes that gap.
Hybrid search and reranking
Hybrid search combines dense vector retrieval with traditional keyword search, often a method in the BM25 family, and fuses the results. The vector side catches semantic matches that share no words; the keyword side catches the exact terms that embeddings miss. For most real corpora, which mix conceptual questions with specific identifiers, the combination retrieves noticeably more of what matters than either method alone. It is one of the highest-return upgrades you can make to a naive pipeline, and it is well supported by mature search infrastructure, so the cost of adopting it is mostly configuration rather than research.
Reranking is the second high-return upgrade, and the two compose beautifully. The idea is to retrieve a wider candidate set cheaply, say the top twenty or fifty results from hybrid search, and then pass those candidates through a more expensive but more accurate model, a cross-encoder reranker, that scores how well each candidate actually answers the query. You then keep only the few best for the context. Because the reranker looks at the query and the candidate together rather than comparing precomputed vectors, it judges relevance far more precisely. In practice, adding a reranking step often moves end-to-end quality more than any embedding-model upgrade, because it directly optimizes the thing you care about, which is whether the chunks you finally show the model are the ones that answer the question.
The mental model worth carrying is a funnel. Cast a wide, cheap net first to maximize the chance the right material is somewhere in the candidate pool, then apply progressively more expensive and more accurate filters to narrow that pool down to the few passages you can afford to put in front of the model. Recall first, precision last. Each stage has a different cost and accuracy profile, and arranging them as a funnel lets you buy accuracy only where it counts.
Query transformation
The user query is not always the best thing to search with, and treating it as immutable leaves quality on the table. Users ask short, ambiguous, or conversational questions that do not map cleanly onto the language of your documents. Query transformation bridges that gap. You can rewrite the query into a cleaner search form, expand it into several sub-queries that cover different facets of the question and retrieve for each, or generate a hypothetical answer and search with that, on the theory that a plausible answer is lexically closer to the real passages than the terse question was.
In multi-turn settings, you almost always need to rewrite a follow-up question into a standalone one first, because a phrase like that follows from a prior turn is meaningless to a retriever that sees only the current message. The retriever has no memory of the conversation; it sees one query, so that query must carry all the meaning on its own. These transformations cost an extra model call but frequently rescue queries that would otherwise retrieve nothing useful, and they are cheap insurance against the large class of failures where retrieval silently returns the wrong thing because the query, not the index, was the problem.
Advanced retrieval patterns
Beyond the core funnel, several patterns earn their keep on harder problems. Metadata filtering narrows the search space before ranking by constraining on structured attributes such as date, author, document type, or access level, which both improves relevance and enforces permissions. Parent-document retrieval embeds and searches over small, precise chunks but then returns the larger surrounding passage to the model, combining the retrieval accuracy of small chunks with the contextual completeness of large ones. Multi-hop retrieval handles questions that cannot be answered by any single passage by retrieving, reasoning about what is still missing, and retrieving again, which is essential for questions that require chaining facts across documents.
Graph-based approaches go further by building an explicit structure of entities and relationships extracted from your corpus, so the system can traverse connections rather than relying on similarity alone, which helps on questions about how things relate across many documents. None of these is mandatory, and each adds complexity and cost. The right move is to start with the simple funnel, measure where it fails, and reach for an advanced pattern only when the failures you observe are the specific kind that pattern addresses. Complexity you add speculatively is complexity you will debug later for no measured benefit.
Agentic and iterative retrieval
The patterns above mostly assume retrieval happens once, up front. A more powerful and more expensive approach lets the system retrieve iteratively, treating search as a tool the model can call as many times as it needs. The model reads what came back, decides whether it has enough to answer, and if not, issues a refined query, follows a lead, or searches a different source. This agentic retrieval handles open-ended research questions that no single query could satisfy, and it degrades gracefully when the first attempt misses, because the model can simply try again with what it learned. The trade-off is latency, cost, and the need for guardrails so the loop terminates, which connects context engineering directly to the discipline of building reliable agent loops. Reserve it for questions whose difficulty justifies the extra calls.
Compression and summarization
When the relevant material is genuinely larger than your token budget, the answer is not to truncate arbitrarily but to compress with intent. Summarize retrieved documents down to the portion that bears on the question before they enter the final prompt. Extract only the passages that matter rather than passing whole documents. For long running conversations or large corpora, maintain summaries that capture the essential state in a fraction of the tokens. The objective throughout is information density: maximize the relevant signal per token spent, so that the budget you do have is working as hard as possible.
Compression has a cost beyond the extra model calls, namely the risk of summarizing away a detail that turns out to matter, so it pays to compress in a way that preserves specifics like names, numbers, dates, and decisions rather than smoothing them into generic prose. A summary that reads well but has quietly dropped the one figure the user will ask about is worse than no summary, because it looks trustworthy while being incomplete. When in doubt, compress toward extraction of concrete facts rather than abstraction into themes.
Conversational and long-term memory
In any multi-turn application the conversation history is itself context that grows without bound, and managing it is a core context-engineering task rather than an afterthought. Naively appending every turn forever eventually blows the budget and, well before that, degrades attention as the transcript crowds out everything else. The standard solution is tiered memory. Keep the most recent turns verbatim, because immediate context matters most for coherence. Replace older turns with a running summary that preserves what was decided and what the user wants without the full back-and-forth. And lift durable facts, such as the user name, stated preferences, and commitments made, into a structured store that you retrieve from explicitly rather than re-reading the entire history each turn.
That structured store is the seed of long-term memory, the capability that lets an assistant feel continuous across sessions rather than amnesiac. The engineering questions are what to remember, how to retrieve it relevantly when it applies, and how to update or forget it when it goes stale. Done well, long-term memory means the system brings the right past detail into context at the right moment; done carelessly, it means stale or irrelevant facts leak in and degrade answers, which is its own failure mode worth guarding against. Memory is not a database you append to forever; it is a curated, retrievable, prunable asset that needs the same disciplined design as the rest of the pipeline.
Position and formatting matter
Where you place information inside the window changes how well the model uses it, which is a direct consequence of the lost-in-the-middle effect. Put the most important material near the top or, often better, near the very end, closest to where the model begins generating, and avoid stranding critical content in the middle of a long block. Beyond raw position, label everything. Mark clearly what is the user question, what is retrieved evidence, what is prior conversation, and what are the standing rules. Explicit structure lets the model reason about provenance, which is exactly what you want when some context is authoritative instruction and some is merely retrieved text that might be wrong or even adversarial. A well-formatted context is easier for the model to use correctly and easier for you to debug when it does not.
Spend the token budget deliberately: most relevant material, compactly summarized, labeled by provenance, and placed where attention is strongest.
Grounding and citations
One of the strongest reasons to engineer context carefully is that it is your most effective lever against hallucination. When you instruct the model to answer only from the provided material and to cite which passage supports each claim, you dramatically reduce its room to invent, because the answer is now a function of text you supplied rather than of half-remembered training data. Citations are not just a trust feature for the user; they are a forcing function on the model, tying each statement back to a source and making unsupported claims conspicuous.
Pair grounding with an explicit instruction that it is acceptable, even expected, to say the answer is not contained in the provided material, so the model has an honest exit rather than a pressure to fabricate something that fits. Grounding does not make a system infallible, but it converts a freewheeling generator into something much closer to a careful reader of your documents, and it gives you and your users a way to check the work, which is often more valuable than the answer itself.
Context is an attack surface
Everything that enters the context window can influence the model behavior, and that includes content you did not author: retrieved documents, user messages, and the outputs of tools the model called. This makes the context an attack surface. A poisoned document in your knowledge base, or a malicious tool result, can carry injected instructions that attempt to override your system rules, exfiltrate data, or trigger unwanted actions. Because the model has no innate way to tell your instructions from instructions embedded in the data, the defense has to be engineered around it.
Maintain a hard boundary between trusted instructions and untrusted content, delimit external material clearly and label it as data, and never let retrieved text silently take precedence over your standing rules. Where the model can take consequential actions based on context, gate those actions behind validation or human review. Treat untrusted context with the same suspicion you would treat untrusted input anywhere else in a system, because that is exactly what it is, and the convenience of piping it straight into a privileged position is exactly how injection attacks succeed.
Evaluating the context pipeline
You cannot improve a pipeline you do not measure, and the great advantage of context engineering is that much of it is measurable independently of the final answer. Evaluate retrieval on its own terms: for a set of queries with known relevant documents, did the right documents come back, and how highly were they ranked? Metrics that capture whether the relevant material was retrieved at all, and whether it was ranked near the top, tell you where the pipeline leaks.
This component-level evaluation is invaluable because an end-to-end failure does not tell you where to look, whereas knowing that retrieval missed the relevant chunk points you straight at chunking, embeddings, or ranking rather than at the prompt or the model. Build a small evaluation set of real queries with known good sources, run it whenever you change the pipeline, and you turn context engineering from guesswork into a measurable, improvable discipline. The teams that move fastest are not the ones with the cleverest retrieval; they are the ones who can tell within minutes whether a change helped or hurt.
Cost and latency engineering
Because context dominates token usage in most applications, it is also where you win or lose on cost and speed. Several levers help. Prompt caching, where supported, lets you reuse the processed form of a stable prefix such as your system instructions and few-shot examples across many calls, cutting both cost and latency substantially when much of your context repeats. Retrieving fewer, better chunks, courtesy of reranking, both improves quality and shrinks the bill. Compressing long histories keeps multi-turn sessions affordable. And choosing the right model for each step, reserving the most expensive model for the work that truly needs it, keeps the economics sane.
The point is that context decisions are not only quality decisions; they are the main determinant of whether your system is cheap and fast enough to run at scale. A pipeline that produces beautiful answers but costs too much per query or takes too long to respond will not survive contact with real usage, so treat tokens as the scarce resource they are and design the context to use as few as the task genuinely requires.
Common failure modes
A handful of patterns account for most context-related problems, and naming them lets you audit your own system against the list. The dump-everything pipeline floods the window with marginal material and suffers on cost, latency, and the lost-in-the-middle effect all at once. The character-count chunker shreds documents into fragments that match queries but do not answer them. The vector-only retriever quietly fails on exact-term queries that hybrid search would have nailed. The no-rerank pipeline shows the model topically similar but irrelevant passages and gets confidently wrong answers.
The unbounded history lets a conversation grow until it crowds out the task and the budget. The trusting pipeline pipes untrusted retrieved text straight into a privileged instruction position and opens itself to injection. And the unmeasured pipeline changes retrieval settings on intuition with no way to know whether quality went up or down. Each of these has a known fix described above; the discipline is to recognize which one you are suffering from rather than reaching reflexively for a bigger model that will not address any of them.
The engineering mindset
The throughline of all of this is a single reframing. The context window is not a chat box; it is a constructed artifact, built fresh on every request by a pipeline you design and control, comprising retrieval, ranking, compression, formatting, and memory. The model is the easy part to swap and the hard part to blame; the context pipeline is the part you actually own and the part that determines whether the model can succeed. Most of the gap between a system that demos well and one that works in production is a gap in that pipeline, not in the model weights.
So before you reach for a bigger model or a longer prompt, look at the context. Is the right information being retrieved at all? Is it ranked so the best material is in front? Is it chunked so each piece is coherent? Is it compact enough to leave the model attention undivided, and positioned where attention is strongest? Is it labeled so the model knows what to trust? Is it measured so you know when you have improved it? Answer those questions well and the model will usually rise to meet the context you have built for it. Context engineering is the discipline that makes that possible, and it is, for most applications, the highest-leverage place to spend your effort.
Multimodal and structured context
Not all context is prose. Increasingly you will need to place images, tables, code, and structured records into the window alongside text, and each kind has its own handling. Tables lose their meaning when flattened into a line of comma-separated values, so preserve their structure or describe them explicitly. Code benefits from being delimited and labeled with its language and file, because the surrounding signal helps the model interpret it. Structured records from a database are often better passed as clean, labeled fields than as a raw dump, so the model can see which value is which. With multimodal models, an image is itself context that consumes budget and competes for attention, so the same discipline applies: include the views that matter, describe what the model should attend to, and do not assume that more inputs are always better. The principle does not change across modalities; only the formatting does.
Choosing what to leave out
If there is one habit that separates mature context engineering from naive context engineering, it is the willingness to leave material out. The naive instinct, when an answer is wrong, is to add more context, on the theory that the model must have been missing something. Sometimes that is true. Just as often the model had what it needed and was distracted by everything else you included. Subtraction is an underrated tool. When a pipeline misbehaves, try removing context as well as adding it, and measure both directions. You will frequently find that a leaner context produces better answers, faster and more cheaply, than the bloated one you started with.
This is psychologically hard because leaving something out feels risky in a way that including it does not. Including a marginal document feels safe, like buying insurance; leaving it out feels like gambling that you will not need it. But the costs of inclusion are real and recurring, paid on every call in money, latency, and degraded attention, while the benefit is hypothetical and rare. Discipline means resisting the safe-feeling default and letting your evaluation set, not your anxiety, decide what earns a place in the window.
A reference shape for the pipeline
Pulling the pieces together, a solid retrieval pipeline tends to look like this. Ingestion parses source documents, chunks them on structural boundaries with sensible overlap, attaches metadata and breadcrumbs, and indexes both vectors and keywords. At query time, the incoming question is rewritten into a standalone, search-friendly form, optionally expanded into sub-queries. Hybrid search casts a wide net over the index, metadata filters enforce scope and permissions, and a reranker narrows the candidates to the few best. Those passages are assembled into a labeled, well-positioned context together with the system instructions and the user question, compressed if they exceed budget, and handed to the model with an instruction to answer only from the provided material and to cite it.
None of those stages is sacred, and a simple application may collapse several of them. The value of the reference shape is as a checklist: when quality is not where you want it, you can walk the stages and ask which one is letting you down, rather than treating the whole thing as an opaque box that either works or does not. Each stage is independently measurable, independently improvable, and independently replaceable, which is exactly the property that makes a system maintainable as it grows.
The bottom line
Context engineering is the quiet discipline that determines whether an LLM application is reliable, affordable, and trustworthy. The model gets the attention, but the context pipeline does the work, and it is the part you actually control. Build it as a deliberate system: retrieve precisely, rank ruthlessly, chunk with care, compress with intent, remember selectively, format for legibility, ground in sources, defend against injection, and measure everything. Spend the token budget like it is scarce, because it is, and be as willing to remove context as to add it. Do that, and most of the problems people blame on the model will simply dissolve, because the model was never the bottleneck. The context was.
A note on iteration
Finally, treat the whole pipeline as something you iterate rather than something you finish. Your documents change, your users ask new kinds of questions, the models you depend on get upgraded, and a configuration that was optimal six months ago drifts out of tune. The teams that keep their retrieval quality high are not the ones who got the architecture perfect on the first try; they are the ones who instrumented the pipeline, watched real queries fail, and fixed the specific stage responsible, over and over. Logging the queries that returned nothing useful, sampling real sessions, and folding new failure cases back into the evaluation set are the unglamorous habits that compound into a system that quietly gets better while everyone else is busy swapping models and hoping. Context engineering is not a one-time build; it is an ongoing practice, and the practice is what produces durable quality.
The quiet truth of building with language models is that the model is rarely the thing standing between you and a great product. The context you feed it almost always is. Master the pipeline that decides what the model sees, and you master the part of the system that actually determines whether it succeeds, the part competitors cannot copy and bigger models cannot rescue. That is why context engineering, unglamorous as it sounds next to the models it serves, is where the real and durable advantage lives, and why it deserves to be treated as a first-class engineering discipline rather than the plumbing behind the prompt.