The thing nobody budgets for
Most teams building on large language models spend their first months obsessing over the prompt and the model, and almost none on memory. That order is backwards. A model with a brilliant prompt and no memory is a brilliant amnesiac: it can reason its way through any single turn and then forget everything the moment the turn ends. The systems that feel intelligent over time — the assistant that remembers your project, the agent that learns from yesterday's mistake, the support bot that knows you already tried the obvious fix — are not running better models. They are running better memory.
Memory is the part of an agent that turns a sequence of disconnected calls into something that behaves like a continuous actor. It is also the part most likely to be improvised, bolted on after the fact as "let's just stuff the history into the context window," and then quietly blamed when the agent starts contradicting itself, forgetting commitments, or recalling things that never happened. This post is about treating memory as a system to be designed rather than a side effect of a long prompt: what the word actually means, the distinct kinds of memory an agent needs, how to read and write them, and — the half everyone skips — how to forget.
The context window is not memory
The single most expensive misconception in this space is that the context window is the agent's memory. It isn't. The context window is the agent's attention — the narrow band of information the model can see on the current turn. It is large, it is getting larger, and it is still the wrong tool for remembering.
Three properties make the context window a bad memory store. It is volatile: when the turn ends, whatever was in the window is gone unless something outside the model wrote it down. It is bounded: even a million-token window fills up, and long-running agents generate far more history than any window can hold. And it is expensive and slow in a way that scales with its contents: every token in the window is paid for, in latency and in money, on every single call, whether or not it matters to the current step.
Real memory lives outside the model. It is a set of stores — files, databases, vector indexes, key-value caches — that persist across turns and sessions, plus the machinery that decides what to pull from those stores into the context window at the right moment. The context window is the desk; memory is the filing cabinet, the notebook, and the librarian who fetches the right folder. Conflating the two leads to systems that try to keep the entire filing cabinet open on the desk at all times, which is exactly as workable as it sounds.
Once you internalize that memory is external and the window is just a viewport onto it, the design questions become concrete. What do we store? Where? How do we decide what to load on a given turn? When do we update what we've stored, and when do we throw it away? Those four questions — store, retrieve, write, forget — are the whole discipline.
A working taxonomy
Borrowing loosely from cognitive science gives us a vocabulary that maps surprisingly well onto what agents actually need. The borrowing is a metaphor, not a claim that agents work like brains, but the categories carve the problem at useful joints.
The first cut is by duration. Working memory is the information relevant to the task right now — the current goal, the last few tool results, the partial plan. It is small, hot, and lives in or near the context window. Long-term memory is everything the agent might need later but doesn't need now: past conversations, learned facts about a user, accumulated procedures. It is large, cold, and lives in external stores, retrieved on demand.
The second cut is by content, and it applies mostly to long-term memory:
Episodic memory is the record of specific events: what happened, when, in what order. "On Tuesday the user asked me to draft a contract and rejected the first version for being too formal." Episodic memory is how an agent answers "what did we do last time."
Semantic memory is distilled facts, stripped of the episode that produced them: "the user prefers an informal tone in contracts." Semantic memory is what's left after you forget the meeting but keep the lesson.
Procedural memory is knowledge of how to do things — the workflows, tool sequences, and heuristics that worked before. In agents this often lives as reusable skills, saved plans, or few-shot examples of successful trajectories.
These are not just academic labels. They have different lifecycles, different storage shapes, and different retrieval patterns. Episodic memory is append-heavy and time-ordered; you rarely edit the past, you just add to it. Semantic memory is update-heavy and deduplicated; a new fact about the user should overwrite the old one, not pile up beside it. Procedural memory is read-heavy and curated; you want the small set of patterns that reliably work, not every approach you ever tried. A memory system that treats all three the same way will be wrong for at least two of them.
Working memory: managing the live context
Before anything persists, the agent has to manage the information in front of it. Working memory is the discipline of deciding what occupies the context window on the current turn, and it is where a surprising amount of agent reliability is won or lost.
The naive approach concatenates everything: the system prompt, the full conversation, every tool call and its complete output, the running plan. This works until it doesn't, and it stops working in two ways at once. The window fills, forcing truncation that often lops off exactly the early instructions that mattered. And even before it fills, performance degrades, because models attend less reliably to information buried in the middle of a long context — the well-documented "lost in the middle" effect. A window that is technically large enough can still be functionally overloaded.
Good working-memory management is mostly curation. A few techniques carry most of the weight:
Scratchpads. Give the agent an explicit place to write its current plan, intermediate results, and open questions — a structured note it rewrites as it goes, rather than re-deriving state from the raw history every turn. The scratchpad becomes the agent's working memory of record, compact and current.
Tool-result distillation. A raw API response or a 4,000-line file dump rarely belongs in the context verbatim. Summarize tool outputs down to what the next step needs, and keep a pointer to the full result in case it's needed again. The agent reads the summary; the detail stays one fetch away.
Externalized state. Anything the agent can recompute or look up should not occupy the window permanently. Store the task list, the file inventory, the configuration in an external structure and load slices of it on demand.
The mental model is that the context window is a budget, not a bucket. Every token spent on stale history is a token not available for the current problem. Working-memory management is the ongoing act of spending that budget well.
From turn to turn: the conversation problem
The simplest long-running agent is a chat that spans many turns, and even this minimal case forces the first real memory decision: how do you carry the conversation forward without dragging the entire transcript along?
The crudest answer is the full buffer — replay every message every turn. It is correct and it is doomed; the cost and the lost-in-the-middle problem grow without bound. The next answer is the sliding window: keep the last N turns and drop the rest. Cheap and bounded, but it amputates the beginning of the conversation, which is often where the user stated the goal and the constraints. An agent that forgets the original ask after twenty exchanges is worse than useless; it's confidently off-task.
The technique that actually scales is progressive summarization, sometimes called compaction. As the conversation grows, periodically replace the oldest raw turns with a running summary that preserves their salient content — decisions made, facts established, commitments outstanding — while discarding the verbatim back-and-forth. The context then holds a compact summary of the distant past plus the verbatim recent turns, which is exactly the shape human conversational memory takes.
Compaction is deceptively subtle. Summarize too aggressively and you lose the specific detail that turns out to matter — the exact figure, the precise wording of a constraint. Summarize too timidly and the summary itself grows unbounded, defeating the purpose. The summary is lossy by design, and the engineering problem is choosing what kind of loss is acceptable. A good compaction prompt is explicit about what must survive: open questions, user preferences, unresolved decisions, hard numbers, and anything the user asked the agent to remember. Everything else is fair game to compress.
A useful test for any compaction scheme: after summarizing, can the agent still answer "what did I ask you to do, and what's left?" If the answer ever degrades, the summary is dropping the wrong things.
Long-term memory: persistence beyond the session
Everything so far stays within a single session. The leap to genuine long-term memory is the leap across sessions: an agent that remembers you tomorrow, next week, after a restart. This is where memory stops being context management and becomes a storage problem with its own architecture.
The core operation is straightforward to state and hard to do well: write salient information to a durable store during or after a session, and retrieve the relevant slice of it at the start of and during future sessions. Everything interesting is in the words "salient" and "relevant." Storing everything is cheap to write and ruinous to read — a store that contains every word the user ever said is a store you can't retrieve from precisely, because everything matches everything. Storing too little defeats the point. The art is in the filtering on both ends.
Long-term stores come in a few shapes, and mature systems usually combine them:
Document and key-value stores hold structured facts and explicit user settings: name, preferences, account details, stated goals. These are precise, cheap to read, and easy to update in place. If you know the key, you don't need search.
Vector stores hold unstructured memories — snippets of past conversation, observations, notes — indexed by semantic embedding so they can be retrieved by meaning rather than exact match. This is how an agent recalls "something relevant to what the user is asking now" without knowing in advance what to look for.
Knowledge graphs hold entities and the relationships between them: this user works at this company, which makes this product, which depends on that service. Graphs shine when the questions are relational — "who else is connected to this" — and when contradictions need to be reconciled against an explicit structure rather than buried in free text.
No single shape wins. Structured stores are precise but rigid; you have to know in advance what fields matter. Vector stores are flexible but fuzzy; they retrieve by similarity, which is sometimes exactly wrong. Graphs are expressive but expensive to build and maintain. A practical memory layer routes different kinds of information to different stores and queries across them, the same way a person uses a calendar for dates, a notebook for ideas, and plain recall for everything else.
Retrieval: the act of remembering
Writing to memory is easy; the hard, high-leverage problem is retrieval — pulling the right handful of memories into the context at the right moment. An agent with a perfect store and poor retrieval behaves exactly like an agent with no memory, because the relevant fact is sitting in the database, unread. Most "the agent forgot" bugs are retrieval bugs, not storage bugs.
The baseline technique is semantic search: embed the current query, find the nearest memories in vector space, and inject them. It works well enough to be the default, and it fails in predictable ways that a serious system has to correct for.
Pure similarity ignores recency. A memory from an hour ago and one from six months ago score identically if they're equally similar, even though the recent one is usually more relevant. It ignores importance: a throwaway aside and a critical standing instruction are weighted the same if they happen to match the query equally. And it is blind to the difference between similar and relevant: a memory can be semantically close to the query and still be the wrong thing to surface — the same words, the opposite meaning.
Production retrieval blends multiple signals into a single ranking, an approach popularized by the generative-agents line of work:
Relevance — semantic similarity between the memory and the current situation.
Recency — a decay term that favors recent memories, so the agent weights the present over the distant past unless the past is strongly relevant.
Importance — a score assigned when the memory is written, marking standing instructions and significant events above routine chatter.
Combine these into a weighted score, retrieve the top results, and you get behavior that feels far more like remembering and far less like keyword matching. Two more refinements earn their keep. Reranking takes a generous first-pass set of candidates and runs a more expensive, more accurate model to reorder them, trading a little latency for markedly better precision. And query transformation rewrites the raw user message into a better search query before retrieval — expanding pronouns, adding context, or splitting a compound question — because the literal text of a user's message is often a poor query for the memory that would actually help answer it.
Writing memory: what's worth keeping
If retrieval is where most bugs surface, writing is where most of them originate. Garbage in, garbage retrieved. The decision of what to commit to long-term memory, and in what form, sets a ceiling on how good retrieval can ever be.
The first question is what to store. Saving the raw transcript is tempting and usually wrong: it is bulky, repetitive, and full of conversational filler that pollutes retrieval. The better pattern is extraction — after a turn or a session, run a step that pulls out the durable, reusable information and stores that. From "I'm working on a React app and honestly I've never liked CSS-in-JS, it always feels clunky to me," the memory worth keeping is the distilled fact: user dislikes CSS-in-JS; works in React. The episode can be discarded; the lesson is what persists.
The second question is when. Writing on every turn is expensive and noisy; most turns produce nothing worth remembering. Writing only at session end risks losing information if the session is abandoned. A common compromise is to extract continuously but commit selectively — buffer candidate memories as they appear, and write them through when they clear an importance threshold or when the session closes.
The third, and most neglected, question is reconciliation. New information frequently contradicts old. The user said they preferred email last month; today they asked for Slack. A memory system that simply appends both ends up with a store that contradicts itself, and retrieval will surface whichever happens to match — a coin flip between current truth and stale fact. Handling this requires treating semantic memory as mutable: when a new fact arrives, check whether it updates or contradicts an existing one, and revise rather than accumulate. This is the single biggest quality difference between a memory system that gets better over time and one that slowly fills with noise.
Forgetting: the half nobody builds
Engineers instinctively treat forgetting as failure. In a memory system it is a feature, and the absence of it is a slow-acting bug. A store that only grows will, given enough time, become slower to search, more expensive to maintain, more internally contradictory, and — counterintuitively — worse at remembering, because the signal drowns in accumulated noise. Human memory forgets aggressively and is better for it; agent memory should too.
Deliberate forgetting takes several forms, and a mature system uses more than one:
Decay. Let memories lose weight over time unless reinforced. A fact retrieved and used repeatedly stays strong; one never touched fades and eventually drops below the threshold where it's worth retrieving. This mirrors the recency weighting in retrieval but operates on the store itself, not just the query.
Consolidation. Periodically compress clusters of related episodic memories into a smaller number of semantic ones. Ten separate observations that the user keeps asking about deployment become one durable fact: user is focused on deployment. The episodes can then be pruned, and the lesson survives at a fraction of the storage.
Deduplication and supersession. Detect when a new memory duplicates or replaces an old one, and collapse them. This is reconciliation applied as housekeeping rather than at write time — a sweep that keeps the store internally consistent.
Explicit expiry. Some memories are inherently temporary — a one-off instruction, a fact tied to a deadline that has passed. Tagging memories with a time-to-live lets them remove themselves when they stop being true.
There is a hard constraint wrapped around all of this: some forgetting must be honored on demand, not just scheduled. When a user asks an agent to forget something, or exercises a legal right to deletion, the system must be able to actually remove it — from the store, from derived summaries, from any consolidated memory it fed into. A memory architecture that can't reliably delete is not just inelegant; in many jurisdictions it is non-compliant. Forgetting, in other words, is both a quality feature and a governance requirement, and it has to be designed in from the start rather than retrofitted.
Architectures in the wild
The pieces — working memory, summarization, durable stores, scored retrieval, selective writes, deliberate forgetting — assemble into a handful of recognizable architectures. They are points on a spectrum from cheap-and-dumb to expensive-and-rich, and the right choice depends on how long your agent runs and how much continuity it needs.
Buffer memory. Keep the recent conversation verbatim, drop the rest. Trivial to build, fine for short single-session tasks, useless across sessions. The right default for a stateless tool, the wrong one for an assistant.
Summary memory. Maintain a running summary that compresses the conversation as it grows. Bounded cost, preserves the gist, loses fine detail. The workhorse for long single sessions.
Vector memory. Write memories to a vector store and retrieve by similarity each turn. The standard approach for cross-session recall, and the one most teams reach for first. Powerful, and only as good as its retrieval scoring and its write filtering.
Knowledge-graph memory. Maintain an explicit structure of entities and relations, updated as new information arrives. More work to build and maintain, but uniquely good at relational queries and at surfacing contradictions instead of burying them.
Hierarchical or generative memory. Combine several of the above into layers — raw episodes at the bottom, consolidated semantic facts in the middle, high-level reflections at the top — with a periodic process that promotes information upward by reflecting on what the lower layers contain. This is the architecture behind the most convincing long-running agents, and also the most complex to operate.
The recurring mistake is reaching for the richest architecture first. A knowledge graph with reflective consolidation is impressive and entirely unnecessary for an agent that runs for ten minutes and never sees the user again. Start with the simplest memory that meets the continuity requirement, and add layers only when observed behavior shows you need them. As with agent design generally, complexity is a cost you pay on every turn, not a feature you get for free.
Memory and personalization
The most visible payoff of long-term memory is personalization — the agent that knows who you are, how you work, and what you've done together. This is also where memory becomes a product feature rather than an implementation detail, and where the difference between a tool people try once and a tool people adopt is often decided.
Personalization memory is mostly semantic and procedural: stable facts about the user and the patterns of working with them. The right tone, the recurring projects, the preferences stated once and expected to hold. Done well, it is invisible — the agent just behaves as if it knows you, and you stop re-explaining yourself. Done poorly, it is uncanny in both directions: an agent that forgets what you told it last week feels broken, and an agent that surfaces something you'd rather it hadn't remembered feels invasive.
That tension is the design problem at the heart of memory-as-product. Users want to be remembered and they want control over what's remembered, and those goals pull against each other. The systems that get it right tend to make memory legible and editable: the user can see what the agent has stored about them, correct it, and delete it. A memory the user can inspect is a memory they can trust; a hidden one, however accurate, is a liability the first time it surfaces something unexpected.
Memory is not RAG, exactly
It is worth drawing a line between memory and retrieval-augmented generation, because they share machinery and get conflated constantly. Both embed text, store it in a vector index, and retrieve by similarity to enrich the context. Under the hood, an agent's vector memory and a RAG pipeline can look nearly identical. But they answer different questions, and treating them as the same thing leads to systems that do neither well.
RAG retrieves from a corpus of knowledge — documentation, a knowledge base, a set of reference documents — that exists independently of any conversation and is largely the same for every user. Its job is to ground the model in facts it wasn't trained on. Memory retrieves from a record of interaction — what this user said, what this agent did, what was decided — that is specific to a relationship and accumulates over time. Its job is continuity, not grounding.
The practical differences follow from that. RAG corpora are mostly read-only and curated up front; memory is written continuously by the agent itself, which means write quality and reconciliation matter in a way they don't for a static knowledge base. RAG rarely needs to forget; memory does, constantly. RAG is shared across users; memory is scoped to one, with all the privacy obligations that implies. A system can and often should have both — a RAG layer for what's true in general, and a memory layer for what's true about this user and this history — but they are different subsystems with different lifecycles, and the cleanest designs keep them separate even when they share an index.
Shared memory in multi-agent systems
When a single agent becomes a team of agents, memory acquires a new dimension: what's private to one agent and what's shared across all of them. A research agent, a writing agent, and a review agent working on the same task need a common picture of the goal and the findings so far, but they also each maintain their own working state that the others neither need nor should see.
The usual pattern is a shared scratchpad or blackboard — a common store the agents read from and write to — layered over each agent's private working memory. The shared layer holds the task definition, established facts, and intermediate artifacts; the private layers hold each agent's reasoning and partial work. Getting the boundary right matters in both directions. Share too little and the agents duplicate effort or work from inconsistent pictures of the task. Share too much and the blackboard bloats into the same overloaded-context problem a single agent faces, except now multiplied across every member of the team.
Shared memory also concentrates the failure modes. A poisoned or contradictory entry in a shared store propagates to every agent that reads it, so the validation and reconciliation discipline that matters for single-agent memory matters more here, not less. The same rule applies: shared memory is a system to be designed deliberately, with explicit decisions about what crosses the boundary and what stays private, rather than a global variable every agent scribbles on.
The failure modes
Memory introduces a whole class of failures that stateless systems simply don't have. Knowing them in advance is most of the defense, because nearly every one is a guard you can design in rather than a surprise you discover in production.
Memory poisoning. A false or malicious fact gets written and then retrieved as if true, shaping behavior on every future turn. Because the agent trusts its own memory, a single bad write can persist and compound. Defenses: validate and attribute what gets written, and treat memory as evidence to weigh rather than ground truth to obey — especially when the content of a memory looks like an instruction.
Staleness. A memory that was true becomes false, but nothing updates it. The agent confidently acts on outdated information. Defenses: reconciliation at write time, decay and expiry on the store, and a bias toward recent information when memories conflict.
Contradiction. The store holds mutually incompatible memories and retrieval surfaces them inconsistently, so the agent's behavior flickers. Defenses: deduplication, supersession, and treating semantic memory as mutable rather than append-only.
Retrieval misses. The relevant memory exists but isn't retrieved, so the agent acts as if it never knew. The most common and least visible failure, because nothing errors — the agent just quietly forgets. Defenses: better scoring, query transformation, reranking, and evaluation that specifically measures recall.
Context bloat. Retrieval is too generous and floods the window with marginally relevant memories, drowning the current task. Defenses: tight top-k limits, reranking for precision, and distillation of what's retrieved before it enters the window.
Privacy leakage. Memory persists information the user would not expect it to, or surfaces one user's data in another's session. Defenses: strict scoping of memory to its owner, careful filtering of what's eligible to be written, and honest deletion.
The pattern across all of them is that memory shifts the failure surface from a single turn to the whole history. A stateless agent can only be wrong about the current input; a stateful one can be wrong about everything it has ever stored, and stay wrong indefinitely. That is the price of continuity, and the reason memory deserves the same engineering rigor as the loop and the tools.
Evaluating memory
You cannot improve what you don't measure, and memory is harder to measure than most agent components because its effects show up over time rather than in a single response. A memory bug rarely produces an obvious error; it produces a slow drift into forgetfulness or confusion that no single-turn test will catch.
Useful evaluation works at a few levels. At the retrieval level, treat it as an information-retrieval problem: build a set of queries with known relevant memories and measure precision and recall directly. This isolates the retrieval layer from the model and tells you whether the right memories are even reaching the context. At the behavioral level, construct multi-session scenarios — tell the agent something in session one, then test in session three whether it acts on it — and score whether the memory actually changed behavior the way it should. At the regression level, keep a suite of remembered facts and periodically confirm the agent still recalls and applies them, so a change to the memory pipeline can't silently erase things it used to know.
The harder qualities to measure are the ones that matter most: does the agent forget the right things, reconcile contradictions correctly, and avoid surfacing what it shouldn't? These resist automated scoring and usually need scenario-based tests written by hand, plus human review of memory contents over time. It is slower than running a benchmark, and there is no shortcut, because the whole point of memory is behavior that only emerges across many interactions.
Where to start
The temptation, having read all of this, is to build the full hierarchical, reflective, multi-store memory system on day one. Resist it. Memory rewards incremental construction, because each layer's value depends on the one beneath it working, and because most agents need far less than the maximal design.
A sane progression looks like this. Start with working-memory hygiene: a scratchpad and tool-result distillation, so the agent manages its live context well even within a single session. Add summarization next, so long sessions stay bounded without amputating their own beginnings. Only then introduce a long-term store, and start with the simplest one that fits — a structured store for explicit user facts often delivers more value than a vector store, because precise recall of known fields beats fuzzy recall of everything. Add semantic vector memory when you need recall over unstructured history. Add reconciliation and forgetting as soon as the store is more than a few sessions old, because that is exactly when contradiction and bloat begin to bite. Reach for graphs and reflective consolidation last, only when relational queries or genuine long-horizon learning demand them.
At every step, instrument before you elaborate. The most common way memory projects go wrong is investing in sophisticated storage while retrieval and write-filtering stay crude — a beautiful filing cabinet with a librarian who grabs the wrong folder. The leverage is almost always in deciding what to write and what to retrieve, not in the storage technology underneath. Get those two right with a simple store before you make the store fancy.
Where this is going
Memory is the frontier along which agents will most visibly improve over the next few years, precisely because it is the part that has been least engineered. The models are already capable of remarkable single-turn reasoning; what they lack is continuity, and continuity is a memory problem, not a model problem. The agents that feel like they're getting to know you, learning your domain, and improving from their own experience will be the ones whose builders treated memory as a first-class system.
The fundamentals will not change with scale. Larger context windows will keep raising the ceiling on working memory, but they will not abolish the need for external stores, because windows are still volatile, bounded, and expensive per token, and history still outgrows them. Better models will make extraction and reconciliation more reliable, but they will not decide for you what is worth remembering or what should be allowed to fade. Those are design choices, and they belong to whoever is building the system.
The teams shipping agents that genuinely remember are not the ones with the largest context windows or the cleverest prompts. They are the ones who decided, early, that memory was a system to be designed — stores chosen deliberately, retrieval scored thoughtfully, writes filtered carefully, and forgetting built in on purpose rather than avoided out of instinct. An agent's intelligence is borrowed from its model. Its character — the sense that it knows you, remembers what matters, and learns — is something you build, one well-designed memory at a time.
