An agent that can read your CRM, search your ticket history, and draft a reply has, in the space of a single turn, assembled a dossier that no employee would have been permitted to compile by hand. Nothing was breached. Every call was authorized. The privacy problem in agentic systems is not that they break the rules; it is that they follow rules written for a world where assembling context was slow, deliberate, and visible.
The perimeter moved and nobody redrew the map
Traditional data protection is built on a geography. There is a database, and it sits inside a network, and access to it is mediated by a service that checks who you are before it answers. The controls follow the geography: encrypt the disk, restrict the network, authenticate the caller, log the query. When the shape of the system is stable, the shape of the protection can be stable too, and an auditor can point at a diagram and ask reasonable questions about each box on it.
Agents dissolve that geography. The unit of access is no longer a query against a table; it is a goal expressed in natural language that gets decomposed, at runtime, into an unpredictable sequence of reads across systems that were never designed to be traversed together. The customer record comes from one place, the payment history from another, the support transcripts from a third, and the model stitches them into a single reasoning context that exists nowhere in your architecture diagram and is destroyed a few seconds later, having been transmitted to an inference provider in the meantime.
This is not merely a bigger version of the old problem. It is a different problem, because the thing you now need to govern is not storage and not access but composition. Each individual read may be entirely legitimate under the permissions you granted. The aggregate, assembled in a context window and summarized into a paragraph, may be a disclosure that would have required legal review had a human produced it. Privacy engineering for agents is largely the discipline of governing the aggregate when your existing controls only understand the parts.
What counts as personal data when a model is reading it
Most engineering teams carry a mental model of personal data that is essentially a list of field names. Email address, phone number, national identifier, card number, date of birth. This list is where data loss prevention tools start, it is what the schema annotations mark, and it is comfortably finite. It is also, in an agent context, dangerously incomplete.
Regulation has never actually defined personal data by field name. It defines it by identifiability: any information relating to a person who can be identified, directly or indirectly, by reference to that information alone or in combination with other information reasonably available. The operative phrase is in combination, and combination is precisely what a language model does for a living. A ticket that says the customer on the third floor of the Rotterdam office who reported the printer issue last Tuesday contains no field that any classifier would flag, and identifies exactly one person.
The practical consequence is that free text has to be treated as presumptively sensitive rather than presumptively safe. Structured fields are the easy case, and teams that focus their controls there end up with excellent protection over the ten percent of their data that was never the problem. The transcripts, the notes, the email bodies, the chat logs, the comments in a project tracker: this is where the identifiability lives, and it is also, not coincidentally, exactly what retrieval systems were built to surface.
There is a second category that catches people out, which is inference. An agent that reads a customer's purchase history and concludes that they are probably pregnant has created a piece of health information out of retail data. Nothing in the source was sensitive. The output is. Systems that reason produce new facts about people, and those facts inherit the protections of their category rather than the protections of their inputs. A model that infers is a model that can manufacture a special category of data from ordinary material, which is a capability no data map anticipates.
Where the data actually goes
Before you can protect anything you need an honest account of the paths data takes through an agent system, and the honest account is usually longer than the design document. There is the prompt, which carries whatever context the orchestration layer assembled and is transmitted to an inference provider. There is the retrieval layer, which pulled that context from indexes that may themselves be hosted elsewhere and which almost certainly logged the query. There is the tool layer, which sends arguments to external services chosen by the model at runtime. There is the output, which goes to a user, a downstream system, or a file. And there is the exhaust: traces, logs, evaluation datasets, error reports, and the cache.
Each of these is a boundary crossing and each deserves separate treatment, but the exhaust is the one that consistently surprises people. A team that has thought carefully about what goes into the prompt will frequently discover that their observability platform has been storing every prompt verbatim for ninety days in a third-party service in another jurisdiction, because that is what the default configuration of the tracing library does and nobody read it as a data transfer. The most rigorous prompt hygiene in the world is undone by a debug log that captured the same content on its way past.
Mapping this properly is unglamorous work and it is the only foundation that holds. For each hop, you want to know what data is present, where it physically goes, how long it is retained, who can read it, and what contractual terms cover it. Teams that skip this step are not doing privacy engineering; they are doing privacy theatre with good intentions, and the difference becomes apparent the first time somebody asks a question with a deadline attached.
Minimization at the source, not at the boundary
The instinct when facing this is to build a filter at the edge: let the pipeline assemble whatever it wants, then scrub the result before it goes to the model. This is appealing because it is a single control point, and it is weaker than it looks, for the same reason that output filtering is weaker than input validation everywhere else in computing. You are trying to recognize sensitive content in arbitrary text, which is a recall problem with no ceiling, and every miss is a disclosure.
Minimization at the source inverts this. Rather than retrieving everything and removing what should not have been there, you retrieve only what the task requires. This sounds obvious and it is structurally difficult, because the whole appeal of retrieval is that you do not have to know in advance what will be relevant. The resolution is to make relevance a function of the task rather than of the query. An agent handling a billing dispute needs the invoice, the payment record, and the account status. It does not need the medical notes attached to the same customer, and no amount of semantic similarity should be able to reach them, because the retrieval scope was bounded by the task before the search ever ran.
This means task-scoped retrieval contexts: a declaration, per workflow, of which sources and which categories of field are in scope, enforced at the index level rather than filtered afterward. It is more work to set up and it fails safe. When a new data source is added to the warehouse, it does not silently become available to every agent; it becomes available to the workflows that declared it. The default is exclusion, which is the only default that survives organizational growth.
The same logic applies to fields within a record. A model reasoning about whether a refund is warranted needs to know that a payment succeeded, not the card number that made it. Projecting records down to the attributes a task actually consumes eliminates entire categories of exposure without any detection machinery at all, and unlike detection, projection has no false negative rate.
Redaction, tokenization, and the difference that matters
When data genuinely must pass through a model but its specific values are not needed, there are two families of technique and they behave very differently in practice.
Redaction removes. The name becomes a placeholder, the number disappears, and what reaches the model is text with holes in it. This is safe and it is lossy in ways that damage reasoning. An agent asked to determine whether the same person appears in two documents cannot do so if both names were replaced by the same generic token, and cannot do so if they were replaced by different ones. Redaction destroys the relational structure that reasoning depends on, and the failure it produces is not a refusal but a confidently wrong answer built on flattened evidence.
Tokenization substitutes consistently and reversibly. Each distinct value is replaced by a stable surrogate, so the same person is the same token everywhere, relationships survive, and the mapping is held in a vault the model never touches. The agent reasons over PERSON_A and ACCOUNT_7, produces an output referring to them, and the orchestration layer rehydrates the real values on the way back to an authorized recipient. Reasoning quality holds up remarkably well, because models are largely indifferent to whether an entity is called Sarah Chen or PERSON_A as long as the reference is consistent.
The trap in tokenization is that consistency is itself a disclosure channel. If the same surrogate is used forever across all contexts, an observer with access to enough outputs can correlate them, and a token that appears in a medical context and a financial context has linked those records for anyone reading both. The mitigation is scoping: surrogates that are stable within a session or a task and unpredictable across them. You keep the reasoning benefit and you lose the long-range correlation, which is the right trade in nearly every case.
Format-preserving substitution deserves a mention because it solves a specific practical annoyance. Replacing a date of birth with a placeholder breaks any downstream code that expected a date. Replacing it with a different, structurally valid date keeps the pipeline working. The risk is obvious and worth stating plainly: synthetic values that look real will eventually be mistaken for real ones by someone reading a log, so they need to be marked as synthetic somewhere the reader will actually look.
Detection is not a solved problem
Every redaction strategy rests on detection, and detection is where the honest engineering conversation happens. Regular expressions handle the structured cases well: card numbers have checksums, national identifiers have formats, email addresses have shapes. For these, precision and recall are both high and the tooling is mature.
Everything else is hard. Named entity recognition models find names, places, and organizations with accuracy that is good but not close to sufficient when the cost of a miss is a disclosure. They miss unusual names, they miss names in scripts they were undertrained on, they miss the possessive forms, and they produce false positives on product names and technical terms in ways that quietly mangle content. Running a language model as the detector improves recall and introduces a new dependency: you are now sending the sensitive data to a model in order to determine whether it is safe to send to a model, which is fine if it is the same trust boundary and absurd if it is not.
The uncomfortable truth is that quasi-identifiers defeat all of it. The combination of a job title, a city, and a company name identifies a person with high probability and contains nothing any detector will flag. There is no regular expression for the third floor of the Rotterdam office. Teams that believe their redaction layer makes text anonymous are describing a property their tooling does not have, and the belief is more dangerous than the exposure, because it licenses decisions that would otherwise have been made carefully.
The reasonable posture is to treat detection as risk reduction rather than as a guarantee. Measure it: build a labelled corpus from your own data, because detector performance is domain-specific and vendor benchmarks will not predict your recall. Report false negatives by category so you know which kinds of exposure you are living with. And design the system so that a detection miss is contained by something else, whether that is a trusted inference boundary, a strict output policy, or a retention window short enough that the miss expires.
The context window is a disclosure surface
There is a habit of thinking about the prompt as ephemeral. It is constructed, sent, answered, and discarded, and because nothing is written to a database it feels less consequential than a query. This intuition is wrong in three distinct ways and each one has produced real incidents.
The first is that transmission is disclosure. Sending personal data to an inference provider is a transfer to a processor, with everything that implies contractually and jurisdictionally, whether or not it is retained. The second is that context is shared. In multi-turn sessions, in shared workspaces, in any system where one agent's output becomes another's input, data placed in a context window has a way of persisting into places it was never scoped for. A summary written for an internal reviewer becomes the seed of a message to a customer, and the summary contained a detail the customer should not have.
The third is cross-tenant leakage, which is the one that ends contracts. Any component that caches, batches, or shares state across users is a potential channel, and the components most likely to do this are the ones added for performance. A prompt cache keyed on a normalized prefix without a tenant identifier will serve one organization's context to another, and it will do so intermittently, which makes it nearly undetectable in testing and spectacular in production. Tenant identity belongs in every cache key, every embedding namespace, and every vector index partition, enforced by a wrapper that makes it impossible to omit rather than by a convention that developers are expected to remember.
The practical rule that follows is to treat the assembled prompt as a first-class artifact subject to policy. Before it goes out, it should be checkable: what categories of data does it contain, which tenant does it belong to, which provider is it destined for, and does that combination satisfy the policy for this workflow. This is a real component that someone has to build, and it is the single highest-leverage control in the entire stack, because everything downstream inherits its decisions.
Memory is the privacy problem that compounds
An agent with memory is an agent that accumulates a profile, and profiles are regulated in ways that transient processing is not. A system that reads a customer record to answer a question and forgets it has processed personal data. A system that writes down what it learned and consults it next time has created a record, and the record has a controller, a retention period, a lawful basis, and a subject who has rights over it.
This distinction is routinely missed because memory is usually built as an engineering feature rather than as a data store. It starts as a table of facts the agent found useful, added to improve continuity, and nobody classifies it because it did not come from a system of record. Six months later it contains a detailed and largely unreviewed behavioural profile of every user, assembled from inferences rather than declarations, with no deletion path and no documentation of what is in it.
Three properties make agent memory tractable. The first is typing: distinguish memory that stores facts the user explicitly provided from memory that stores inferences the system made. They have different accuracy characteristics, different correction obligations, and different sensitivity, and storing them in the same undifferentiated blob makes it impossible to treat them differently later. The second is expiry: memories should have a default lifetime tied to why they were stored, with renewal on use rather than permanent persistence, so that the profile decays toward the subset that is actually load-bearing. The third is legibility: a user or an auditor should be able to see what the system remembers about a person in a form a human can read, because a memory store that cannot be inspected cannot be corrected, and correction is not optional.
The inference case deserves particular care. An agent that records the user seems frustrated by the billing process has written down an opinion about a person and will act on it in future interactions. It may be wrong. It may have been true once and stopped being true. Persisted inferences are the fastest route from a helpful feature to a system that treats people according to a judgement they never saw and cannot contest.
Logs, traces, and the observability paradox
Agents are difficult to debug, and the standard remedy is to record everything: the full prompt, the retrieved documents, the tool calls with arguments, the intermediate reasoning, the output. This is the right instinct for reliability and it produces, as a byproduct, the most concentrated collection of personal data in the entire system. A trace store contains not just the data an agent touched but the data in the exact assembled form that made it dangerous, indexed for search, retained by default, and readable by anyone on the engineering team.
The paradox is genuine. You cannot operate a system you cannot observe, and you cannot observe an agent without capturing what it processed. The resolutions are all partial and all worth applying together.
Structural logging captures the shape of an interaction without its content: which sources were queried, how many documents came back, which tools were called, how long each step took, what the token counts were, whether policy checks passed. This is sufficient for a surprisingly large fraction of debugging, especially the performance and orchestration problems that make up most operational work. Content capture is then the exception rather than the default, enabled deliberately for a bounded window when investigating a specific failure.
Tiered retention acknowledges that the value of detailed traces decays fast while the risk does not. Full content for a small number of days, structural records for months, aggregate metrics indefinitely. Most teams have this backwards by accident, retaining everything for the longest period any component required.
Access control on traces is the one most often missing entirely. Production traces are personal data and should require the same authorization as the underlying systems, with their own audit trail. An engineer who cannot query the customer database directly but can read a trace containing the same record has a permission gap, and the gap exists in almost every organization that has shipped an agent, because the trace store was provisioned by the platform team as infrastructure rather than by the data team as a data store.
Tools are the real exfiltration path
Retrieval brings data in. Tools send it out, and that asymmetry is under-appreciated because the security conversation around agents has focused so heavily on what they read. A tool that sends an email, writes to an external system, posts to a webhook, or performs a web search is a channel from your data to somewhere else, and the arguments to that call are chosen by a model whose judgement about sensitivity is not something you should be relying on.
Web search is the clearest illustration. An agent debugging a customer issue that pastes an error message containing a customer identifier into a public search engine has just published it. The model did nothing wrong by its own lights; it was gathering information, which is what you asked for. The system was wrong to make that possible.
The controls here are boring and effective. Tools should be classified by where their data goes, with internal, partner, and public as the minimum granularity. Tool arguments should be validated and, where necessary, scrubbed before dispatch, with the scrubbing rules attached to the tool rather than applied globally, because the right rule for a search query is different from the right rule for an internal database call. Tools that cross a trust boundary should not accept free text where a structured parameter would do, since free text is where the accidental inclusion happens.
And prompt injection has to be part of this analysis, because it converts every tool into a directed exfiltration channel. A document in your corpus containing an instruction to summarize the customer file and send it to an external address is an attack that requires no vulnerability in your code, only an agent that reads instructions from data. The defence is not better instruction-following; it is ensuring that the set of destinations an agent can reach is bounded by policy rather than by content, so that the worst case of a successful injection is an action inside the boundary you already accepted.
Third-party inference and the question of who is processing what
Most teams call a model they do not host, which makes the provider a processor and makes the contract a control. This is where privacy engineering stops being purely technical, and it is worth understanding what the terms actually say rather than what the marketing page implies.
The questions that matter are specific. Is input retained, and for how long, and for what purpose. Is it used to train or improve models, and is the default opt-in or opt-out. Is there human review, and under what circumstances. Where does inference physically occur, and can that be constrained to a region. What happens to data that triggers a safety classifier, since content flagged for review frequently follows a different retention path than ordinary traffic, and that path is often the one nobody read about. Are subprocessors disclosed, and does the provider notify before adding them.
Enterprise tiers generally offer zero-retention or short-retention processing, regional deployment, and contractual assurances against training use. These are meaningful and they are not free, and the gap between the free tier defaults and the enterprise terms is exactly the gap between a prototype and something that can hold regulated data. A prototype built against consumer terms that graduates to production without anyone revisiting the contract is one of the most common ways an organization ends up in violation without a single line of bad code.
Self-hosting eliminates the transfer question and replaces it with an operational one. Running a model on infrastructure you control means the data never leaves, which is genuinely simpler from a compliance standpoint and genuinely harder from an engineering standpoint. For workloads involving special categories of data, health records, biometrics, criminal history, the calculus often favours a smaller self-hosted model over a larger external one, because the reduction in capability is easier to manage than the increase in exposure. This is a real architectural fork and it should be decided by the sensitivity of the data rather than by the preference of whoever is writing the code.
Purpose limitation for systems that improvise
Purpose limitation is the principle that data collected for one reason may not be casually used for another, and it is the principle agents are structurally worst at respecting. The entire value proposition of an agent is that it finds its own path to a goal, which means it will reach for whatever data seems useful, and useful is not the same as permitted.
Consider an agent built to answer support questions that has access to a customer database. A user asks why their delivery is late. The agent, reasoning helpfully, notices a field indicating the customer has an open complaint with the regulator, and mentions it. Every access was authorized. The purpose was not. The data was collected for compliance handling and used for support conversation, and the fact that both were performed by the same system with the same credentials is exactly the collapse that purpose limitation exists to prevent.
Encoding purpose is more than a prompt instruction, because prompt instructions are advisory and this needs to be structural. The workable pattern binds purpose to credentials: the agent operating in the support workflow holds a token whose scope excludes the compliance fields, and the restriction is enforced by the data layer rather than by the model's discretion. When the same agent switches to a compliance task, it acquires different scope through an explicit transition that is logged and, where appropriate, approved.
This is more infrastructure than most teams expect to build for what sounds like a policy concern. It is also the only version that holds up under adversarial conditions and under the ordinary condition of a model that is trying to be helpful. A model asked not to use certain data will comply most of the time, and most of the time is not a control.
Consent that survives contact with an agent
Consent frameworks assume a stable relationship between a purpose, a data category, and a moment of collection. A user is told what will happen and agrees or declines, and the system honours that. Agents strain this in two directions.
The first strain is granularity. Consent obtained for personalized recommendations does not obviously extend to an agent reading your message history to draft a reply on your behalf, but both could be described as improving your experience, and the description in the consent flow was written before agents existed. Teams inheriting consent records collected under older wording have a genuine question about scope, and the honest answer is usually that the old consent does not cover the new processing, which means re-consent rather than reinterpretation.
The second strain is dynamism. An agent's data use is determined at runtime, so the system cannot always tell in advance which categories a given task will touch. This argues for consent checks at the point of access rather than at the point of design: before the retrieval layer returns a record in a restricted category, it verifies the consent state for that subject and that purpose, and returns nothing if the check fails. The agent then reasons over what it was given, which may be less than it asked for, and the interesting design question becomes how it should behave when the answer it can give is incomplete.
The answer is that it should say so. An agent that silently omits data it was not permitted to use produces an answer that looks complete and is not, and the user has no way to know. An agent that reports that some information was not available for this request is being honest about a boundary, and honesty about boundaries is what makes the boundaries survivable. This is the same principle that governs how agents should report tool failures, and it applies with more force here, because the omission is invisible by design.
Deletion in a system that copies everything
A deletion request is a test of whether you understand your own architecture, and agent systems fail it in more places than conventional applications do. The record comes out of the primary database easily enough. Then there is the vector index, which contains embeddings derived from that record and which is not straightforward to selectively delete from in some engines. There is the cache. There is the agent's memory store. There are the traces. There is the evaluation dataset that someone exported for regression testing. There is the fine-tuning corpus, if you built one, and if the model was actually trained on it then the data is in the weights and no deletion operation exists at all.
The last of these is the one with no clean answer, and it should shape decisions made much earlier. Training on production personal data creates an obligation you may not be able to discharge, and the standard mitigations, retraining on request or approximate unlearning, are expensive and imperfect. The defensible approach for most organizations is not to train on identifiable production data in the first place, using synthetic or thoroughly de-identified corpora instead and accepting a smaller improvement in exchange for a much simpler position.
For everything else, deletion is an engineering discipline that has to be designed in rather than retrofitted. Every derived store needs a link back to the subject, which means embeddings carry subject identifiers in their metadata, memories carry them, traces carry them, and exported datasets carry them. A deletion request then becomes a fan-out across a registry of derived stores, each of which knows how to remove records for a subject and confirms that it did. Teams discover the need for this registry at the worst possible moment, which is when the first request arrives with a statutory clock attached and somebody has to answer where the data is.
Verification matters as much as execution. A deletion that silently failed in one store is worse than one that failed loudly, because it is reported as complete. The fan-out should collect confirmations, and stores that do not confirm should raise an alert rather than being assumed to have succeeded.
Anonymization and the limits of the word
Anonymization has a specific meaning in regulation, which is that the data is no longer personal data because no one can reasonably identify a person from it. Achieving that is much harder than most implementations assume, and the word is used loosely in ways that create false comfort.
Removing direct identifiers produces pseudonymized data, not anonymous data, and pseudonymized data remains fully regulated. The distinction is not pedantry: a team that believes it has anonymized a dataset will store it longer, share it more widely, and subject it to less scrutiny than the law actually requires. The re-identification literature is unambiguous that small combinations of quasi-identifiers are sufficient to single out individuals in large populations, and text is dense with quasi-identifiers.
Differential privacy is the one technique with a rigorous guarantee, and it is worth understanding why it appears rarely in agent systems. It works by adding calibrated noise so that the presence or absence of any single individual cannot be detected in the output, and it fits aggregate analytics naturally. It fits individual reasoning tasks poorly, because the whole point of the task is to produce a correct answer about one specific person, and there is no noise budget under which that is possible. Where it genuinely helps is at the edges of an agent system: in the analytics you run over usage, in the aggregate evaluation metrics you publish, and in any statistics derived from a corpus. Applying it there is valuable. Claiming it for the reasoning path is not credible.
Synthetic data occupies a similar position. Generated records that preserve statistical structure without corresponding to real people are genuinely useful for development, testing, and demonstration, and they remove a real category of exposure from environments that are usually less well protected than production. The caution is that a generator trained on real data can memorize and reproduce it, so synthetic corpora need to be checked for near-duplicates of their source before being treated as safe.
The regulatory picture, briefly and without pretending to be legal advice
Engineers do not need to be lawyers and do need to know which questions have legal answers, because the expensive mistakes are the ones where nobody realized a question existed.
The general data protection regimes, of which the European framework is the most influential and the most copied, establish the principles that have run through this entire piece: lawful basis, purpose limitation, minimization, accuracy, storage limitation, and the rights of subjects to access, correct, and erase. There is also a provision on automated decision-making that agent builders should read directly rather than in summary, because it constrains decisions producing legal or similarly significant effects on people, and an agent that approves or declines an application is squarely in that territory regardless of what the product calls it.
Sector rules add obligations that are frequently stricter and more specific. Health data, financial records, children's data, and biometrics each carry their own regimes, and the common feature is that the general framework's flexibility narrows considerably. An agent touching any of them is not a general-purpose agent with an extra feature; it is a regulated system that happens to use a model.
The newer wave of AI-specific regulation adds transparency and risk-classification duties on top of data protection rather than replacing it. Obligations to disclose that a user is interacting with an AI system, to document what the system does, and to assess risk before deployment are becoming the baseline expectation in multiple jurisdictions. None of this removes the underlying data protection duties, which is the point most commonly misunderstood: complying with an AI act does not discharge a privacy obligation, and the two sets of requirements interact rather than substitute.
The practical instruction for an engineering team is narrower than the landscape suggests. Know which categories of data your system touches, know which jurisdictions your subjects and your compute are in, involve counsel before the design is frozen rather than after it ships, and write down the assessment. The written assessment is not bureaucracy; it is the artifact that demonstrates you thought about it, and its absence is what turns a defensible decision into an indefensible one.
Measuring privacy the way you measure quality
Privacy in most organizations is a review gate rather than a metric, which means it is assessed once, before launch, on a system that then changes weekly. Treating it as something measurable and continuously monitored is the shift that makes it hold.
Leakage testing is the direct measure. Build a corpus of inputs designed to elicit inappropriate disclosure: questions that ask about other users, requests that would require combining data across purposes, prompts crafted to extract memory contents, injections that attempt to route data outward. Run them on every release and count failures. This is an eval suite like any other, and like any other it is only useful if it grows every time a real failure teaches you something new.
Detection accuracy needs its own measurement against a labelled sample of your own text, reported as recall by data category rather than as a single number, since the categories fail unevenly and an aggregate hides which exposure you are carrying.
Coverage metrics answer a different question: what fraction of derived stores are registered for deletion, what fraction of tools have a data classification, what fraction of workflows have a declared retrieval scope. These are boring and they catch the drift that individual tests miss, because the common failure is not a control that broke but a component that was added without one.
And access review over the exhaust closes the loop that is most often left open. Periodically ask who read production traces, how often, and whether their role justified it. The answer is frequently uncomfortable and always informative.
Failure modes worth naming
The prototype that inherited consumer terms. Built quickly against a free API tier, moved to production without revisiting the contract, and now regulated data is flowing under terms that permit training on it. Cheap to prevent at design time, expensive to discover during an audit.
The helpful summary. An agent asked to summarize a case includes a detail from a source the recipient was not entitled to see. Every read was authorized for the agent; the disclosure was to the human. Output policy has to be evaluated against the recipient's permissions, not the agent's.
The cache without a tenant key. Intermittent cross-tenant leakage that testing will not reproduce and that becomes obvious only when a customer reports seeing someone else's data. Wrap the cache so the key cannot be constructed without a tenant identifier.
The trace store nobody classified. Complete prompts retained for months in a third-party service, readable by the whole engineering organization, outside every access control the underlying data has. Provisioned as infrastructure, never reviewed as a data store.
The memory that became a profile. Started as continuity, accumulated into an unreviewed behavioural dossier assembled largely from inferences, with no expiry and no way for the subject to see or correct it.
The deletion that half worked. The primary record is gone and the embedding remains, so retrieval still surfaces the content and the agent still answers questions about a person who exercised their right to erasure. Reported as complete because no store was asked to confirm.
The redaction that created false confidence. A detection layer catches names and card numbers, the team declares the output anonymous, and quasi-identifiers walk straight through. The exposure is modest; the decisions licensed by the false confidence are not.
What to build first
The sequence matters, because privacy work has a way of expanding until it blocks everything, and the goal is a system that ships with defensible properties rather than a programme that never converges.
Start with the map. One page per workflow describing what data is touched, where it goes, how long it persists, and who can read it. Nothing else can be prioritized sensibly without it, and producing it usually surfaces two or three problems that were already live.
Then fix the exhaust, because it is almost always the largest exposure relative to its value. Turn off verbatim content capture by default, set a short retention on what remains, and put access control on the trace store. This is a week of work and it typically removes more risk than anything else on the list.
Then scope retrieval. Declare, per workflow, which sources are in scope, and enforce it at the index rather than in a filter. This converts an open-ended exposure into a bounded one and makes every subsequent control easier to reason about.
Then classify tools by destination and put scrubbing rules on the ones that cross a boundary. Then build the deletion registry, before you need it. Then add leakage tests to the eval suite so that regressions are caught by the same machinery that catches everything else.
Redaction infrastructure comes later than most teams expect, because it is the most visible control and the least load-bearing. A system with scoped retrieval, classified tools, short retention, and a working deletion path has a defensible privacy posture with no redaction at all. A system with excellent redaction and none of the rest has a filter in front of an open door.
The bottom line
The privacy questions raised by agents are not new questions. Minimization, purpose limitation, retention, and the rights of the people whose data you hold have been the substance of the field for decades. What agents change is the speed and the invisibility. Composition that used to require a person with a spreadsheet and an afternoon now happens in two seconds inside a context window, and the artifact that made it possible is deleted before anyone could have inspected it.
That is why the controls have to move upstream. You cannot review an aggregate that exists for two seconds, so you have to constrain what can be aggregated. You cannot audit a decision whose inputs were discarded, so you have to record the shape of what was assembled even when you do not record its content. You cannot ask a model to exercise judgement about sensitivity, so you have to make the unsafe path structurally unavailable rather than merely discouraged.
The test is not whether your system passed a review before launch. It is whether you can answer, for any interaction it had last week, what data it touched, why it was entitled to touch it, where that data went, and what remains of it now. Teams that can answer those four questions have built something they can defend and, more importantly, something they can change without fear. Teams that cannot are operating a system whose behaviour they will only understand retrospectively, at a moment of someone else's choosing.