Long-Running Agents: Designing AI Systems That Work While You're Away

An agent that answers in a single turn is a conversation: you ask, it thinks for a few seconds, it replies, and the interaction is over before you have looked away from the screen. An agent that runs for an hour, a day, or a week is a different kind of thing entirely, because the assumption that quietly shapes almost every piece of agent tooling — that a human is watching the whole time, that the process will still be alive when it finishes, that nothing important will change in the world between the first tool call and the last — stops being true the moment the work outlives the attention span of the person who started it. Long-running agents are not chat agents given more time; they are systems that must survive their own duration, and surviving duration turns out to demand an entirely different set of engineering habits than producing a good single response ever did.

This shift matters more every year because the tasks worth delegating to an agent are increasingly the ones that do not fit inside a single sitting. Researching a market over hundreds of sources, migrating a codebase file by file, monitoring a system and responding to what it finds over days, drafting and revising a document across many rounds of feedback — these are jobs whose value comes precisely from their length, and an agent that can only operate inside the attention span of a human hovering over a chat window cannot touch them. The interesting frontier of agentic AI is not making the single turn smarter; it is making the work durable enough to keep going after the human has closed the laptop, and durability is a property you have to design for, not one that falls out of a capable model.

The synchronous assumption and why it breaks

Most agent architectures are built, often without anyone deciding this explicitly, around a synchronous assumption: the process that started the task is the process that will finish it, the state of the task lives in that process's memory, and if the process dies the task simply did not happen. This assumption is invisible in a chat agent because the whole exchange takes seconds and the odds of a crash inside that window are vanishingly small, so nobody notices that the architecture never planned for one. Stretch the same architecture to a task that runs for six hours and the odds of a crash, a deploy, a network blip, or a machine restart happening somewhere in the middle stop being vanishingly small and start being close to certain, and an architecture that never planned for interruption meets an environment that guarantees it.

The break is not merely about crashes; it is about everything that changes while the agent is running that the agent's original picture of the world did not account for. A synchronous agent forms its plan against the state of the world at the moment it starts and executes that plan in the next few seconds, so the world it acts on is, for all practical purposes, the world it assessed. A long-running agent forms a plan against the state of the world at hour zero and may still be executing steps of that plan at hour twelve, by which point the file it read has been edited by someone else, the price it quoted has moved, the ticket it was asked to close has already been closed by a colleague. Time itself becomes an adversary the synchronous agent never had to face, and pretending otherwise is how long-running agents produce results that were correct when planned and wrong when delivered.

What changes when the loop runs for hours instead of seconds

The core agent loop — observe, decide, act, repeat — does not change shape when it runs for a long time, but nearly everything around it does, and it is worth being explicit about what. The state that used to live safely in a single process's memory now has to live somewhere that outlives the process, because no single machine is guaranteed to stay up for the task's whole duration. The cost of redoing work, which used to be trivial because there was so little work to redo, becomes a real design constraint, because redoing twelve hours of tool calls from scratch after a failure at hour eleven is not a rounding error, it is most of the job. And the question of what the agent should do if nobody is watching right now — because for a long-running agent, nobody usually is — moves from an edge case to the default case that the whole system has to be built around.

None of this is exotic engineering; it is the same set of concerns that distributed systems have wrestled with for decades under names like durability, idempotency, and fault tolerance, applied now to a component whose behavior is generated by a model rather than by hand-written logic. The novelty is not the problem, which is old, but the fact that the component in the middle of it reasons in ways you cannot fully predict, which makes the discipline more necessary rather than less. A long-running agent that has not been built with these concerns in mind will appear to work in every demo, because demos are short, and will fail in exactly the way a synchronous system with no persistence, no idempotency, and no fault tolerance always fails once it meets a duration long enough for something to go wrong — which for a long enough task is not a matter of if but when.

Checkpointing: making state survive the process that made it

Figure 1. A checkpoint captures enough state to resume without repeating already-completed work.

The first and most fundamental adaptation a long-running agent needs is checkpointing: writing enough of its state to durable storage, at regular intervals, that a fresh process can pick up roughly where the last one left off rather than starting over. A checkpoint is not the whole of the agent's memory, which would be wasteful to persist continuously, but the minimum that matters — what step it has reached, what it has already learned, what it has already done, and what remains — captured often enough that losing the process between two checkpoints loses only the small sliver of work done since the last one rather than the whole task. The value of a checkpoint is measured precisely by how little is lost when everything after it disappears.

Deciding what belongs in a checkpoint is a genuine design problem, because too little and a resumed agent repeats expensive work or loses track of decisions it already made; too much and every checkpoint becomes slow to write and unwieldy to load, which discourages taking them often enough to matter. The useful checkpoint captures the agent's plan and its position within that plan, the results of steps already completed that would be costly to redo, and any external commitments already made — an email already sent, a record already updated — that must never be repeated. It deliberately omits the ephemeral scratch reasoning that led to those results, because that reasoning was a means to an end already reached, and reconstructing the end from the checkpoint is cheaper than replaying the means.

The deeper discipline is choosing when to checkpoint, and the right answer is almost always after any action that is expensive to redo or impossible to undo, rather than on a fixed timer that has no relationship to what the agent has actually done. A timer-based checkpoint might fire right before an expensive step completes, losing exactly the work you most wanted to keep, while a checkpoint tied to completed, consequential steps guarantees that the boundary of "already done" is always exactly where the last checkpoint says it is. Checkpointing well is less about the mechanics of writing to storage, which are simple, and more about knowing which moments in the agent's progress are worth freezing, which requires understanding the task's own structure rather than treating every agent the same way.

Idempotency: retrying without redoing damage

Checkpointing solves half the problem of interruption — resuming from a reasonable point — but the other half is what happens to the step that was in flight when the interruption occurred, because that step's effect on the world is now ambiguous. Did the email send before the crash or after? Did the record get written or not? An agent that resumes from its last checkpoint and simply repeats the in-flight step, without knowing whether that step already partially happened, risks sending the email twice, charging the payment twice, or writing the record twice, and each of these is a failure that checkpointing alone does nothing to prevent. The fix is idempotency: designing every consequential action so that performing it twice has the same effect as performing it once, which turns "did this already happen" from a question the agent must answer perfectly into a question it no longer needs to ask.

Idempotency is usually achieved by attaching a stable identifier to each action before it is attempted — a request ID generated once and reused on every retry — so that whatever system receives the action can recognize a duplicate and discard it rather than executing it again. An email send tagged with an idempotency key lets the mail system notice it has already sent that exact key and quietly no-op the retry rather than delivering the message twice; a payment tagged the same way lets the payment processor apply the charge once no matter how many times the agent, uncertain whether the first attempt succeeded, asks it to happen. The identifier does the remembering that the crashed process no longer can, moving the burden of exactly-once behavior from the fragile, interruptible agent to the durable system on the other end of the call.

Not every action can be made naturally idempotent, and for those the fallback is to make the uncertainty itself visible rather than silently guessing. An action with no idempotency key and no way to check whether it already happened should, on resume, be treated as unknown rather than as safely repeatable, which might mean querying for evidence the action already occurred, logging the ambiguity for a human to resolve, or accepting the small risk of a duplicate only where a duplicate is cheap. The goal is never to eliminate every possible ambiguity, which is not always achievable, but to know precisely which actions in the agent's repertoire are safe to retry blindly and which are not, and to design the risky ones with enough care — idempotency keys, existence checks, explicit uncertainty handling — that a retry never becomes a silent second occurrence of something that was only supposed to happen once.

Crash recovery and resumability

Figure 2. Recovery replays only the gap between the last checkpoint and the interruption, not the whole task.

Checkpointing and idempotency together make crash recovery possible, but recovery is its own discipline, because resuming correctly requires more than having the data — it requires a process that knows how to reconstruct intent from a checkpoint rather than merely replaying whatever is written there. A recovering agent must reconstruct not just what it had done but what it was in the middle of trying to do, so that it resumes toward the same goal with the same plan rather than treating the checkpoint as a fresh, contextless starting point that happens to have some accumulated state attached. Recovery that loses the why behind the what tends to produce agents that technically continue but drift from the original intent, finishing a task that only superficially resembles the one that was interrupted.

The practical shape of good recovery is a clear separation between the durable record of what has already happened and the transient process of figuring out what to do next, so that the second can be thrown away and rebuilt cheaply while the first never is. On restart, the agent loads its last checkpoint, treats everything in it as settled fact rather than as something to reconsider, and re-derives its next action from that settled state using the same reasoning it would use at any other point in the loop. This makes recovery indistinguishable, from the agent's own perspective, from simply continuing — there is no special "recovery mode" with its own fragile logic, only the ordinary loop applied to a state that happens to have been reloaded from storage rather than held continuously in memory, which is exactly the property that makes the recovery path as trustworthy as the normal path, because it is the normal path.

The failure to design for this is usually invisible until the first real crash, which is precisely the wrong time to discover it, because a recovery path that has never been exercised is a recovery path you only hope works. The remedy is to treat restart as a routine event rather than a rare emergency — to kill the agent process deliberately during testing, at various points in a long task, and confirm that what resumes is a faithful continuation rather than a confused or duplicated one. An agent whose recovery has been exercised dozens of times in testing before it is ever needed in production is an agent whose durability you can actually trust, in the same way a backup you have never restored from is not really a backup, only a hope wearing its name.

Scheduling and event-driven wake-ups

Not every long-running task needs a process sitting continuously alive for its whole duration, and for a large class of work the better architecture is one where the agent does not run at all between the moments it has something to do, waking only when a schedule or an event calls for it. A monitoring agent that checks a system every morning does not need to occupy a process for the twenty-four hours between checks; it needs to be invoked once, do its work, and disappear until the next invocation, and building it this way is both cheaper and more robust than keeping a process alive and idle, because an idle process is still a process that can crash, leak memory, or drift out of sync with a world that kept moving while it sat there doing nothing.

This scheduled or event-driven shape trades the question "how do I keep this process alive for a long time" for the arguably easier question "what does this agent need to know when it wakes up," and the second question has a cleaner answer: everything it needs is whatever was true the last time it ran, plus whatever has changed since, which is exactly what a good checkpoint captures. An agent invoked daily does not need continuous memory of yesterday; it needs yesterday's checkpoint and today's fresh look at the world, and reconstructing continuity from those two things, rather than from an unbroken chain of process uptime, is both simpler to build and far more resilient to the outages, redeploys, and infrastructure changes that an always-on process would have to survive unscathed.

Event-driven wake-ups extend the same idea to tasks that should not wait for a fixed schedule at all but should react promptly to something happening — a new support ticket, a file landing in a folder, a webhook firing — and the discipline here is to keep the triggering mechanism completely separate from the agent's own reasoning, so that "when do I run" is answered by infrastructure built for exactly that question rather than by the agent polling in a loop and burning time and money watching for something that might not happen for hours. An agent that is invoked by an event arrives with a clear, bounded reason to act, does its work, checkpoints or finishes, and exits, and this rhythm of invoke-act-exit, repeated as many times as the world produces triggers, is a far sturdier foundation for long-horizon work than a single process trying to stay alive and attentive indefinitely.

The context window over long horizons

A model's context window is generous by the standards of a single conversation and hopelessly small by the standards of a task that runs for days, and a long-running agent that simply appends everything it has seen and done to a growing context will eventually hit a wall no amount of patience will move, because the window has a fixed size and the history of a long task does not. The naive fix — truncate the oldest content when the window fills up — throws away exactly the information that gave the early part of the task its purpose, which is usually the worst possible thing to lose, because an agent that forgets why it started rarely does the right thing with what remains.

The sturdier answer is active compaction: periodically pausing to compress what has happened so far into a smaller, denser summary that preserves the decisions, commitments, and open threads while discarding the raw, moment-to-moment detail that produced them. Done well, compaction is not merely deletion but distillation, turning forty tool calls and their outputs into a few sentences that capture what those forty calls established, so that the agent's context stays bounded while its effective memory of the task's arc does not shrink at all. The skill is compacting aggressively enough that the context never overflows, while compacting carefully enough that nothing load-bearing is lost in the compression — a balance that has to be tuned to the task rather than applied uniformly, because a task with many small, low-stakes steps can tolerate lossier compaction than one with a few decisions that everything else depends on.

The relationship between compaction and checkpointing is close enough that the two are worth designing together rather than separately, because a checkpoint is, in a sense, the most extreme form of compaction — the version of the agent's state small enough to persist and reload cheaply, stripped of everything except what recovery actually needs. A long-running agent that has thought carefully about what belongs in a checkpoint has already done most of the thinking needed to compact its working context well, and the two disciplines reinforce each other: better compaction produces leaner, more meaningful checkpoints, and the exercise of writing good checkpoints sharpens the sense of what is actually essential to keep.

Asynchronous human check-ins

Figure 3. Asynchronous approval queues a decision for a person instead of blocking the agent's process on their reply.

An approval gate in a synchronous agent blocks the process and waits, because the human is right there and the wait is measured in seconds; a long-running agent cannot afford that pattern, because the human is not there, may not check for hours, and a process that blocks on a human reply for that long is a process burning resources to do nothing, or worse, a process that a restart or timeout will kill before the reply ever arrives. The asynchronous version of the same idea — pause, surface the decision, and wait for a person — has to be re-architected so that pausing does not mean an idle process holding its breath but a checkpoint written, the process exited, and a notification sent, with the resumption triggered by the human's eventual response rather than by the same process sitting and waiting for it.

This reframing turns a human check-in into just another event the agent wakes up for, no different in kind from a scheduled tick or an incoming webhook, which is a considerable simplification once you see it that way: the agent does not need a special mechanism for "waiting on a person" distinct from its general mechanism for "resuming when something happens," because a person's approval is simply one more something that can happen. What it does need is a clear, durable record of exactly what was being asked and why, attached to the checkpoint at the moment of pausing, so that whether the reply comes back in five minutes or five days, the resumed agent can reconstruct the question it asked and interpret the answer correctly rather than receiving a bare "yes" disconnected from the context that made it meaningful.

The design lesson that carries over from synchronous approval gates is unchanged even as the mechanics differ: the check-in is only as good as what it shows the person, and an asynchronous notification that says only "approval needed" without conveying what the agent intends to do and why gives the person nothing real to evaluate when they eventually do look. Because the asynchronous version may sit unanswered far longer than a synchronous prompt ever would, the cost of an unclear request is higher, not lower — the person has had a coffee, moved on to three other things, and returned to a notification they must now reconstruct the context for entirely from what it says, so the discipline of writing check-ins that stand on their own, comprehensible without the asker present to clarify, matters more for long-running agents than it ever did for one that could just ask a follow-up question in the same breath.

Observability across time, not just across a trace

Ordinary agent observability is built around the trace — the sequence of steps, tool calls, and outputs within a single run — and that shape works well when a run takes seconds and the question is simply what happened in this one execution. A long-running agent's observability problem is different in kind, because the run is not one short trace but a long, possibly interrupted, possibly resumed sequence spanning many separate process lifetimes, and the question a person actually wants answered is rarely "what happened in this particular process" but "where is this task right now, and is it on track," which a stack of disconnected per-process traces does not answer on its own.

What a long-running agent needs, on top of ordinary tracing, is a persistent notion of progress: a way to say, at any moment, how far through the task the agent has gotten, what it has already accomplished, what remains, and whether its rate of progress suggests it will finish in a reasonable time or has stalled. This is closer to a project's status report than to a debugger's stack trace, and building it requires the agent, or the system around it, to maintain a running account of progress that survives every restart the same way the checkpoint does — indeed the progress account and the checkpoint are natural companions, since a checkpoint that records what has been done is most of what a progress report needs to say as well.

The payoff of this temporal observability is that it turns "is my long-running agent okay" from a question you can only answer by reading raw logs after something has clearly already gone wrong into one you can answer at a glance at any point during the run, which matters enormously for a task nobody is watching continuously. A dashboard that shows a task's declared progress against its expected pace lets a person notice, hours before the deadline, that something has stalled or drifted, rather than discovering only at the end that the twelve hours produced nothing usable — and noticing early, while there is still time to intervene, is most of the value that observability was ever supposed to provide.

Failure modes unique to long horizons

Long-running agents fail in ways that short-running agents structurally cannot, because failure needs time to accumulate and a task measured in seconds never gives it the chance. Drift is the clearest example: small deviations between the agent's model of the task and the task's actual current state, each individually harmless, that compound over a long enough run into a final result meaningfully disconnected from what was actually needed, even though every individual step along the way looked reasonable in isolation. A synchronous agent cannot drift because it never has enough steps between it and its answer for small errors to compound; a long-running agent drifts by default unless something periodically re-anchors it to ground truth rather than letting it extrapolate indefinitely from an increasingly stale starting picture.

Staleness is drift's cousin, and it is specifically about the world outside the agent changing while the agent's picture of that world does not. An agent that read a document, a price, or a status at hour zero and continues to act on that reading at hour ten is acting on information that may no longer be true, and the failure here is not a mistake in the agent's reasoning at all but a mistake in trusting an input whose shelf life has quietly expired. The remedy is to treat any externally sourced fact the agent relies on as having a freshness requirement proportional to how long the task runs and how quickly that fact changes in the world, and to re-fetch or re-verify it rather than assuming that what was true at the start is still true now — a discipline that a short-running agent never needs because its whole task fits inside the freshness window of everything it reads.

Goal decay is the subtlest of the three, and the hardest to catch, because it is a failure not of information but of purpose: over a long enough run, especially one involving many intermediate sub-decisions, an agent can end up optimizing for a proxy of the original goal that has quietly become disconnected from what the goal actually was, satisfying the letter of an instruction it set for itself three hours ago while missing the spirit of what the task was for in the first place. The defense against goal decay is to periodically re-ground the agent in the original, top-level objective — not the sub-goal it derived from that objective an hour ago, but the objective itself — so that its trajectory is checked against the actual destination at intervals, rather than only against the last waypoint it set for itself, which is precisely the kind of check that a short task never needs because it never travels far enough from its starting instructions to wander off course.

Concurrency and shared resources

A single long-running agent raises hard enough questions on its own, and a system that runs many of them at once, potentially touching the same files, the same records, or the same external accounts, raises a second, orthogonal set of problems that only appear once concurrency enters the picture. Two agents each reading a value, computing an update based on it, and writing the result back can each individually behave correctly while their combined effect corrupts the value, not because either agent reasoned incorrectly but because neither knew the other existed, and this is a failure mode that has nothing to do with model quality and everything to do with the absence of coordination between processes that happen to act on the same piece of the world.

The tools for this are, again, borrowed from decades of distributed-systems practice rather than invented fresh for agents: locking a resource for the duration of an update so a second agent cannot act on a stale read of it, using optimistic concurrency checks that detect and reject a write based on data that has since changed underneath it, or partitioning work up front so that no two agents are ever assigned overlapping resources in the first place. Which of these fits depends on how often conflicts actually occur and how expensive a conflict is when it does — locking suits frequent contention over precious resources, optimistic checks suit rare contention where retrying is cheap, and partitioning suits work that can be divided cleanly enough that contention is designed out rather than merely managed.

The organizational discipline underneath all of these techniques is simply naming, explicitly, which resources any given agent is allowed to touch and ensuring that claim is visible to every other agent that might also want to touch it, because the alternative — agents silently assuming they have exclusive access to something they were never actually promised — is how uncoordinated concurrent systems produce the exact corruption that careful individual agents were supposed to prevent. A long-running agent that declares its resource claims as part of its checkpointed state gives the rest of the system something concrete to coordinate against, turning an implicit and fragile assumption of exclusivity into an explicit and checkable one.

Testing long-running agents: replay and time travel

Figure 4. Replay reconstructs a specific past state from the recorded log, so a failure hours in can be re-examined directly.

Testing an agent that finishes in seconds is straightforward: run it, inspect the output, repeat. Testing an agent that takes many hours to reach the state where a bug appears is a fundamentally harder problem, because naively rerunning the whole task from scratch every time you want to inspect an issue that only shows up at hour nine turns every debugging cycle into another nine-hour wait, which is slow enough to make careful testing impractical and therefore, in practice, skipped — exactly the wrong incentive for the part of the system most in need of scrutiny.

The answer that makes long-running agents testable is the same durable, checkpointed record that makes them resumable in production: if every consequential step and its inputs are logged in enough detail to reconstruct the agent's state at any prior point, then reaching hour nine to investigate a bug does not require re-running the first nine hours, it requires loading the checkpoint or log entry closest to hour nine and resuming analysis from there directly. This is effectively time travel for debugging — the ability to jump to any recorded point in a long task's history and inspect or even re-execute from that exact state — and it turns a debugging cycle that would otherwise take as long as the original run into one that takes as long as it takes to load a checkpoint, which is a difference of hours versus seconds.

Building this capability is mostly a matter of taking logging seriously enough, during the original run, that the log is genuinely sufficient to reconstruct state rather than merely sufficient to explain what happened in prose after the fact — recording not just that a decision was made but the exact inputs that produced it, in a form precise enough to feed back into the agent and get a resumable state out. Investing in this before it is needed pays for itself many times over the life of a long-running system, because every long-running agent will eventually produce a bug that only manifests deep into a run, and the difference between a system built for replay and one that was not is the difference between diagnosing that bug in minutes and dreading every report of it because reproducing it costs the better part of a day.

The shape of the discipline

If there is a single idea underneath all of this, it is that a long-running agent has to be engineered to survive its own duration, because duration is exactly the dimension that a synchronous agent never had to contend with and that a long-running one cannot escape. State has to live somewhere durable enough to outlast any single process; actions have to be safe to retry because retries are not a possibility but a certainty over a long enough run; recovery has to be a routine path rather than a hoped-for one; context has to be actively compacted rather than left to overflow; human check-ins have to be asynchronous events rather than blocking waits; observability has to track progress over the whole arc of a task rather than just the inside of one trace; and the agent's own trajectory has to be periodically re-anchored against staleness, drift, and goal decay, because time is exactly what erodes each of those if nothing pushes back against it.

None of these ideas are new to computing, which is the reassuring part rather than the discouraging one — durability, idempotency, checkpointing, and replay are the vocabulary that distributed systems have used for a long time to make unreliable components add up to a reliable whole, and long-running agents are simply the latest kind of unreliable component that vocabulary needs to be applied to. The model at the center reasons in ways you cannot fully predict and will, over a long enough run, encounter a crash, a stale fact, a duplicate action, or a moment where nobody is watching, and the entire discipline described here exists to make sure that when it does, the system absorbs the disruption rather than being defined by it.

Built this way, a long-running agent becomes something you can actually hand a multi-day task and walk away from, trusting that a crash at hour six means a resumed process at hour six and not a task quietly lost, that a retried action means no duplicate rather than a second charge, and that the thing you left running will still be pointed at the goal you gave it when you finally check back in, rather than at some drifted proxy of it. That trust is the entire point: an agent that can survive its own duration is an agent you can actually leave alone with something that matters, and being trusted to work unattended is, in the end, the whole reason long-running agents are worth building at all.