Why one model for everything is the wrong default
The easiest way to build an agent is to pick the strongest model available and route every call to it: every tool selection, every summarization, every classification, every final answer. It is also, almost always, the wrong default once the agent leaves the demo and starts running at volume. An agent loop is not one decision, it is dozens or hundreds of small decisions strung together, and those decisions are not equally hard. Deciding whether a user's message is a billing question or a technical question is not the same problem as synthesizing a nuanced answer from five conflicting sources. Extracting a date from a paragraph is not the same problem as planning a multi-step research task. Treating them the same, by sending them all to the same frontier model, is like hiring a senior architect to also staff the reception desk: the work gets done, but at a price and a pace that make no sense for what the task actually required.
Model routing is the practice of deciding, per call, which model should handle it. Done well, it is close to a free lunch: you spend less, you respond faster, and — this is the part people find counterintuitive — you can often end up more reliable, not less, because small, fast, cheap models tuned or prompted for a narrow task frequently outperform a generalist frontier model on that narrow task, and because the money and latency you save on easy steps can be reinvested in the steps that actually need a bigger model. The goal of this piece is to make routing a deliberate engineering decision rather than an afterthought bolted on when the bill arrives.
It helps to say up front what routing is not. It is not the same as model selection at build time, where you pick one model for your whole application and move on. It is not fine-tuning, though the two pair well. And it is not a way to avoid using a strong model at all — a well-routed system still sends its hardest problems to its best model, it just stops sending everything else there too. The point of routing is precision in spending, not austerity.
Routing is not about using worse models. It is about not paying frontier prices for problems that do not require a frontier model.
What actually varies between steps
To route well you first have to see the variation that a single-model design hides. Inside almost any agent loop, calls differ along at least four axes. There is task difficulty: some steps require multi-step reasoning over ambiguous, conflicting information, while others are simple classification, extraction, or formatting that a small model handles correctly nearly every time. There is stakes: a wrong answer in an internal scratchpad step costs nothing, while a wrong answer in a customer-facing summary or a tool argument that triggers a side effect can cost real money or trust. There is latency sensitivity: a step a user is actively waiting on has a different budget than a step running in the background of a long, asynchronous task. And there is volume: a step that runs once per task looks very different, cost-wise, from a step that runs in a tight loop hundreds of times per task, where even small per-call savings compound fast.
A useful exercise, before you write a single line of routing logic, is to actually enumerate the distinct call sites in your agent and score each one on these four axes. Most teams have never done this, and the exercise alone is often revealing: it is common to find that three or four call sites account for the overwhelming majority of both cost and error rate, and that a large fraction of remaining calls are trivially easy, high-volume, and low-stakes — exactly the calls that should never have been going to your most expensive model in the first place. You do not need a sophisticated router to capture most of the available savings; you need an honest inventory of what your agent is actually asking its model to do at each step.
It is worth being specific about the kinds of steps that are almost always over-served by a frontier model. Intent classification — is this message a question, a command, a complaint — is usually solvable by a small model with high accuracy. Structured extraction from clean text, pulling a date, a name, or an amount out of a paragraph, rarely benefits from extra reasoning capacity. Format conversion and light rewriting, turning a bullet list into prose or vice versa, is mechanical. Tool argument construction for a well-specified tool with a small number of parameters is often closer to a lookup than a reasoning task. None of these need your best model. What does need it: open-ended synthesis across long or contradictory context, planning under genuine ambiguity, code that has to be correct on a codebase the model has not seen before, and anything where a subtle wrong answer is expensive and hard to catch downstream.
The three levers: cost, latency, and capability
Every routing decision is, underneath, a trade among three quantities. Cost is the most visible lever and the one people reach for first, because it shows up on an invoice: larger, more capable models cost meaningfully more per token, often by an order of magnitude or more between the smallest and largest models in a given family. Latency is the lever users feel directly: bigger models are slower to respond, and that gap widens further when a task chains many calls together, because latency compounds across a loop in a way that cost merely adds. Capability is the lever that is hardest to quantify but matters most: it is the model's actual ability to get the specific task right, and it does not move in a straight line with size or price — a small model tuned or prompted well for a narrow task can beat a much larger generalist on that task, while the same small model can fail badly on a task just outside its comfort zone.
The mistake to avoid is optimizing any one of these levers in isolation. A router that minimizes cost by always picking the cheapest model will eventually produce a wrong answer on a task that needed real reasoning, and the downstream cost of that wrong answer — a bad tool call, a support ticket, a user who loses trust — usually dwarfs the tokens you saved. A router that minimizes latency by always picking the fastest model has the same problem in a different shape. A router that always reaches for the most capable model regardless of need solves the accuracy problem while reintroducing the exact cost and latency problem routing exists to fix. The three levers have to be reasoned about jointly, per call site, against the actual consequences of getting that call wrong.
A practical way to think about this jointly is to ask, for each call site, what does a wrong answer here actually cost, and how would I know if it happened? Those two questions, more than any formula, determine where on the cost-latency-capability triangle a given step should sit. A step where a wrong answer is invisible and cheap to correct later can tolerate a fast, cheap, occasionally-wrong model. A step where a wrong answer is expensive, hard to detect, and hard to undo needs either your most capable model or a cheap model plus a verification layer that catches the cases it gets wrong — which brings us to cascades.
.jpg)
Cascades: try cheap, escalate on doubt
The single most useful routing pattern is the cascade: send a call to a cheap, fast model first, and escalate to a more capable model only when the cheap model's response looks uncertain, fails a check, or is explicitly flagged as out of its depth. Done well, a cascade captures most of the cost and latency benefit of always using a cheap model while capturing most of the accuracy benefit of always using an expensive one, because the expensive model is only invoked on the fraction of calls that actually need it. In a well-tuned cascade over a realistic workload, it is common for eighty to ninety percent of calls to resolve at the cheap tier, with only the genuinely hard tail escalating.
The mechanics are straightforward to describe and take real effort to tune well. First, send the call to the cheapest model in your ladder that has a reasonable chance of getting it right. Second, evaluate the response against some check: a confidence signal from the model itself, a structural validator that confirms the output matches an expected schema, a consistency check against other information the agent has, or in some designs a cheap second model whose only job is to judge the first model's answer. Third, if the check passes, use the response and move on; if it fails, escalate the same call to the next model up the ladder and repeat. The ladder can have two rungs or several, but two or three is enough for the overwhelming majority of applications — a cheap tier, a mid tier, and a top tier reserved for the calls that fail both of the cheaper attempts.
The part teams get wrong is the escalation trigger. If the check that decides whether to escalate is too permissive, you accept wrong answers from the cheap model that a slightly more careful check would have caught, and the cascade quietly degrades your overall accuracy in exchange for savings you did not need to make that badly. If the check is too strict, nearly everything escalates, the cheap tier does no real work, and you have built an elaborate way to route everything to your expensive model anyway, paying for two calls instead of one on the majority of requests. Tuning the escalation threshold against a labeled evaluation set, not against intuition, is the difference between a cascade that pays for itself and one that quietly costs more than routing everything to a single mid-tier model would have.
A cascade is only as good as its escalation trigger. Too loose and you ship wrong answers cheaply; too tight and you pay for two models on every call.
How do you know the cheap model failed? Confidence signals
Escalation only works if you have a reasonably reliable way to tell that the cheap model's answer needs a second look, and this turns out to be a harder problem than it first appears, because raw model confidence — a model asserting how sure it is — is notoriously unreliable and tends to be miscalibrated, often confidently wrong in exactly the cases where escalation matters most. A handful of more trustworthy signals exist and are worth using in combination rather than any one in isolation.
Structural validity is the cheapest and most reliable signal: if the call expects a JSON object matching a schema, a number in a range, or a value from an enumerated set, and the response fails to parse or violates the constraint, that is unambiguous evidence the call needs escalation, no judgment call required. Log-probability-based signals, where the API exposes token-level probabilities, can flag cases where the model was genuinely uncertain about its own output even when the output is well-formed, though not every provider exposes this and it needs calibration against your own data before you trust it. Self-consistency, running the same prompt more than once and checking whether the answers agree, is a strong but expensive signal — a model that gives three different answers to the same question three times is telling you something a single confident-sounding answer would have hidden, but the extra calls eat into the savings a cascade is trying to capture, so this is best reserved for calls where the stakes justify the cost. And a cheap judge model, a small model whose only job is to look at the input and the candidate output and flag whether it looks right, can catch errors a self-report would miss, particularly for tasks with checkable structure like tool arguments or extracted facts.
The practical recommendation is to layer these rather than pick one: use structural validation as a hard gate that always escalates on failure, add a lightweight judge or consistency check for calls above a stakes threshold, and treat raw self-reported confidence as a weak signal to combine with the others rather than trust on its own. None of these signals is perfect, and the honest goal is not a router that never escalates incorrectly, it is a router whose escalation errors are rare enough and cheap enough that the system as a whole is clearly better off than either extreme of always-cheap or always-expensive.
Static routing rules vs. learned routers
Once you have decided what varies between call sites and how to detect when a cheap model has failed, you need a mechanism that actually assigns a model to each call. The simplest version is a static rule table: a mapping, written by a human, from call type to model — classification calls go to the small model, synthesis calls go to the large model, tool-argument construction for a specific well-understood tool goes to the small model, and so on. Static rules are transparent, easy to debug, and cost nothing to build beyond the initial analysis, and for a large share of applications they capture nearly all the available benefit, because the variation between call sites is usually more about call type than about the specific content of any individual call.
A more adaptive approach uses a learned router: a small, fast classifier, sometimes a small language model itself, trained or prompted to look at an incoming request and predict which tier of model it needs. This buys you routing decisions that respond to the actual difficulty of a specific input rather than just its category — two customer messages that both look like billing questions on the surface can differ enormously in how hard they actually are to resolve, and a learned router has a chance of telling them apart where a static rule keyed only on message type cannot. The cost is real, though: a learned router needs training data, ongoing evaluation, and its own maintenance burden, and a badly calibrated learned router can misroute in ways that are much harder to debug than a static rule that is simply wrong for one call type.
The sensible default is to start with static rules, because they get you most of the win with almost none of the complexity, and to introduce a learned router only for the specific call sites where you have evidence that difficulty varies meaningfully within a single call type and that variance matters enough to be worth capturing. Reaching for a learned router on day one, before you have data showing static rules are leaving real value on the table, is a common overinvestment — build the simple version, measure where it falls short, and add sophistication only where the measurement justifies it.
Start with a static rule table keyed on call type. Add a learned router only where you have evidence that difficulty varies within a call type enough to matter.
Building a capability ladder
Routing presupposes that you have more than one model to route between, arranged in some order of capability, cost, and latency — a ladder. Building a good ladder is less about chasing the newest model in every tier and more about knowing, concretely, what each rung is for. A typical ladder has three or four rungs. At the bottom sits a small, fast, cheap model for high-volume, low-difficulty work: classification, extraction, light formatting. In the middle sits a general-purpose mid-tier model that handles the bulk of moderately complex reasoning and tool use, the workhorse that most calls that are not trivial will land on. At the top sits your most capable model, reserved deliberately for the calls that are genuinely hard, high-stakes, or where earlier rungs have already failed. Some ladders add a specialized rung outside this cost-ordered hierarchy entirely: a model fine-tuned or specifically strong at one narrow task, like code generation or a particular extraction format, that may not be the cheapest or the most capable in general but is the best tool for that one job.
It is worth resisting the temptation to build the ladder around brand-new models the moment they release, before you have evaluated where they actually sit relative to what you already use. A newer model that is marketed as an upgrade to your mid tier is not automatically a drop-in replacement; it may be stronger on some tasks and weaker on others relative to your specific workload, and swapping it in without re-running your evaluation suite against your own call types is how routing ladders quietly degrade over time. Treat the ladder itself as something you evaluate and revise periodically against real traffic, not as a fixed decision you make once. Providers ship new models on a cadence measured in months, and a ladder built a year ago is worth re-checking against what is available now, both for cost and capability shifts.
Also worth naming: the ladder does not have to come from a single provider. Nothing about routing requires that your cheap tier and your top tier come from the same model family, and in practice the strongest ladders often mix providers, choosing whichever model is genuinely best and cheapest at each rung rather than staying loyal to one vendor's lineup out of convenience. This adds integration overhead — different APIs, different prompt formats, different quirks — but for a system where routing is doing real work, that overhead is usually worth paying.
Caching as the zeroth tier
Before any model is invoked at all, there is a cheaper option than even your smallest model: not calling a model. Caching, in its various forms, is best understood as the zeroth rung of the routing ladder, the tier you check before you route to any model at all. Exact-match caching, where an identical input to a call site returns a stored prior output, is nearly free to implement and captures real savings in any workload with repeated or templated requests — the same classification question asked in slightly different words, the same tool call issued repeatedly against a stable resource, the same summarization of a document that has not changed since it was last summarized.
Semantic caching goes further, using an embedding similarity check to catch inputs that are not identical but close enough that a prior answer is still valid, which captures a larger share of traffic at the cost of needing a similarity threshold tuned carefully enough that it does not serve a stale or wrong answer to a request that only looks similar on the surface. Prompt caching, a feature many model providers now offer natively, caches the processing of a long, repeated prefix — a system prompt, a large tool schema, a big block of reference context — so that only the varying suffix of each call incurs full cost, which matters enormously for agents whose calls share a large, stable context across many turns.ssing of a long, repeated prefix — a system prompt, a large tool schema, a big block of reference context — so that only the varying suffix of each call incurs full cost, which matters enormously for agents whose calls share a large, stable context across many turns.
The reason to treat caching as part of routing rather than a separate concern is that it changes the actual traffic your router sees. A call site that looks expensive when you total up every model invocation can look completely different once you realize that sixty percent of its traffic is cache hits that never reach a model at all. Measure cache hit rates before you tune model tiers, because the routing problem you think you have — this call site is expensive — is sometimes actually a caching problem in disguise, solvable without touching which model handles the calls that do go through.
Latency budgets and user-facing vs. background steps
Cost is not the only thing routing should optimize for, and in user-facing applications it is often not even the primary thing. A step a person is actively watching, waiting for a response to render, has a hard latency budget that a step running quietly in the background of an asynchronous task does not. Routing should treat these differently even when the underlying task difficulty is identical, because the cost of being slow is not symmetric: a slow background step delays a result nobody is staring at, while a slow foreground step is a user watching a spinner, and users have a well-documented tendency to abandon interactions that stall past a few seconds regardless of how good the eventual answer would have been.
A practical pattern is to assign each call site an explicit latency budget as part of its routing configuration, not just a target model. For foreground steps with tight budgets, that budget can itself be the deciding factor in model selection, overriding what a pure difficulty assessment would otherwise choose — sometimes the right call is a smaller, faster model that gives a good-enough answer within budget rather than a better model that answers correctly but late. For background steps, the budget can be generous enough that the router is free to optimize purely for accuracy and cost without latency as a binding constraint, which is exactly where a cascade's occasional escalation to a slow, expensive model does the least harm, because nobody is watching the clock.
Streaming is worth mentioning here because it changes the latency calculus rather than the routing decision itself: a model that streams its response as it generates can feel fast to a user even when the total generation time is unchanged, because the first tokens appear quickly and the perception of speed is driven more by time-to-first-token than by total completion time. For foreground steps where the full response is long, enabling streaming is often a cheaper win than downgrading the model, and the two are not mutually exclusive — a well-designed foreground path streams from whichever model the router selects, rather than treating streaming and routing as competing solutions to the same latency problem.
Fallback and degradation when your best model is unavailable
Routing logic tends to be designed around the happy path, where every model in the ladder is available, responsive, and within its rate limit. Production traffic does not cooperate with that assumption. Providers have outages, individual models get rate-limited under load, and a model your router assumed it could reach for a hard escalation is sometimes simply not there when the call happens. A routing system that has no answer for this failure mode will either hang, error out to the user, or — worse — silently degrade in a way nobody notices until a metric looks wrong days later.
The fix is to design fallback paths as explicitly as you design the primary ladder. If the top-tier model is unavailable, what is the next best option: a different provider's comparable model, a slightly weaker model with a note to flag the response for review, or a queued retry with backoff for calls where waiting is acceptable and correctness cannot be compromised? The right answer depends on the call's stakes, which is exactly the same analysis that informed the original routing decision, applied again under a failure condition rather than a normal one. A call whose failure mode is a user seeing a slightly worse but still reasonable answer can degrade gracefully to a lower rung; a call whose failure mode is an incorrect side effect, like a tool call that moves data or money, should fail closed and queue for retry or human attention rather than silently downgrade to a model less equipped to get it right.
It is worth explicitly testing these fallback paths rather than assuming they work because the code exists. Simulate a provider outage or a rate-limit response in a staging environment and confirm the system actually falls back the way you intended, because fallback logic that has never been exercised under a real failure is exactly the kind of code that turns out to have a bug the first time it matters, at the worst possible moment to discover it.
 - DSC06823.JPG)
Observability: logging routing decisions
You cannot improve a router you cannot see into, and routing decisions deserve the same seriousness as any other piece of agent observability: every call should log which model handled it, why — which rule or score triggered that choice — what it cost, how long it took, and whether it was escalated from a cheaper tier that failed a check. Without this, a routing system is a black box whose behavior you can only infer from an aggregate bill at the end of the month, which tells you that something changed without telling you what or why.
This log is the raw material for the two questions that matter most about a routing system over time: is it actually saving what you think it is saving, and is it costing accuracy anywhere you have not noticed? The first question is answered by comparing actual spend against a hypothetical baseline of routing everything to your top-tier model, broken down by call site, which tells you where the routing system is earning its keep and where it might not be worth the added complexity. The second question is harder and more important, and it requires periodically sampling escalated and non-escalated calls alike and checking the actual outputs against ground truth or human judgment, because a router can look like it is working — cache hit rates healthy, escalation rates stable, latency within budget — while quietly shipping more wrong answers than the single-model system it replaced, if the escalation trigger has drifted or a model in the ladder has degraded on a task it used to handle well.
A dashboard that shows routing distribution across tiers, cost per call site, and escalation rate over time turns this from an occasional audit into an ongoing signal. A sudden spike in escalation rate for one call type is often the first sign that either the traffic mix has shifted, a provider has changed a model's behavior underneath you, or a recent prompt change made the cheap tier's job harder than it used to be, and catching that spike in a dashboard is considerably cheaper than catching it in a wave of user complaints.
A router you cannot see into is a router you are trusting blindly. Log the model, the reason, the cost, and the escalation on every call, and audit outputs periodically, not just the aggregate metrics.
Failure modes and anti-patterns
A handful of recurring mistakes explain most of the trouble teams run into with model routing, and each has a fairly direct fix once named. The race-to-the-bottom router optimizes cost so aggressively that it silently ships wrong answers on calls that needed more capability, with nobody noticing until the error rate shows up somewhere downstream, disconnected from its actual cause; the fix is to tie every routing decision to a measured accuracy check, not just a cost target, and to treat accuracy regression as a routing bug with the same urgency as a cost regression. The thrashing router escalates and de-escalates the same call type repeatedly because its trigger is noisy or its threshold sits right at a boundary that ordinary variance crosses back and forth, burning the cost of multiple calls without a stable benefit from either tier; the fix is hysteresis, requiring a clearer margin before escalating and a clearer margin before trusting the cheap tier again, rather than a knife-edge threshold.
The stale ladder keeps routing to a model lineup that was current when it was built, missing months of provider releases that would have shifted the optimal assignment, sometimes paying premium prices for a rung that a newer, cheaper model would now fill just as well; the fix is a scheduled, recurring re-evaluation of the ladder against current models and current pricing, not a one-time decision. The invisible router has no logging beyond what is strictly needed to make the call, so when cost or quality drifts, nobody can tell why without reconstructing the routing logic from source code under time pressure; the fix is the observability layer described above, built in from the start rather than added after the first confusing incident. And the premature router builds an elaborate learned routing system before establishing that static rules were actually insufficient, paying real engineering cost for adaptiveness the workload did not need; the fix is the same discipline that applies to agent architecture generally — start simple, add complexity only where measurement shows the simple version falling short.
A reference routing architecture
Pulling the pieces together, a mature routing system tends to have a recognizable shape regardless of the specific application it serves. At the front, a cache layer checks for an exact or semantically similar prior answer before any model is invoked. If that misses, a classifier — a static rule keyed on call type in the common case, a lightweight learned router where the evidence justifies it — assigns the call to an initial tier based on its type, its stakes, and its latency budget. The call is dispatched to the model at that tier, and the response is checked against whatever validation applies: schema conformance, a confidence signal, a consistency check, or a cheap judge, depending on the stakes of the call. If the check passes, the response is used, the outcome is logged with the model, the cost, the latency, and the check result, and the call is done. If the check fails, the call escalates to the next tier up, repeating the same validation, up to a ceiling tier beyond which the system either accepts the best available answer with a flag for review or fails closed, depending on what the call's consequences demand. Running alongside all of this, independent of any single call, is the fallback layer watching for provider errors or rate limits and rerouting around them, and the observability layer aggregating every decision into the dashboards and audits that keep the whole system honest over time.
None of these pieces is exotic on its own, and a small application may fold several of them together into a much simpler implementation than this description implies. The value of laying out the reference shape is the same as it is for any other piece of agent architecture: it gives you a checklist for diagnosis when something goes wrong, cost is too high, latency is too slow, or accuracy has dipped, letting you ask which specific piece is responsible rather than treating the routing system as an opaque whole to be rebuilt from scratch every time something drifts.
The economics of a router
It is worth stepping back and being honest about when a routing system is worth building at all, because it is not free — it is an additional piece of infrastructure with its own bugs, its own maintenance burden, and its own failure modes, all of which this piece has spent considerable space describing. The case for routing gets stronger with volume, because savings and latency improvements multiply across every call, and weaker for genuinely low-volume applications where the fixed cost of building and maintaining a router may exceed what a full year of always-use-the-best-model spending would have cost. The case for routing also gets stronger the more heterogeneous your call sites are: an agent whose every call is roughly the same difficulty has little for a router to exploit, while an agent whose calls span trivial classification and genuinely hard synthesis has enormous room between the cost of serving both with one model and the cost of serving each appropriately.
A useful rule of thumb, before investing in a routing system, is to estimate the savings a simple two-tier cascade would produce on your current traffic, using nothing more than a spreadsheet: what fraction of calls plausibly belong on a cheap tier, what does the cheap tier cost relative to what you pay now, and what escalation rate would you expect. If that back-of-envelope number is small relative to your total spend, routing is a nice-to-have you can defer; if it is large, and for most agents handling any real volume of heterogeneous calls it is, then the router pays for its own engineering cost quickly and keeps paying after that. The honest failure mode to watch for is building a routing system as a point of engineering pride rather than in response to a measured opportunity — check the number first, build second.
Where this is going
Model routing sits at an interesting point in how agentic systems are built, because it is one of the few pieces of agent infrastructure that gets easier, not harder, as the underlying models improve. Every new generation of small models closes some of the gap with the previous generation's large models, which pushes more call types down the ladder over time without any change to your routing logic beyond periodically re-checking where each model sits. Providers are also increasingly building routing-adjacent features directly into their platforms — automatic model selection, tiered pricing for the same model family, native prompt caching — which suggests that some of what teams build by hand today will eventually be table stakes rather than custom infrastructure, in the same way load balancing moved from custom code to a standard feature of nearly every cloud platform.
None of that changes the underlying judgment this piece has tried to build, though. Whatever the tooling looks like in a year, the questions that determine good routing will still be the same ones: what does this specific call actually require, what does getting it wrong actually cost, and how would you know if it happened. A router is only as good as the honesty of the analysis behind it, and that analysis — enumerating your call sites, scoring them on difficulty and stakes and latency sensitivity, measuring rather than assuming where the cheap tier succeeds and fails — is work no amount of better tooling will do for you.
The teams that get real value from routing are not the ones with the most sophisticated learned classifiers or the longest model ladders. They are the ones who took the time to actually understand what their agent was asking of a model at each step, who built the simplest router that captured the obvious wins, and who kept watching the output quality as closely as the bill, because a router that saves money while quietly degrading answers has not saved anything at all — it has just moved the cost somewhere harder to see. Build routing as a measured, observed, continuously revisited part of the system, the same way you would build any other piece of infrastructure that touches every request, and it becomes one of the best-leverage investments available in an agent's engineering: cheaper, faster, and, done carefully, no less correct than sending everything to your most expensive model would have been.