Language models speak prose, but software speaks data. This is a practical guide to structured outputs: schemas, constrained decoding, validation and repair, and the design decisions that determine whether the data your model produces can actually be trusted by the code that consumes it.
The impedance mismatch at the heart of every LLM application
Every application built on a language model eventually collides with the same uncomfortable fact: the model produces text, and the rest of your system does not want text. Your database wants rows with typed columns. Your API wants a JSON body that matches a contract. Your frontend wants a predictable shape it can render without defensive checks on every field. Your downstream service wants an enum value from a closed set, not a creative paraphrase of one. The model, left to its own devices, produces none of these things. It produces prose, and prose is the least machine-friendly format ever devised.
This mismatch is easy to underestimate because it looks trivial in a demo. You ask the model to "respond in JSON," it responds in something that looks like JSON, you parse it, and everything works. Then you ship, and the model wraps the JSON in a markdown code fence. Or it adds a friendly sentence before the opening brace. Or it invents a field, omits a required one, returns a string where you expected a number, or produces a value for your status enum that has never existed in your system. Each of these is rare on any single call, but you are not making a single call. You are making thousands or millions, and at that scale the rare deviation becomes a daily production incident.
Structured output is the discipline of closing this gap deliberately: getting data out of a language model in a shape that downstream code can rely on, with guarantees that are engineered rather than hoped for. It sits at the boundary between the probabilistic world of the model and the deterministic world of everything else, and like most boundary work, it is unglamorous, essential, and full of subtle decisions that determine whether the whole system is dependable. This post is a tour of that boundary: how we got here, how the modern mechanisms actually work, and how to design schemas and pipelines that hold up under real traffic.
A short history of parsing hope
The earliest LLM applications extracted structure the same way we scraped websites in 1999: with regular expressions and optimism. You prompted the model to follow a format, then wrote brittle parsing code that looked for the patterns you asked for. When the model deviated, the parser broke, and you patched the prompt with increasingly desperate instructions. Anyone who has written "IMPORTANT: Respond ONLY with valid JSON. Do NOT include any other text" in a production prompt knows this era intimately, and knows that the model treated such instructions as strong suggestions at best.
The next stage was the era of JSON mode. Providers added a switch that constrained the model to emit syntactically valid JSON, which eliminated an entire class of failures: no more markdown fences, no more chatty preambles, no more truncated braces. But JSON mode guaranteed only syntax, not shape. The model still chose its own field names, its own nesting, its own types. Valid JSON that does not match your expected structure is only marginally more useful than prose, because your code still cannot consume it without a layer of defensive interpretation.
Function calling arrived as the first real schema mechanism. By describing tools as JSON Schema and letting the model emit a call with arguments, providers effectively let you specify the shape of the output, not just its syntax. Developers quickly noticed that you could define a "function" that was never going to be executed, purely as a way to force the model to produce arguments matching a schema. Extraction-by-fake-function became one of the most widely used patterns in the industry, a sign of how badly the underlying need was going unmet.
The current stage is native structured output backed by constrained decoding: you hand the provider a schema, and the generation process itself is constrained so the model cannot emit output that violates it. This is a fundamentally different kind of guarantee from everything that came before, because it moves conformance from the realm of prompting, where you request, into the realm of decoding, where you enforce. Understanding what that enforcement does and does not buy you is the core technical knowledge of this whole topic, and it is where we go next.
How constrained decoding actually works
A language model generates text one token at a time. At each step, it produces a probability distribution over its entire vocabulary, tens of thousands of possible next tokens, and the sampler picks one. Ordinarily every token is in play, which is precisely why the model can wander off format: nothing stops it from choosing a token that begins a polite apology in the middle of your JSON object.
Constrained decoding intervenes at exactly this point. Before sampling, the system consults a formal description of what is allowed next, typically your JSON Schema compiled into a grammar or a state machine, and masks every token that would violate it. If the grammar says the next character must be a quotation mark or a digit, then every token that starts with anything else has its probability set to zero. The model then samples only from the tokens that keep the output legal. It is not that the model is trying harder to follow your format; it is that the format violations have been made unrepresentable.
The elegance of this approach is that it composes with everything else the model does. The model still brings all of its intelligence to choosing among the permitted tokens; the constraint only removes the forbidden ones. The engineering challenge, which the inference frameworks handle for you, is doing this efficiently: compiling the schema into a form that can be evaluated against the vocabulary at every step without stalling generation, and handling the awkward fact that tokens are not characters, so a single token may span a boundary in the grammar.
What constrained decoding buys you is absolute: the output will parse, and it will match the schema's structure. What it does not buy you is correctness. The model can produce a perfectly schema-valid object whose values are wrong, invented, or subtly off. Constrained decoding guarantees the shape of the container, never the truth of the contents. Keeping that distinction sharp in your mind is the single most important idea in this post: syntax is now a solved problem, and semantics is still entirely yours.
Constrained decoding guarantees the shape of the container, never the truth of the contents. Syntax is a solved problem; semantics is still entirely yours.
The schema is a contract, and also a prompt
It is tempting to treat your output schema as a purely mechanical artifact, a config file for the decoder. This misses half of what the schema does. The schema is read by the model. Field names, descriptions, enum values, and structure all function as prompt material that shapes what the model produces. A schema is simultaneously a contract enforced by the decoder and a set of instructions interpreted by the model, and designing it well means taking both roles seriously.
Consider a field named x1 with no description versus a field named customer_sentiment described as "The customer's overall emotional tone in this conversation, judged from their messages only, not the agent's." The decoder treats these identically; the model absolutely does not. The second gives the model everything it needs to fill the field well; the first invites it to guess. In practice, well-written field descriptions are among the highest-leverage prompt engineering you can do, because they sit exactly where the model is making its decision about what value to emit.
The same is true of enums. An enum is a closed set of permitted values, and constrained decoding will hold the model to it. But the values themselves teach the model what the categories mean. A ticket triage enum of billing, technical, account_access, and other works dramatically better when each value is documented with what belongs in it and, just as importantly, what does not. The most common enum failure in production is not an invalid value, the decoder prevents those, but a systematic misclassification because the boundaries between categories were never made explicit anywhere the model could see them.
Designing schemas the model can actually fill
Schema design for language models is its own craft, related to but distinct from schema design for databases or APIs. The guiding principle is that the model fills the schema in generation order, token by token, with no ability to go back and revise. Every design choice should be evaluated against that reality.
Field order matters because generation is sequential. If your schema asks for a final answer first and supporting reasoning second, the model must commit to the answer before it has done the thinking, and the reasoning it emits afterward is post-hoc rationalization. Reversing the order, reasoning first, then the answer, lets the model use its own generated analysis as context for the conclusion. This one reordering is among the cheapest accuracy improvements available in structured extraction, and it falls directly out of understanding how generation works.
Flat schemas outperform deeply nested ones. Every level of nesting adds bookkeeping the model must maintain while generating, and deeply nested structures invite errors in ways that flat ones do not. If you find yourself designing a schema with four levels of nesting and arrays of objects containing arrays, consider whether the task can be decomposed into multiple simpler extractions instead. Two clean calls with simple schemas routinely beat one call with a schema that resembles an org chart.
Optionality deserves deliberate thought. A required field forces the model to produce something even when the source material contains nothing, which is how you get confident fabrication in your database: the schema demanded a value, so the model invented one. Fields that may legitimately be absent should be nullable, and their descriptions should say explicitly that null is the correct answer when the information is not present. The description "The invoice due date, or null if no due date is stated; do not guess" does real work. Conversely, making everything optional gives the model permission to be lazy and return sparse objects. The discipline is to make required exactly the fields that are always knowable, and nullable exactly the ones that are not.
Types should be as narrow as truth allows. Use an enum rather than a free string wherever a closed set exists. Use integers where fractions are meaningless. Constrain formats for dates and identifiers. Every narrowing removes a family of failure modes at the decoder level rather than in your cleanup code, which is the cheapest possible place to remove them. But do not narrow past the truth: an enum that fails to cover a real case forces the model to shove reality into the nearest wrong bucket, and no amount of validation catches an error that the schema itself made mandatory. An escape hatch value like other, paired with a free-text field explaining it, is often the difference between a schema that works and one that quietly lies under pressure.
The tax on thinking
Constraints are not free. A model generating under a strict grammar is doing something subtly different from a model generating freely, and a body of practical experience suggests that heavy constraint can degrade the quality of the reasoning embedded in the output. The intuition is straightforward: models are trained on oceans of natural text, and their reasoning is entangled with the freedom to produce it. Force every token through a narrow grammatical keyhole and you are asking the model to think in a straitjacket.
For simple extraction, pulling names, dates, and amounts out of a document, the effect is negligible; there is not much thinking to degrade. It matters for tasks where the value in the output is the product of judgment: nuanced classification, weighing evidence, resolving contradictions in the source material. For those tasks, the strongest pattern is to separate thinking from formatting. Let the model reason in free text first, either in a separate call or in a designated reasoning field placed before the conclusions, and then capture the conclusions in the tightly constrained fields. You pay for the reasoning tokens, but you preserve the quality that was the whole point of using a capable model, and you keep the machine-readable guarantee where it matters.
This two-phase shape, think freely, then bind tightly, recurs across the field, from chain-of-thought fields inside schemas to fully separate extraction passes over the model's own prose. Reasoning models with native thinking phases formalize the same idea at the platform level: deliberation happens unconstrained, and structure is applied to the final answer. However you implement it, the principle is the same: apply constraints to conclusions, not to thought.
Apply constraints to conclusions, not to thought. Let the model reason freely, then bind the answer to the schema.
Validate, then repair: the loop that makes it production-grade
Even with constrained decoding, a validation layer between the model and your system is not optional, because schema conformance is only the first of the checks that matter. Validation is where you enforce everything the grammar cannot see: that the end date is after the start date, that the line items sum to the stated total, that the referenced customer identifier actually exists in your database, that the confidence score is calibrated against a value you are willing to act on. These are semantic invariants, and they live in exactly one trustworthy place: deterministic code you wrote.
The modern toolchain makes the mechanical part almost pleasant. Libraries like Pydantic in Python and Zod in TypeScript let you define the schema once, in the type system of your language, and derive everything from it: the JSON Schema you send to the provider, the runtime validation of what comes back, and typed objects for the rest of your code. The schema stops being a string in a prompt and becomes a first-class artifact in your codebase, versioned, reviewed, and tested like any other interface. Any pipeline still hand-writing JSON Schema in one place and validation checks in another is maintaining two sources of truth that will inevitably drift apart.
When validation fails, the distinguishing move of a production pipeline is the repair loop: feed the invalid output and the specific validation errors back to the model and ask it to correct them. Models are remarkably good at fixing their own output when told precisely what is wrong; "the total does not match the sum of line items, which is 417.50" produces a fix far more reliably than simply retrying from scratch. A repair loop needs the same guardrails as any agent loop: a small bounded number of attempts, escalation to a fallback or a human when repairs fail, and logging of every repair so you can see which schema fields are chronically problematic. That last part quietly becomes one of your best diagnostic instruments, because a field that constantly needs repair is almost always a field whose definition is ambiguous, and the fix belongs in the schema, not in more retries.
Function calling is structured output wearing a tool belt
It is worth being explicit about a unification that the industry arrived at gradually: tool use and structured output are the same mechanism. When an agent calls a tool, the model emits arguments conforming to the tool's declared schema; that is structured output. When you ask for extraction into a schema, you are defining the interface through which text becomes action; that is a tool call whose executor happens to be your own pipeline. The mental model that treats these as one thing is more accurate and more productive than the one that treats them as separate features.
This unification means everything in this post applies with extra force inside agents, because in an agent the structured output is not a terminal artifact but an action about to be executed. A malformed extraction in a batch pipeline produces a bad row you can fix tomorrow; a malformed tool call executes against the world now. The validation layer between the model's chosen action and its execution, checking the arguments against bounds, permissions, and sanity before anything happens, is the most important safety checkpoint in an agentic system, and it is nothing more or less than structured output validation deployed at the moment of highest consequence.
Agent builders also learn quickly that argument schemas are interface design for a very literal-minded user. A tool whose parameters are ambiguous will be misused in every way its ambiguity permits. The same craft that makes an extraction schema fillable, sharp names, explicit descriptions, narrow types, honest optionality, is what makes a tool callable. If you have internalized schema design for extraction, you already know how to design tool interfaces, and vice versa; it is one skill with two deployments.
Long documents and the reconciliation problem
Extraction tutorials operate on documents that fit comfortably in a context window. Production documents do not always cooperate: hundred-page contracts, year-long email threads, transcripts of eight-hour depositions. When the source exceeds what one call can see, structured extraction becomes a distributed systems problem, and the schema work acquires a second layer.
The standard approach is to chunk the document, extract from each chunk with the same schema, and then merge. Every step of that sentence hides a decision. Chunking must respect the document's semantic boundaries, because a clause split across two chunks may be extracted by neither, or worse, half-extracted by both. Extraction from a chunk must be told, in the prompt or the schema descriptions, that it is seeing a fragment, so that absence of information in this chunk is not treated as absence in the document; this is where nullable fields and no-guessing instructions stop being good hygiene and become load-bearing.
The merge is where the real difficulty lives. Two chunks may yield conflicting values for the same field: the contract's original payment terms in chunk three and the amendment that supersedes them in chunk forty-one. A naive merge takes the first or the last and is wrong half the time. A robust pipeline treats reconciliation as its own model task, presenting the conflicting candidates with their source locations and asking for a resolution with a stated reason, or escalates genuine conflicts to a human. Carrying source spans through the whole pipeline, which chunk, which offsets, produced each extracted value, costs a few extra schema fields and pays for itself the first time someone asks "where did this number come from?" In regulated settings that question is not curiosity; it is a requirement, and a pipeline that cannot answer it is a pipeline that cannot be audited.
Schema evolution without breakage
A schema that ships is a schema that will change. Categories get added, fields get renamed, a subteam wants one more attribute extracted. In ordinary software, interface evolution is managed with versioning discipline, and the model boundary deserves exactly the same treatment, plus one extra consideration: a schema change is also a behavior change. Adding a field does not just extend the contract; it alters what the model attends to, and can shift the accuracy of fields you never touched. The schema is a prompt, and you have edited the prompt.
The practical consequences are straightforward. Version your schemas explicitly, and record which version produced every stored extraction, because a table mixing outputs of three schema generations without labels is an archaeology project. Rerun the per-field eval suite on every schema change, including changes that look purely additive, because "purely additive" is a claim about the decoder, not about the model's behavior. And when a change is semantic, a category split in two, a field's definition sharpened, decide deliberately what happens to historical data: re-extract if the sources are still available, or mark the boundary clearly if they are not. Teams that skip this discipline rediscover it later, at the moment a dashboard silently blends two incompatible definitions of the same field and someone makes a decision based on the blend.
The open-model toolbox
Everything so far applies regardless of whose model you run, but constrained decoding has a special relationship with open-weight models: when you control the inference stack, the grammar machinery is yours to command directly. The open ecosystem, libraries like Outlines and llguidance, and the grammar support built into inference servers such as vLLM and llama.cpp, lets you constrain generation with arbitrary grammars, not just JSON Schema: regular expressions for identifiers, context-free grammars for domain-specific languages, custom formats no hosted API would offer as a checkbox.
This is more useful than it first sounds. Teams generating SQL constrain output to a grammar of the allowed query subset, making injection-shaped output unrepresentable rather than merely discouraged. Teams generating configuration files constrain to the exact dialect their parser accepts. On small self-hosted models, tight grammars punch above their weight: a modest model that cannot produce an illegal token competes surprisingly well, on narrow extraction tasks, with a much larger model that is merely asked nicely. Constraint substitutes for scale precisely where the task is format-heavy and judgment-light, which describes a great deal of the extraction work that runs in production at volume, and it is often the difference that makes on-premise deployment economical at all.
The trade-offs mirror the hosted case, with the dials exposed. Grammar compilation has a cost, paid at startup or first use; token masking adds per-step overhead that efficient implementations keep negligible; and the tax on thinking applies with the same force, so the think-freely-then-bind pattern remains the right shape for judgment-heavy tasks even when the grammar engine is entirely under your control. What the open toolbox really adds is range: the ability to enforce exactly the language your system consumes, however idiosyncratic, instead of the least common denominator a provider chose to support.
Cost, caching, and the economics of structure
Structured output changes the economics of a pipeline in ways worth naming. Schemas and tool definitions are prompt content, and elaborate ones are not small; a richly described schema can run to thousands of tokens, paid on every call. Prompt caching largely neutralizes this for high-volume pipelines, since the schema is exactly the kind of stable prefix caches are built for, but only if you keep it stable: a pipeline that assembles its schema dynamically per request forfeits the cache and pays full price for its own flexibility.
On the output side, structure is usually a savings. A schema-bound response contains no throat-clearing, no restatement of the question, no closing pleasantries, just the data, and output tokens are the expensive ones. The exception is the reasoning field, which deliberately buys tokens for accuracy; the point is to spend where spending works. Measured per correct extracted field rather than per call, well-designed structured pipelines are routinely cheaper than their prose-parsing ancestors, while also being the versions you can actually trust.
Streaming structure: parsing a river
Structured output and streaming pull in opposite directions. Streaming exists to show progress before generation finishes; structure traditionally exists only when the document is complete, because a JSON object is not valid until its final brace. Wait for completion and you lose the responsiveness users expect; parse eagerly and you are parsing a document that is, at every moment before the end, syntactically broken.
The resolution is partial parsing: parsers designed to consume incomplete JSON and yield the best current view of the object, field by field, as tokens arrive. A well-designed streaming pipeline can render the fields that have fully arrived, show the string that is still growing, and withhold anything not yet stable enough to act on. Libraries in both the Python and TypeScript ecosystems now do this well enough that streaming structured output is a solved plumbing problem, but two design consequences deserve attention.
First, field order becomes user experience. Whatever the schema emits first is what the user sees first, so the fields that make a stream feel alive, the title, the summary, the headline judgment, belong early, with slow bulky detail later. This is the same ordering lever discussed earlier, now serving latency perception instead of reasoning quality, and occasionally the two uses conflict; when they do, decide deliberately rather than by accident. Second, never act on a field until it is complete and validated. Rendering partial data is fine; executing on it is not. A truncated string is not yet the model's answer, and an object that has streamed halfway may still fail validation at the end. The rule is: stream for the human, but act only on the validated whole.
Failure modes: a field guide
The failures of structured output pipelines are distinctive enough to deserve names, because naming them is most of the way to catching them.
The confident fabrication is the classic: a required field, an absent fact, and a model that fills the gap with something plausible. It is the single most damaging failure because it produces data that looks exactly like good data. The remedies are nullable fields with explicit permission not to know, descriptions that forbid guessing, and spot-check evals that compare extracted values against source documents.
The empty-but-valid response is fabrication's quiet sibling: the model returns a schema-perfect object full of nulls, empty arrays, and hedged nothing. Every check passes; no information was extracted. Guard against it with a semantic floor, validation that a minimum of substance is present, and treat consistently sparse output as a signal that the task or the schema is confusing the model.
The nearest-bucket error afflicts enums: reality did not fit any permitted value, and the decoder would not let the model say so, so it chose the least-wrong option. From the outside this looks like a clean classification. The defense is schema design, an other escape hatch with an explanation field, plus monitoring of category distributions over time, because a bucket that suddenly grows is often absorbing cases that belong nowhere.
The unit slip is the subtlest: right field, right type, wrong interpretation. Cents where you expected dollars, a European date read American-style, a percentage as 0.15 in one row and 15 in the next. Types do not catch these; only explicit unit conventions in field descriptions and range checks in validation do.
What unites these failures is that none of them is a parse error. The era when structured output failed loudly, with an exception on the first malformed brace, is over. Constrained decoding has moved every failure inside the schema, where it is quieter and better-dressed. Your defenses had to move accordingly: from parsing, where the problem used to be, to validation and evaluation, where it lives now.
Constrained decoding did not eliminate failure; it moved every failure inside the schema, where it is quieter and better-dressed.
Evaluating extraction like you mean it
Because structured output fails semantically rather than syntactically, its quality cannot be observed casually; it must be measured. The good news is that structured output is the easiest LLM capability to evaluate rigorously, precisely because the output is comparable data rather than prose. There is no judgment call in whether an extracted date equals the labeled date. Field-level exact match, numeric tolerance, set comparison for arrays: these are cheap, deterministic, and unambiguous.
Build the eval set from real documents, including the ugly ones, the scanned invoice, the email thread with the correction three messages down, the contract with the amended clause, because the clean cases were never the risk. Label the truth once, carefully, and then every schema change, prompt tweak, and model upgrade can be scored in minutes. Per-field accuracy is the metric that matters, not per-document, because a pipeline that is 99 percent right on eight fields and 70 percent right on the ninth is a pipeline with one broken field, and the aggregate number hides it.
Track those per-field numbers over time and they become the instrument panel for the whole pipeline. A field whose accuracy sags after a model upgrade, a field that was never good and nobody noticed, a field whose errors cluster on one document type: each is invisible in aggregate and obvious per-field. Combined with the repair-loop logs from earlier, which tell you which fields chronically fail validation, you have the two data sources that between them locate almost every schema problem you will ever need to fix.
Choosing your level of enforcement
Not every task deserves the full apparatus, and part of the craft is matching the mechanism to the stakes. When the output feeds code, a database, an API, an execution engine, native structured output with constrained decoding is the default; there is no reason to accept probabilistic formatting when deterministic formatting is a parameter away. When the output is for a human to read, structure may be pure overhead; prose was the point. In between sit the judgment-heavy tasks where the two-phase pattern earns its keep: free reasoning, then bound conclusions.
A useful habit is to write down, for each model integration, what actually consumes the output. If the answer is software, name the schema, enforce it at decode time, validate it semantically, and eval it per-field. If the answer is a person, skip the ceremony. If the answer is both, a human reads it now and a workflow acts on it later, structure it, because the workflow's needs are stricter and the human can always be shown a rendering. Output that will ever be consumed by code should be structured from day one, because retrofitting schemas onto a pipeline that grew up parsing prose is a rewrite wearing the costume of a refactor.
The bottom line
Structured output is where the probabilistic world of the model meets the deterministic world of everything else, and the quality of that boundary determines how much of the model's intelligence your system can actually use. The mechanisms have matured remarkably: constrained decoding has made syntactic failure obsolete, schema-first libraries have made the contract a first-class artifact, and streaming parsers have reconciled structure with responsiveness. What has not changed, and will not, is that the schema guarantees shape, never truth. The engineering that matters now is semantic: schemas designed for how models generate, with honest optionality and narrow types and descriptions that do real work; validation that enforces the invariants no grammar can see; repair loops that recover from the failures that remain; and per-field evals that catch the quiet degradations before your users do.
None of this is glamorous, and all of it is leverage. A team that treats the model-to-software boundary with the same seriousness it gives any other interface, defined contracts, enforced invariants, measured conformance, gets to build on the model as if it were a dependable component, because at that boundary, it has been made into one. The teams that skip this work do not avoid it; they distribute it, unlabeled, across every downstream system that has to cope with data that was almost right. Speak to your models however you like. Make sure that when they answer your software, they answer in data you have engineered yourself the right to trust.