Every company that has ever built a dashboard has had the same conversation. Someone in a meeting asks a question the dashboard almost answers, revenue by region is right there, but they wanted it by region and product line, excluding the pilot customers, compared to the same quarter last year. The analyst who could produce that number has a backlog measured in weeks. So the question dies in the meeting, or someone eyeballs an approximation from the charts that do exist, and a decision gets made on a number nobody actually computed. Multiply that moment across every meeting in every company and you have one of the largest reservoirs of unmet demand in software: the gap between the questions people have and the questions their data tools can answer without an engineer in the loop.
Data agents are the current best attempt to close that gap. The premise is simple to state: let people ask questions of their data in plain language, and let a language model do what the analyst would have done, find the right tables, write the query, sanity-check the result, and explain what it found. The premise is also, as anyone who has tried to ship one knows, deceptively hard to deliver. The distance between a demo that answers "how many orders did we get last month" and a production system a CFO will trust with a board-deck number is enormous, and the interesting engineering lives entirely in that distance. This post is a tour of that territory: what makes querying data genuinely difficult for language models, the architectures that work, the guardrails that are not optional, and how to tell whether the thing you built is actually right.
Why data is the killer app hiding in plain sight
Most of the excitement around agentic AI clusters on the dramatic capabilities: agents that write whole features, operate browsers, run for hours unattended. Data analysis looks pedestrian next to that, and the neglect is a mistake, because it is close to an ideal match for what language models are actually good at today. The tasks are short-horizon: a question comes in, a query goes out, a result comes back, and the loop closes in seconds rather than hours, which means the compounding-error problem that plagues long-running agents barely applies. The output is checkable: a SQL query is an artifact a human can read, a result is a number that can be cross-verified, and the whole interaction leaves an audit trail in a way that a long freeform reasoning chain does not. And the demand is bottomless, because every organization has orders of magnitude more questions than analysts.
There is also a structural reason the timing is right. A decade of investment in data infrastructure, warehouses, lakehouses, transformation pipelines, cataloging tools, has left most mid-size and large companies with data that is, for the first time, mostly in one queryable place. The bottleneck moved. It used to be that the data was scattered and dirty and the analyst spent eighty percent of their time assembling it; increasingly the data is assembled and the scarce resource is the person who knows how to interrogate it. A bottleneck made of human query-writing capacity is exactly the kind of bottleneck language models are suited to attack.
None of this means the problem is easy. It means the problem is valuable, and that the difficulty is concentrated in specific, nameable places rather than smeared evenly across the whole endeavor. The rest of this post is about those places.
Text-to-SQL: solved in the demo, unsolved in the warehouse
The core capability underneath a data agent is text-to-SQL: translating a natural-language question into a query against a real schema. On academic benchmarks this looks close to solved, with frontier models scoring impressively on standard test suites. Anyone who has pointed a model at a production warehouse knows better, and the reasons for the gap are instructive because they define what the surrounding system has to provide.
Benchmark schemas are small, clean, and self-describing: a handful of tables with names like customers and orders, columns that mean what they say, and one obvious join path between any two entities. Production warehouses are none of these things. A real warehouse has thousands of tables accumulated across years of pipeline changes, many of them abandoned, duplicated, or subtly wrong. The table that looks canonical, orders, was deprecated two years ago in favor of orders_v3, which everyone materializes through a transformation model called fct_orders, and the difference between them is a deduplication rule that only the data team remembers. Columns carry names like amt_usd_adj whose precise meaning, adjusted how? net of refunds? tax-inclusive?, lives in a Slack thread from 2023. Revenue is not a column anywhere; it is a calculation, and finance and sales compute it differently on purpose.
A language model given raw access to this environment does not fail loudly. It fails politely, writing fluent, syntactically perfect SQL against the wrong table, joining on a column that looks like a key but is not one, summing a field that was never meant to be summed. The query runs. A number comes back. The number is wrong, and nothing about the interaction signals it. This is the central failure mode of data agents and it deserves a name that keeps it in view: the plausible wrong answer. Everything else in this post is, one way or another, a defense against it.
A data agent's worst output is not an error. It is a confident, well-formatted, plausible number that is wrong.
The real problem is context, not SQL
Once you have watched a capable model fail against a real warehouse, the diagnosis becomes obvious: the model does not lack SQL ability, it lacks the context a human analyst carries. A new analyst joining a company writes wrong queries for their first month too, and for exactly the same reasons, until they absorb the tribal knowledge: which tables are trustworthy, what the grain of each table is, how this company defines active user, which joins are safe and which produce fan-out, what the known data-quality landmines are. Making a data agent work is overwhelmingly an exercise in writing that tribal knowledge down and delivering the right slice of it into the model's context at query time.
This reframing matters because it relocates the engineering effort. Teams that treat text-to-SQL as a modeling problem burn months on fine-tuning and prompt tricks and plateau at a quality level that is impressive and unusable. Teams that treat it as a context problem spend their time curating: selecting the few hundred tables that matter from the few thousand that exist, writing real descriptions for tables and columns, documenting the canonical joins, defining the metrics, and encoding the caveats. The model is the same; the outcomes are not remotely comparable. In this sense the data agent problem is a special case of the general truth about agents, that context engineering is the real bottleneck, but with an unusually concrete shape: the context that matters is a description of what the data means, and most organizations have never written it down anywhere.
The practical form this takes is a curated semantic catalog that the agent retrieves from at question time. For each blessed table: what it contains, its grain, its freshness, which columns are safe to aggregate, and example queries that use it correctly. For each metric: its exact definition, in SQL or in a metrics layer, along with who owns it and when it changed. For the warehouse as a whole: the short list of rules an analyst would rattle off on your first day, always filter test accounts, fiscal year starts in February, the events table double-counts before March 2024. Retrieval over this catalog, not over raw schema dumps, is what the agent should see, because feeding a model three thousand raw table definitions does not just waste context, it actively invites the plausible wrong answer by presenting deprecated tables as live options.
The semantic layer question
There is a live architectural debate about how far to formalize that catalog, and it maps onto a spectrum. At one end, the agent writes raw SQL against the warehouse with the catalog as advisory context. At the other, the agent never writes SQL at all; it calls a semantic or metrics layer, a governed API where metrics like revenue and dimensions like region are predefined by the data team, and the layer compiles the request into correct SQL deterministically. The middle ground, increasingly common, lets the agent compose queries through the semantic layer for governed metrics and fall back to raw SQL, clearly labeled as such, for exploratory questions the layer does not cover.
The trade is between coverage and correctness. Raw SQL can answer anything, including incorrectly. The semantic layer can only answer what has been defined, but its answers inherit the layer's correctness guarantees: the join logic, the deduplication, the fiscal calendar are all encoded once by people who know, rather than re-derived by a model on every question. For the questions that matter most, the ones that end up in board decks and revenue reports, that guarantee is worth far more than flexibility, which is why the strongest practical recommendation in this space is unglamorous: the highest-leverage work in most data-agent projects is not agent engineering at all, it is finally building the metrics layer the data team has been proposing for years. The agent then becomes a natural-language interface to something already true, rather than a fluent guesser about something ambiguous.
A useful discipline for the hybrid setup is to make the provenance visible in the answer itself: a result computed through the governed layer carries a different badge than one computed through freehand SQL, so consumers learn to calibrate trust accordingly. This mirrors how analyst teams already work, a number from the certified dashboard versus a number from a quick ad-hoc query, and preserving that distinction through the agent, rather than flattening everything into one confident voice, is a small design choice with large consequences for how much damage a wrong answer can do.
Beyond the query: the analysis loop
Answering a question rarely ends with one query, and the gap between a text-to-SQL tool and a data agent is exactly the loop around the query. A real analysis has stages: understand the question, explore what data exists, draft a query, inspect the result for red flags, refine, and finally interpret, turning numbers into a claim about the business. Language models can perform every stage of that loop, and the systems that feel qualitatively different to use are the ones that let them.
Concretely, this means the agent gets more tools than a SQL executor. It gets schema-inspection tools, list the columns and types of a table, profile the distribution of a column, count the nulls, so it can look before it queries, the way an analyst pokes at a table before trusting it. It gets a scratch space for intermediate results, so a complex question can decompose into steps instead of one heroic query. It gets a code-execution environment, because past a certain point analysis outgrows SQL: cohort retention curves, statistical tests, forecasting, anything with a loop in it wants a dataframe library and a real programming language, and the natural pattern is SQL to fetch, code to analyze. And it gets a charting capability, because for many questions the honest answer is a shape, a trend, a distribution, an outlier, not a scalar.
The loop structure also gives the agent its most underrated capability: self-checking. An agent that just answered "monthly recurring revenue for March" can be instructed, cheaply, to verify before responding: run the same aggregate a second way, compare the total against the governed dashboard number if one exists, check whether March's figure deviates wildly from February's and flag it if so. These verification steps are individually trivial, and together they catch a substantial fraction of plausible wrong answers before a human ever sees them, converting silent failures into visible questions. An analyst does this instinctively, glancing at a result and thinking that can't be right; giving the agent an explicit budget of turns to be suspicious of its own output is how you approximate that instinct.
Ambiguity is a feature of the questions, not a bug in the users
A large fraction of natural-language questions about data are underspecified, and this is not because users are sloppy but because the precision was never needed in human conversation. "How did sales do last quarter" hides half a dozen genuine decisions: gross or net, bookings or recognized revenue, calendar or fiscal quarter, all regions or the ones this person manages, compared to plan, to prior quarter, or to the same quarter last year. A human analyst resolves these silently using context about who is asking and why. An agent has three options: guess, ask, or hedge, and the right choice is situational in ways worth designing deliberately.
Guessing is correct when a dominant interpretation exists, and the agent should guess the way a good analyst does: state the assumption inline, "using fiscal Q2 and net revenue, here's the picture", so the interpretation is visible and correctable rather than buried. Asking is correct when interpretations genuinely diverge and the cost of the wrong one is high, but an agent that interrogates the user before every answer is exhausting, so clarifying questions should be spent like a scarce currency, one at a time, only on ambiguities that materially change the answer. Hedging, computing two or three interpretations and presenting them side by side, is underused and often best, because it converts an invisible decision into a visible comparison, and frequently the user learns something from the gap between definitions that they would never have learned from either number alone.
What makes all three strategies work better over time is memory. The agent that learns this user means fiscal quarters, that this team's "revenue" is the finance definition, that the Tuesday question about pipeline is always scoped to enterprise accounts, stops re-litigating the same ambiguities. Persisting resolved interpretations, per user and per team, and retrieving them as context on subsequent questions, is one of the highest-leverage memory applications in any agent domain, because the vocabulary of a company's questions is small and stable; a few dozen learned definitions cover most of the traffic.
A good data agent resolves ambiguity the way a good analyst does: visibly. The assumption belongs in the answer, not buried under it.
Guardrails: the boring ones are the important ones
The security conversation around data agents tends to reach for the dramatic scenario, an injected prompt exfiltrating the customer table, and that scenario is real and worth defending against. But the guardrails that earn their keep daily are duller and should be non-negotiable from the first prototype. The agent's database credentials are read-only, full stop; no data agent has any business holding write permissions, and the enforcement belongs in the database's own permission system, not in a prompt instructing the model to be careful. Query cost is capped: a timeout, a row limit on results, a bytes-scanned ceiling on warehouses that bill by scan, because a model exploring a petabyte warehouse can innocently write a query that costs hundreds of dollars, and it will do so at whatever frequency your users ask casual questions. Concurrency is limited, so a burst of enthusiasm does not become a denial-of-service against the warehouse the rest of the company is using.
Access control is the guardrail that turns subtle, because a data agent inherits a hard problem from the systems underneath it: not everyone is allowed to see everything, and a natural-language interface makes it dangerously easy to forget that. The agent must operate under the asking user's permissions, not under a service account with global read, and the enforcement again belongs at the data layer, row-level security, column masking, governed views, where it cannot be talked around. The failure modes without this are not hypothetical: salary tables, customer personal data, and deal-desk pricing all live in warehouses, and an agent with a god-credential and a chat interface is an internal data breach with good UX. Aggregation adds a wrinkle worth knowing: even permitted aggregates can leak, when a filter narrows to a group of one and the average salary of engineers hired in March in Lisbon is one person's salary. Mature deployments borrow the old defenses from statistical disclosure control, minimum group sizes and suppression of small cells, for exactly this reason.
Prompt injection, finally, deserves its specific due, because data agents read data, and data is an input channel. A text field in the warehouse, a support ticket, a product review, a customer name, can contain instructions, and an agent that reads query results back into its own context can be steered by what it reads. The defenses are the standard ones, treat retrieved content as data rather than instructions, constrain what the agent can do downstream of reading untrusted fields, keep the blast radius small, but the data-specific point is about that blast radius: an agent that is read-only, cost-capped, permission-scoped, and unable to call arbitrary external endpoints has little worth hijacking. The boring guardrails are also the injection defense.
Architecture: three patterns that actually ship
Surveying the deployments that work, three architectural patterns recur, and they compose rather than compete. The first is the governed-query agent: natural language in, semantic-layer call or carefully-contexted SQL out, result back, answer with provenance. This is the workhorse for the recurring business questions that make up most traffic, and its defining property is that correctness is engineered into the path, curated catalog, governed metrics, verification turns, visible assumptions, rather than hoped for. It runs on a fast model, answers in seconds, and is cheap enough to put in front of everyone in the company.
The second is the analyst-in-a-sandbox: for open-ended investigative work, the agent gets a code-execution environment with dataframe tooling, a workspace it can persist intermediate artifacts into, and a longer leash, multiple queries, exploratory passes, charts, a written summary at the end. This is the pattern for "why did churn spike in the Northeast" rather than "what was churn", and everything about running agent-generated code applies: the sandbox is isolated, the data mounted into it is already permission-filtered, egress is closed, and the environment is destroyed afterward. The output is not just an answer but a small reproducible analysis, the queries, the code, the figures, which a human analyst can review, correct, and promote into something durable.
The third is the pipeline agent, which points the same capabilities inward at the data platform itself: watching for schema drift and freshness failures, triaging data-quality alerts, drafting the fix for a broken transformation, opening the pull request for a human to review. It is the least discussed of the three and possibly the highest-value per invocation, because the audience is the data team and the alternative is paging a human at 3 a.m. The common thread across all three patterns is that the agent's autonomy is calibrated to the reversibility of its actions: the query agent acts freely because reads are safe; the sandbox agent explores freely inside walls; the pipeline agent proposes but does not merge.
Evaluation: golden questions and the gap they leave
Nothing about a data agent can be trusted until it is measured, and measurement here has a convenient property: answers are checkable in a way most agent outputs are not. The foundation is a golden set, a curated suite of real questions from your organization, each paired with the verified correct answer and, ideally, the query an expert would have written. Building it is unglamorous work that the data team is uniquely positioned to do, and a hundred well-chosen questions, spanning easy lookups, multi-table joins, metric definitions, ambiguous phrasings, and known trap cases like deprecated tables, will teach you more about your agent than any benchmark score. Run it on every change to the model, the prompt, the catalog, or the semantic layer, because regressions in this domain are silent: nothing crashes when the agent starts picking the wrong revenue definition again.
Grading needs care beyond string comparison. The same correct answer can arrive as a differently-ordered result set, a rounded number, or an equivalent query with different syntax, so grade on result equivalence, execute both queries and compare outputs, with tolerance where appropriate, rather than on query text. Track not just accuracy but the failure taxonomy: wrong table, wrong join, wrong metric definition, wrong filter, misread question. The taxonomy tells you where to invest, because a cluster of wrong-metric failures is a catalog problem, while a cluster of wrong-join failures might be a schema-documentation problem, and neither is a model problem. Teams that skip the taxonomy end up swapping models to fix what a paragraph of table documentation would have fixed for free.
The golden set has a blind spot worth naming: it measures the questions you thought to include, and production traffic is a long tail of questions you did not. Complement it with production monitoring that exploits the checkability of the domain: log every question, query, and result; track how often users correct, rephrase, or abandon; sample sessions for expert review the way support teams audit tickets; and watch the disagreement rate between the agent's answers and governed dashboard numbers for overlapping questions, because that disagreement rate is the closest thing this domain has to a free, continuously-computed accuracy metric. When a user rephrases the same question three times, that session is a failed eval case waiting to be added to the golden set, and the loop from production failure to eval case to catalog fix is the actual mechanism by which these systems get good.
Showing your work: trust is a UX problem
Suppose the engineering all works: the agent is right, say, ninety-plus percent of the time on your golden set. The remaining design problem is that users cannot tell which answers are in the ninety and which are not, and the difference between a system people trust and one they quietly stop using lives in how the answer is presented. The single most important element is showing the work: the query, in expandable form for those who read SQL; the tables touched and their freshness; the assumptions made, in plain language, for those who do not. An answer that says "using fct_orders, updated 2 hours ago, counting net revenue, excluding internal test accounts" invites exactly the right kind of scrutiny, and it does something subtler: it teaches users, over time, what the data can and cannot say, the way working with a good analyst does.
Calibrated presentation matters as much as transparency. The agent should sound different when it is sure than when it is guessing, a governed-metric answer delivered flatly, a freehand exploratory answer delivered with its caveats attached, and it should escalate gracefully: for a number that is about to leave the building, into a board deck or a customer email, the honest move is to recommend verification by a human, and the systems that do this earn more usage, not less, because trust compounds. The anti-pattern is uniform confidence, every answer delivered in the same authoritative register regardless of provenance, which trains users either to trust everything, until the first visible wrong answer destroys the relationship, or to trust nothing, which makes the system decorative.
There is one more UX decision with outsized consequences: making correction cheap. A user who spots a wrong assumption should be able to fix it in one utterance, "no, fiscal quarters", and see the answer recompute, with the correction remembered. Every corrected answer is simultaneously a satisfied user and a harvested training signal, and products that treat corrections as annoyances rather than as the primary feedback channel are discarding the most valuable data they generate.
Users cannot tell your agent's right answers from its wrong ones. The interface has to carry that information, or the trust never forms.
Cost, latency, and routing the easy questions down
The economics of data agents follow the same lopsided distribution as every agent workload: a large majority of questions are simple, a small minority are genuinely hard, and the architecture should refuse to pay hard-question prices for easy questions. A question that maps cleanly onto a governed metric, "MRR last month", needs schema-free routing to a semantic-layer call and a small, fast model to phrase the answer; burning frontier-model tokens and a twelve-turn agent loop on it is pure waste, and worse, it is slow, and speed is a feature with unusual leverage here because the competing workflow, asking an analyst, has latency measured in days. A data agent that answers common questions in under five seconds changes how often people ask questions at all, which is the actual prize.
Routing by difficulty is easier in this domain than most because difficulty signals are legible before answering: does the question match a known metric, does it reference entities the catalog recognizes, has this user asked something isomorphic before. A cheap classifier handles the triage, sending metric lookups down the fast path, novel multi-table questions to the full loop with a stronger model, and investigative "why" questions to the sandbox pattern with a human-visible heads-up that this will take a few minutes. Caching adds another layer the domain is unusually suited to: the same organizational questions recur on a weekly and monthly rhythm, and a semantic cache that recognizes "how's pipeline looking" as last Tuesday's question with a fresh date range can serve a large share of traffic for the cost of a lookup plus a re-execution of a known-good query.
Warehouse cost deserves its own line item, because it can quietly dominate the model bill. Agents explore, and exploration means queries a human would never run: full scans of large tables to profile a column, joins materialized just to check a hypothesis. The mitigations are mechanical, enforce scan limits, point profiling tools at table statistics and samples rather than raw scans, maintain pre-aggregated summary tables for the common aggregates, but they need to exist from day one, because the first month's warehouse bill after launching an agent to a few hundred enthusiastic users is a rite of passage nobody enjoys.
When not to build one
An honest survey includes the cases where a data agent is the wrong answer. If the underlying data is not trustworthy, pipelines break weekly, definitions drift, nobody agrees what revenue means, an agent adds fluent confidence on top of chaos, and the project that actually helps is the boring one: fix the pipelines, define the metrics, then come back. If the organization's real need is a handful of stable numbers watched daily, a dashboard remains the right tool, cheaper, faster, and glanceable in a way conversation never is; agents complement dashboards by absorbing the long tail of follow-up questions, not by replacing the front page. And if the questions that matter are high-stakes and low-frequency, quarterly close, regulatory filings, the economics favor the human expert with agent assistance rather than the agent with human oversight, because the cost of an error dwarfs the cost of the analyst's time.
The pattern in all three cases is the same: a data agent amplifies the data culture it lands in. Strong foundations, governed metrics, documented schemas, a team that curates, get amplified into something that feels miraculous. Weak foundations get amplified too, into confident wrongness at conversational speed. The technology is largely ready; the prerequisite that is usually missing is organizational, and the teams that succeed are the ones that treat the agent as a reason to finally do the data-governance work, not as a way to skip it.
Where this is heading
Three trajectories seem reasonably safe to project. The first is that the semantic layer wins, in the sense that the governed-metrics pattern becomes the assumed substrate for serious deployments, the way containers became the assumed substrate for deployment generally. The incentives all point that direction: model vendors want correctness they can advertise, data vendors want the layer to be the product, and buyers want numbers they can defend. Expect the interface between agents and metrics layers to standardize, plausibly through the same protocol machinery that standardized tool connectivity generally, so that a data team defines a metric once and every agent surface in the company, chat, BI tool, spreadsheet copilot, computes it identically.
The second is that the analyst role shifts rather than shrinks. The work moving to agents, translating questions into queries, is the least valued part of the job, and what expands to fill the time is the part that was always scarce: deciding which questions matter, encoding domain knowledge into the catalog and the metrics layer, reviewing and promoting agent-produced analyses, and doing the investigative work whose ambiguity still exceeds what a loop with a sandbox can navigate. The analysts who thrive will be the ones who become, in effect, editors of an agent's output and authors of its context, and the leverage per analyst goes up substantially, which history suggests increases demand for analysts rather than reducing it, because the price of a question falling has always made organizations ask more questions.
The third is that conversational access changes what organizations measure at all. Dashboards impose a filter: only questions someone anticipated, and considered worth engineering, get standing answers. When the marginal cost of a novel question drops to near zero, the long tail of questions that were never worth a dashboard, one-off comparisons, small-segment curiosities, hypothesis checks, starts actually getting asked, and some fraction of them turn out to matter. The organizations that benefit most from data agents will not be the ones that answer their existing questions faster, but the ones that start asking questions they previously could not afford.
Getting started without regretting it
For a team building this today, the sequencing that works is narrow and foundation-first. Start with the catalog, not the agent: pick the fifty most-queried tables, write real descriptions, document the joins and the landmines, and define the ten metrics that account for most questions, in a semantic layer if you have one, in a versioned SQL file if you do not. This week of work does more for eventual accuracy than any amount of prompt engineering. Stand up the governed-query pattern first, read-only credentials, cost caps, and user-scoped permissions from the very first prototype, and put it in front of a friendly pilot group whose questions you log. Build the golden set from their real questions, grade on executed-result equivalence, and establish the loop, production failure becomes eval case becomes catalog fix, before expanding access, because that loop is the engine of quality and it is much easier to install while the system is small.
Add the sandbox pattern when investigative questions start hitting the ceiling of single queries, and add memory, learned definitions, per-team vocabulary, resolved ambiguities, as soon as you notice the same clarifications recurring, which will be immediately. Resist the temptation to launch company-wide the week the demo impresses an executive; the demo always impresses, because the demo questions are the easy ones, and the trust you burn on early plausible-wrong-answers is expensive to rebuild. Expand instead along the gradient of confidence: more users as accuracy on the golden set holds, more autonomy as verification turns catch what they should, more surface area as the catalog grows to cover it.
The bottom line
Data agents sit at an unusual intersection: enormous latent demand, a task profile that plays to current model strengths, and a failure mode, the plausible wrong answer, that is manageable with engineering that is well understood if unglamorous. The work that matters is mostly not model work. It is writing down what the data means, governing the definitions that matter, enforcing the boring guardrails at the database rather than in the prompt, measuring against real questions with real answers, and designing interfaces that let users see which answers to trust. Teams that do that work end up with something transformative in the most literal sense: the cost of asking a question of the business drops to nearly nothing, and the organization starts thinking with its data instead of merely reporting from it. Teams that skip it end up with a fluent narrator of numbers nobody should believe. The difference between the two outcomes was never the model. It is whether anyone wrote down what revenue means.
