The oldest interface is the last one to be automated
Long before anyone typed a command into a terminal, and long before anyone clicked a button on a screen, people got things done by talking to each other. Voice is the interface humans are natively built for, the one that requires no literacy, no device familiarity, and no hand-eye coordination, and yet it is the interface that software has served worst for the longest time. An enormous share of the world's business still happens over the phone: clinics scheduling appointments, contractors quoting jobs, restaurants taking reservations, logistics brokers chasing shipments, insurance companies processing claims, and utilities fielding outage reports. Nearly all of that conversation is unstructured, spoken, and, until very recently, completely beyond the reach of any automation a caller would tolerate for more than thirty seconds.
Voice agents are the industry's attempt to close that gap: systems that hold a real spoken conversation, listening, reasoning, responding, and taking actions in the middle of the exchange rather than at the end of it. This is a fundamentally different proposition from the phone trees everyone has learned to hate, and it is a meaningfully different engineering problem from the text-based agents most teams have built so far, because a voice agent is not just a chatbot with a microphone attached. It is a real-time distributed system in which a large language model happens to be one component, surrounded by speech recognition, speech synthesis, audio transport, and turn-taking machinery that all have to cooperate within a latency budget measured in fractions of a second.
It is worth being precise about what changed to make this feel achievable now rather than a decade ago. Speech recognition crossed the threshold where it handles ordinary telephone audio, with its compression artifacts, background noise, and accents, well enough to be trusted as the ears of a live system. Speech synthesis stopped sounding like a navigation device and started sounding like a person, complete with natural pacing, emphasis, and the small imperfections that make speech feel human. And language models supplied the piece that never existed before: the ability to actually understand what a caller wants, hold the thread of a meandering conversation, and decide what to do about it. Each of these existed in weaker form for years; what is new is that all three are now good enough, and fast enough, to run together in a loop that feels like conversation rather than dictation.
From phone trees to full-duplex conversation
The first generation of voice automation did not attempt conversation at all. Interactive voice response systems, the press one for billing, press two for support machinery that still answers a large fraction of the world's service numbers, reduced the problem to navigating a fixed menu with a telephone keypad. They were, and are, perfectly reliable at what they do, because what they do is almost nothing: route a call down a predetermined tree, collect a digit at each branch, and hand the hard part to a human at the end. Nobody ever accused an IVR of misunderstanding them, because there was nothing to misunderstand; there was also nothing to like, which is why decades of consumer research put phone menus among the most disliked interfaces ever deployed at scale.
The second generation added speech recognition on top of the same fixed structure. Say billing, or say support replaced the keypad, and later systems allowed a full sentence at the start, classifying the caller's opening utterance into one of a few dozen predefined intents before dropping back into the same rigid flow. The natural language understanding stacks of the 2010s, with their intents, slots, and hand-authored dialogue states, were genuinely useful for narrow tasks, and genuinely maddening the moment a caller said anything the designers had not anticipated. The defining failure of that generation was rigidity: every conversation the system could handle had to be explicitly imagined, enumerated, and wired up in advance, which meant the long tail of real human requests, the ones that did not fit any intent, fell through to I'm sorry, I didn't understand that, over and over, until the caller started mashing zero.
Language-model-based voice agents invert that structure. Instead of classifying speech into a fixed set of intents, the model actually follows the conversation, wherever it goes, and decides at each turn what to say and what to do, drawing on tools, retrieved context, and instructions the same way a text-based agent does. The flow is no longer a tree someone drew; it is generated, turn by turn, from the actual content of the call. That flexibility is precisely the appeal and precisely the risk: the system that can gracefully handle a request nobody anticipated is the same system that can confidently mishear, misunderstand, or overstep in ways a fixed menu never could. Most of the engineering that separates a production voice agent from a demo exists to keep the first property while containing the second.
Anatomy of a voice agent: the cascaded pipeline
The dominant architecture in production today is a cascade of three specialized stages, each streaming into the next. Audio arrives from the caller, over a phone network or a browser connection, and flows into automatic speech recognition, which emits a running transcript that updates as the caller speaks. When the system judges that the caller has finished their turn, the transcript goes to the language model along with the conversation history, instructions, and available tools, and the model begins generating a response. Those tokens stream into text-to-speech, which begins producing audio before the model has finished its sentence, and that audio flows back to the caller. Three models, three vendors if you like, stitched together with careful plumbing.
Around that core sit components that have no equivalent in text-based systems and that determine much of the experience. Voice activity detection watches the incoming audio and distinguishes speech from silence and background noise. Endpointing decides the harder question of when the caller has actually finished a thought rather than merely paused to breathe. Barge-in handling notices when the caller starts talking over the agent and stops the agent's audio immediately. And the transport layer, telephony on one side or a real-time browser protocol on the other, moves audio in both directions with latencies low enough that none of the above falls apart. Every one of these components is invisible when it works and defines the entire call when it does not.
The emerging alternative is the speech-to-speech model: a single model that consumes audio directly and produces audio directly, with no intermediate transcript at all. The appeal is real. Latency drops because two model boundaries disappear. Paralinguistic information survives, the hesitation, the frustration, the sarcasm that a transcript flattens away, and the model's own speech can carry prosody that no text-to-speech stage reconstructed from bare words. But the trade-offs are equally real: a system with no transcript in the middle is harder to observe, harder to constrain, and harder to debug, tool calling from native audio models remains less mature than from text models, and swapping any single stage independently, upgrading just the recognizer, just the voice, becomes impossible when the stages no longer exist. The cascade remains the default for systems that act on the world precisely because the transcript in the middle is where control lives.
A voice agent is not a chatbot with a microphone. It is a real-time distributed system in which the language model is only one component.
The latency budget
In human conversation, the gap between one speaker finishing and the other beginning is startlingly short, typically a few hundred milliseconds, and often less, because people begin planning their response while the other person is still talking. A voice agent that takes two seconds to answer does not feel thoughtful; it feels broken, and callers respond to that silence the way they respond to a dropped call, by saying hello? and derailing whatever the agent was about to say. The entire system, therefore, lives inside a budget of roughly half a second to a second between the caller finishing and the agent's voice beginning, and every component in the pipeline is competing for a slice of it.
Walk through where the time actually goes. Audio transport consumes tens of milliseconds in each direction, more over congested networks or international telephony routes. The recognizer needs time after the caller stops speaking to finalize its transcript, and the endpointing logic needs additional time to be confident the caller is actually done rather than mid-pause. The language model's time to first token, not its total generation time, is what matters, because synthesis can begin as soon as the first words exist. And the speech synthesizer needs time to produce its first chunk of audio, not the whole utterance. Add those up carelessly and the total lands at two seconds or more; add them up with streaming at every boundary, aggressive endpointing, and models chosen for time-to-first-token rather than benchmark scores, and the same pipeline lands under a second.
The practical techniques are unglamorous and effective. Stream everything: never wait for a complete transcript, a complete response, or a complete audio file when a partial one can start the next stage early. Begin speculative work on partial transcripts, so the model is already reasoning about the probable request while the caller finishes their sentence, discarding the work in the minority of cases where the ending changes the meaning. Use conversational acknowledgments deliberately, a brief let me check that covers the gap while a slow tool call runs, and callers parse it as natural rather than as latency. And measure the thing that matters, time from caller-stops-speaking to agent-audio-begins, at the ninety-fifth percentile, not the average, because callers do not experience your average call; they experience the worst call they personally had.
Turn-taking, interruption, and the art of shutting up
Deciding when the caller has finished speaking is one of the hardest problems in the entire stack, and it is not primarily an acoustic problem. Silence is a terrible signal on its own: people pause mid-sentence to think, to check a document, to soothe a child, and a system that treats every pause as a completed turn interrupts callers constantly, which surveys of deployed systems consistently rank as the most infuriating behavior a voice interface can have. Modern endpointing therefore combines acoustic evidence with semantic evidence, asking not only has the audio gone quiet but does the transcript so far look like a finished thought. My date of birth is, followed by silence, is obviously an unfinished turn no matter how long the pause stretches, and a system that reads the words rather than just the waveform knows it.
The mirror-image problem is barge-in: the caller starts talking while the agent is mid-sentence. A production system must stop its own audio within a fraction of a second, because two voices talking over each other is intolerable on a phone line, and then it must do something subtler that inexperienced teams always miss: it must correct its own conversation history. The agent generated a full response, but the caller only heard the first third of it before cutting in, and if the transcript records the full generated text as having been said, the model's understanding of the conversation now diverges from the caller's reality. The state must reflect what was actually delivered, truncated where the interruption landed, or every subsequent turn builds on a shared history that was never actually shared.
There is also a category of speech that should not take the turn at all. Humans produce backchannels constantly, the mm-hmm, right, okay murmurs that mean I am listening, keep going rather than stop, my turn. A voice agent that halts every time the caller murmurs agreement becomes stuttering and unusable, so the interruption logic has to distinguish acknowledgment from genuine barge-in, usually by a combination of utterance length, energy, and content. Getting this trio right, endpointing that does not cut people off, barge-in that yields instantly, backchannel tolerance that does not yield at all, is most of the difference between a voice agent that feels like a conversation partner and one that feels like a very fast phone tree.
The hardest part of speaking is knowing when to stop. For a voice agent, it is also the part with the least margin for error.
Acting mid-sentence: tools in a real-time loop
A voice agent that can only talk is a novelty; the ones that matter look things up, book things, change things, and cancel things while the conversation is happening. Every pattern from tool-using text agents applies, with one merciless addition: the clock. A text agent that takes four seconds to query an order system produces a spinner nobody resents. A voice agent that goes silent for four seconds produces a caller asking are you still there, or worse, hanging up. Slow tools therefore have to be wrapped in conversation, the agent announcing what it is doing before it does it, one moment, I'm pulling up that order, because narrated latency reads as competence while silent latency reads as failure.
The sequencing discipline matters more in voice than anywhere else, because speech is irreversible in a way text is not. A text agent that displays payment confirmed prematurely can amend the message; the caller who has already heard your refund has been processed has already hung up and told their spouse. The rule is absolute: the agent never asserts that a consequential action succeeded until the tool result confirming it has actually arrived. If the result is slow, the agent narrates the wait. If the result is a failure, the agent says so plainly. The temptation to let the model presumptively confirm, because the tool almost always succeeds and it makes the conversation flow better, is exactly how systems end up making promises their backends did not keep.
Structured data is its own adversary on a phone line. Reading a confirmation code aloud requires pacing and chunking, three groups of three characters with pauses, not eleven characters in a single breath. Names and email addresses arrive from the recognizer mangled in ways that matter, and a system writing them into a database needs the discipline to confirm spelling, was that K as in kite, before committing. Phonetically ambiguous inputs deserve special paranoia in domains where the ambiguity is dangerous: entire classes of medication names differ by a single syllable, and a recognizer's confident best guess is not a sufficient basis for a pharmacy transaction. Confirmation, spelled out, read back, and explicitly agreed to, is not conversational overhead; it is the voice channel's equivalent of a review screen.
The threat model, stated plainly
Every security property of a voice agent starts from one uncomfortable fact: the primary input to the system is whatever sound arrives on the line, produced by whoever happens to be speaking, and none of it is trusted. Prompt injection does not require a crafted webpage here; it requires a caller who says the right sentence. A voice agent with tools and data access will be probed by callers who ask it to ignore its instructions, who claim to be supervisors, employees, or the account holder's spouse, and who patiently social-engineer it across multiple turns the way they have social-engineered human call center staff for decades. The system's instructions, its tool access, and its escalation rules all have to be designed on the assumption that some fraction of callers are adversaries with unlimited patience and a free phone.
Voice adds an attack surface text never had: the voice itself. Speech synthesis has reached the point where a convincing clone of a specific person's voice can be produced from a small sample, which means a voice on the line that sounds like the account holder is evidence of nothing. A voice agent must never treat voice familiarity, or caller ID, which is trivially spoofed, as authentication. Identity has to come from something stronger: verified callbacks to a number on file, one-time codes sent to a registered device, knowledge checks used with full awareness of their weakness, or step-up authentication for anything consequential. The corollary cuts both ways, because the same cloning technology makes outbound scam calls cheap to industrialize, which is why a legitimate voice agent should always disclose that it is an AI at the start of a call, both because a growing number of jurisdictions require it and because the alternative, a caller discovering mid-call that the person they trusted is software, destroys more trust than the disclosure ever would.
Then there is the data the channel itself generates. Voice calls are recordings of people speaking, frequently about health, money, and identity, and a voice agent pipeline is a machine for producing exactly the transcripts and audio that privacy regulation exists to govern. Call recording consent varies by jurisdiction and sometimes requires all parties to agree. Transcripts containing card numbers, health details, and government identifiers need the same retention discipline, redaction, and access control as any other sensitive store, and the agent itself needs rules about what it reads aloud: an agent that recites the full address on file to an unverified caller has just turned a lookup tool into a data breach with a friendly voice. The safe pattern is confirmation rather than disclosure, asking the caller to provide the detail and checking it against the record, never the reverse.
A worked example: the pharmacy refill line
Consider a task that voice automation has circled for years: a pharmacy's refill line. A caller wants to refill a prescription, a request that is high-volume, mostly routine, occasionally dangerous, and perfectly representative of the trade-offs above. Nothing about it requires a screen, and the population calling includes exactly the people worst served by apps, which is why the phone line still carries the volume.
A well-built agent answers by identifying itself as an automated assistant and asking what the caller needs. When the caller says they need a refill on their blood pressure medication, the agent begins with identity, not inventory: it asks for the caller's date of birth and the phone number on the account, verifies both against the record, and offers escalation to a human at the first sign of friction rather than the fifth. Only after verification does it look up active prescriptions, and when it finds two blood pressure medications on file, it does not guess. It names both, slowly, and asks which one, because medication names are a minefield of near-homophones and the cost of a wrong guess is not a bad user experience, it is a health incident.
The refill itself is a consequential action and gets the full ceremony: the agent states the medication, the dosage, and the pickup pharmacy, asks for an explicit yes, and only then submits, waiting for the pharmacy system's confirmation before telling the caller anything succeeded. If the prescription has no refills remaining, the agent does not improvise clinical advice; it explains that a renewal request will be sent to the prescriber and offers a human pharmacist for anything further. Anything that drifts toward symptoms, dosage changes, or drug interactions triggers the same boundary, because the line between logistics and medical advice is one an agent must never discover it has crossed only after crossing it. The call ends with a text message summary to the number on file, turning an ephemeral spoken agreement back into something the caller can verify at leisure.
How these systems actually fail
The failure patterns of deployed voice agents are consistent enough to catalogue. Transcription cascades top the list: a single misheard digit in a phone number, a name transcribed into a different name, an account number with two characters swapped, and every downstream action executes flawlessly against the wrong target. The recognizer's confidence score rarely saves you, because recognizers are frequently confident and wrong on exactly the short, information-dense utterances that matter most. The defense is structural, not acoustic: read back anything that will be written down, and treat unconfirmed transcription of critical fields as unverified input, because that is what it is.
Hallucinated success is the failure with the worst blast radius: the agent tells the caller something was done when it was not, a refund initiated, an appointment booked, a cancellation processed, because the model generated the confirmation instead of waiting for the tool result. Endpointing failures produce the most complaints per incident, callers cut off mid-sentence, thoughts steamrolled, and they compound because an interrupted caller speaks faster and more irritably, which degrades recognition further. Recognition quality itself is not uniform: accuracy drops for accents underrepresented in training data, for noisy environments, for older callers, and for speech differences, which means a voice agent's failure rate is not evenly distributed across the population and a team that only tests with its own voices will discover this from its angriest users rather than its dashboards.
The rest of the catalogue is operational. Retry loops, where the agent asks a caller to repeat themselves a third and fourth time instead of escalating after the second failure, teach callers that persistence is pointless. Latency spirals under load push response gaps past the threshold where callers start talking over the agent, which triggers barge-in handling, which fragments the conversation further. Synthesis mispronounces exactly the vocabulary the domain cares most about, drug names, local street names, the company's own product names, until someone maintains a pronunciation dictionary. And underneath all of it sits the asymmetry that makes voice unforgiving: a text chat user can scroll up to re-read what was agreed; a caller has nothing but memory, which is why every consequential outcome should leave a written trail, a text, an email, somewhere the ephemeral conversation becomes a durable record.
A caller cannot scroll back. Whatever the agent said is already gone, which means whatever the agent does had better match it.
Evaluating a system you cannot scroll back
Evaluating a voice agent with the tools built for text agents measures the wrong thing. Word error rate, the standard recognizer metric, correlates surprisingly weakly with task success, because a transcript can be ninety-five percent accurate while the five percent it missed was the account number, and a transcript full of small errors in filler words can still carry a perfectly recoverable request. The unit of evaluation has to be the call, not the transcript: did the caller accomplish what they called for, how long did it take, how many times did they repeat themselves, did they ask for a human, and did they hang up mid-task. Those outcomes are measurable at scale from call logs, and they are the numbers that should decide whether a change ships.
The test suite for a voice agent has to include dimensions text systems never worry about. Latency belongs in the regression suite, measured at the ninety-fifth percentile from end of caller speech to start of agent audio, because a model upgrade that improves reasoning and adds three hundred milliseconds is frequently a net regression. Interruption handling needs synthetic tests, calls that barge in early, mid-sentence, and during tool execution, verifying that the agent yields, recovers, and keeps a coherent history. And the accent and acoustic coverage problem demands deliberate test data, recorded or synthesized across the accents, ages, and noise conditions of the real caller population, precisely because the failure modes concentrate in callers the development team does not resemble. Automated judges over transcripts catch reasoning and policy failures cheaply; a standing habit of humans listening to sampled real calls catches the failures no transcript shows, the mispronunciations, the awkward pacing, the tone that reads politely on paper and sounds sarcastic out loud.
Choosing the right architecture for your situation
The cascaded pipeline and the speech-to-speech model are not competitors so much as answers to different questions. Choose the cascade when the agent acts on the world: when there are tools to call, records to change, and policies to enforce, the transcript in the middle is the control surface where logging, filtering, confirmation gates, and debugging all live, and giving it up costs more than the latency it saves. Choose native speech-to-speech when the product is the conversation itself, language practice, companionship, coaching, roles where emotional register matters more than transactional precision, and where the model needing to hear the hesitation in a voice is the feature. Hybrid designs, a native audio front end with a text-mediated tool layer behind it, are emerging as the way to refuse the choice, at the cost of operating both stacks.
The channel constrains the architecture more than teams expect. Telephone audio is narrowband, compressed, and hostile to recognizers tuned on clean recordings; browser and app audio is wideband and clean but demands real-time transport infrastructure of its own. A voice agent for internal use, where a two-second pause costs nothing but patience, can afford heavier models and cheaper engineering; a consumer-facing line where callers hang up at the first awkward silence cannot. Language coverage, on-premise requirements in regulated industries, and the simple question of expected call volume, a system handling forty calls a day can afford per-call costs that a system handling forty thousand cannot, all belong in the architecture decision before a single component is chosen.
Cost deserves its own line in that decision, because voice multiplies every per-token price by the physics of speech. A ten-minute call is a long context of transcribed conversation, re-sent to the model on every turn, plus recognition and synthesis charges billed by the minute of audio, and the total lands an order of magnitude above what an equivalent text chat costs to serve. That arithmetic shapes architecture directly: a cheaper, faster model handling routine turns with a stronger model reserved for ambiguous or consequential moments is the same routing pattern that works in text agents, applied here with sharper incentives. It also shapes the product question underneath, because a voice agent does not have to beat a human agent on cost per call to be worth deploying, but it does have to beat the alternative of the call simply not being answered, the after-hours voicemail, the twenty-minute hold queue, the caller who gives up, and framing the economics against the true baseline rather than the idealized one is what separates deployments that get renewed from pilots that quietly end.
The case against over-engineering this
Not every phone call deserves a language model. A call flow that asks five fixed questions in a fixed order, an appointment confirmation, a delivery window notification, a survey, is a solved problem, and the solution is a modern IVR with good synthesis and reliable speech recognition on constrained answers: deterministic, testable, cheap, and impossible to prompt-inject. Rebuilding it as an open-ended agent buys flexibility the interaction does not need and pays for it with nondeterminism the operation cannot afford. The honest question is whether real callers genuinely deviate from the script, and the honest way to answer it is to listen to recordings of the existing line rather than to theorize.
Voice agents earn their complexity when conversations are actually open-ended: when callers arrive with requests that do not enumerate, contexts that matter, and follow-up questions that a tree cannot anticipate. Even then, the deployment path should be incremental, an agent shadowing the existing line before answering it, handling the single most common call type before handling all of them, and escalating generously to humans while the failure catalogue is still being written. The teams with the worst voice agent stories are, almost without exception, the ones that replaced an entire call center queue in one release and discovered their edge cases in public, at volume, with real customers on the line.
Where this is heading
The trajectory of the underlying models points somewhere specific: toward native audio systems that overlap speech the way people do, backchanneling while the caller talks, starting a response before the turn fully ends, and reading the paralinguistic channel, stress, hesitation, sarcasm, that current pipelines mostly discard. Full-duplex conversation, where both parties can speak simultaneously and the system navigates it gracefully, is the capability that will make current turn-based agents feel as dated as the phone trees they replaced, and early versions of it already exist in research systems. As that arrives, the awkward machinery of endpointing and barge-in handling, the hardest-won engineering in today's stacks, gradually becomes the model's problem rather than the plumbing's.
The surrounding infrastructure is consolidating in parallel. Real-time transport, session management, and telephony integration are becoming managed layers rather than bespoke projects, the same trajectory tool connectivity followed for text agents, and on-device speech models are approaching the quality where the latency-sensitive front of the pipeline can run locally, with privacy and resilience benefits that matter for cars, homes, and healthcare. What has not consolidated is the accountability layer: shared standards for disclosing that a voice is synthetic, for verifying identity over an untrusted audio channel, and for auditing what an agent said versus what it did. Regulation is arriving faster in voice than in text, because the harms, cloned voices, undisclosed bots, recorded medical conversations, are concrete and legible to lawmakers, and teams building now should assume disclosure and consent requirements only tighten from here.
Getting started without overbuilding it
For a team building its first voice agent, the pragmatic path is narrower than the survey above suggests. Pick one call type with real volume and bounded consequences, refills, reservations, status checks, order lookups, and build for it alone, with a cascaded pipeline assembled from managed components rather than anything trained or hosted in-house. Set the latency budget before choosing any component, and measure time-to-first-audio at the ninety-fifth percentile from the first day, because latency problems discovered after launch are architecture problems, while latency problems discovered before launch are configuration problems.
Make escalation to a human a first-class feature rather than a failure state: staffed, fast, and triggered generously, after the second failed understanding, on any request for a person, and on any topic outside the agent's lane. Disclose the agent is an AI at the top of every call. Gate every consequential action behind an explicit spoken confirmation, and never let the agent claim success before the backing system has confirmed it. Record, transcribe, and actually review calls, especially the ones that ended in escalation or hang-up, because the failure catalogue for your specific caller population cannot be predicted from anyone else's. And send a written confirmation of anything that matters, because spoken words are gone the moment they land.
Voice is where agents stop being a feature of software and start being present in the world, on the phone line a customer already knows how to use, in the truck cab, in the kitchen, in the clinic waiting room. The models are already good enough to hold the conversation; whether the resulting system deserves the trust callers automatically extend to a voice is decided by everything wrapped around them, the turn-taking that does not interrupt, the confirmations that are never presumed, the identity checks that do not trust a familiar-sounding voice, and the honest, immediate path to a human when the conversation outgrows the machine.