Production voice AI agent architecture: the 2026 reference

Most voice AI demos are lies of omission. They run on a stable connection, with a cooperative speaker, on the two or three happy-path utterances the demo was built around. Production is none of those things: callers interrupt, networks jitter, accents vary, and the one thing the agent must never do — invent a ticket number or transfer into a dead queue — is exactly what a language model wants to do by default. This is the reference architecture we deploy when a voice agent has to answer real customers, not a stage.

It builds on our earlier write-up of how AI is replacing call centers, but goes a layer deeper: this is the pipeline, the numbers, and the failure modes, in the order they will hurt you.

The six-component pipeline

A production voice agent is not a model. It is a pipeline of six components, each with its own latency and failure characteristics. Get the wiring wrong and it demos beautifully, then falls over in week one.

• Telephony / SIP — the carrier edge: inbound trunking, an SBC, barge-in, DTMF, and codec negotiation. This is where most demos never went.
• Speech-to-text (STT) — streaming, partial-hypothesis transcription tuned to your callers' accents, not a US-English benchmark.
• Orchestration — the state machine that owns the turn: when to listen, when to speak, when to call a tool, when to hand off.
• Language model — Claude or an OpenAI GPT model, constrained to tool calls for anything factual and streamed token-by-token.
• Retrieval & memory — short-term conversation state plus RAG over your knowledge base, scoped per tenant.
• Text-to-speech (TTS) — streaming synthesis that starts speaking before the model has finished thinking.

The unglamorous ones break first. Telephony and barge-in are where real networks defeat prototypes; the tool layer is where 'AI proof of concept' turns into 'integration project that overran'. The model is rarely the hard part.

The latency budget is the product

Response latency is not a metric you optimise later — it is the product. Over roughly 800ms of dead air, a caller assumes the line dropped and starts talking over the agent. The target is sub-500ms from end-of-speech to first audio back. That number has to be assembled from a budget, because every component spends some of it.

End-of-speech to first audio out  (target < 500 ms)
  endpointing / VAD ...............  ~120 ms  (detect the caller stopped)
  final STT hypothesis ...........   ~90 ms
  model time-to-first-token ......  ~180 ms
  TTS time-to-first-audio ........   ~90 ms
  network / jitter buffer ........   ~60 ms
  -------------------------------------------
  budget spent ...................  ~540 ms  -> already over; you must overlap

Summed serially you are already over budget. The only way under 500ms is to overlap: stream partial STT into the model, start the model before the caller has fully stopped where endpointing is confident, and begin TTS on the first sentence while the model is still generating the second. Latency is won by pipelining, not by a faster model.

Endpointing is the hidden lever

The single biggest perceived-latency win is endpointing — deciding the caller has finished. Too eager and you cut people off mid-sentence; too patient and every turn feels sluggish. Semantic endpointing (using a small model to judge whether the utterance is complete) beats a fixed silence timeout by a wide margin, especially for callers who pause to think.

Barge-in and turn-taking

Humans interrupt. When the caller starts speaking while the agent is talking, the agent must stop within ~150ms, flush the queued TTS audio, and re-open the microphone. Skip this and the agent talks over an angry customer — the fastest way to lose trust on a call. Barge-in has to be handled at the audio layer, not the application layer, or the stop is always a beat too late.

Tool calling without hallucinated IDs

Language models will happily tell a caller 'your reference is RMC-485219' — a number that exists nowhere. The rule is absolute: the model never states an identifier, balance, appointment, or status it did not receive from a tool call. Everything factual goes through typed functions that read and write your real systems; the model's job is to decide which function to call and how to phrase the result.

// Typed tools are the guardrail. The model may call these; it may not
// invent their outputs. Every factual claim traces to a tool result.
const tools = [
  defineTool("lookupAccount", { phone: "string" }, lookupAccount),
  defineTool("createTicket", { accountId: "string", summary: "string" }, createTicket),
  defineTool("bookCallback", { accountId: "string", windowStart: "string" }, bookCallback),
];

// System prompt, non-negotiable clause:
// "Never state an account number, ticket ID, balance, or booking time
//  unless it was returned by a tool in this conversation. If you do not
//  have it, call the tool or say you will follow up."

Human handover done right

The measure of a good voice agent is not how many calls it handles alone — it is how cleanly it hands off the ones it shouldn't. A warm transfer carries the full context (who the caller is, what they wanted, what has been tried) to the human so the customer never repeats themselves. Escalation triggers should include detected frustration, high-value accounts, legal or vulnerability signals, and simple low confidence. Handover is architecture, not an afterthought.

Evaluation and observability

You will ship a voice agent that works, and three weeks later it will be quietly worse — a prompt tweak, a model update, a new call type. Without continuous evaluation you find out from a complaint. Every call should be transcribed, scored against a rubric (task success, hallucination, escalation appropriateness, latency), and sampled for anomalies, with alerts when scores drop. We treat this as its own subsystem; it is the difference between a demo and something you can leave running. If you are building that layer, our piece on building an LLM evaluation harness for production agents is the companion to this one.

What it costs

A production voice agent runs around £0.18 per call in 2026 — speech-to-text, model tokens, speech synthesis, telephony, and infrastructure summed across a roughly four-minute average call. Against a UK loaded human cost near £3.20 per handled call, that is the gap that makes the engineering worth it. The full breakdown, and the deflection caveats that turn a per-call unit cost into a realistic blended saving, are on our methodology page — the per-call gap is not the number you bank.

// Per-call economics (illustrative, 2026)
const perCallHuman = 3.20;  // UK loaded, tier-1 inbound
const perCallAI    = 0.18;  // STT + LLM + TTS + telephony + infra

// The honest number is blended, not the raw unit gap:
const deflection   = 0.5;   // realistic tier-1 deflection: 40-60%
const blendedSaving = deflection * (perCallHuman - perCallAI);
// ~ £1.51 saved per inbound call at 50% deflection

When voice AI is the wrong answer

It is not the right tool for high-emotional-load calls (bereavement, complaints with legal exposure, vulnerable customers), for ultra-low call volumes where the build never pays back, or where the underlying systems are too broken for the agent to read and write. Deploying it there is how you get a cost saving on a spreadsheet and a reputation cost in the real world.

“The hard part of a voice agent was never the model. It was the 150 milliseconds, the tool boundary, and the handover — the parts no demo shows you.”

If you are scoping voice automation for a call centre, our call-centre automation solution is where this architecture is packaged into a fixed-scope engagement — and a scoping call will run the cost model against your real handle times and volumes, not a slide.