Voice AI Agents
Voice agents that hold a natural conversation on the phone, engineered as real-time systems with an explicit latency budget, proper turn-taking, and an escalation path that never leaves a caller trapped.
Text agents are judged on whether the answer was right. Voice agents are judged in the first four hundred milliseconds, before the caller has heard anything useful at all. That makes voice a real-time systems problem wearing a language model costume, and it is why voice projects fail for reasons that have nothing to do with the model's intelligence.
A voice AI agent is a system that conducts a spoken conversation in real time: transcribing the caller continuously, deciding when they have finished speaking, generating a response, speaking it back with natural timing, handling interruption, taking actions in your systems, and handing over to a person when it should.
The latency budget is the architecture
Every component in a voice pipeline spends time, and the total is what the caller experiences as awkwardness. We design against an explicit budget, typically well under a second from end of speech to first audio, and every architectural choice is measured against it:
| Stage | Typical share of the budget | Where time is won |
|---|---|---|
| Endpointing | Decides when the caller has finished | Tuned silence and semantic cues, not a fixed timer |
| Transcription | Streaming, partial results | Streaming rather than waiting for the final transcript |
| Reasoning and tools | The largest and most variable share | Short context, cached lookups, parallel tool calls |
| Speech synthesis | First audio, then streaming | Streaming synthesis, starting on the first clause |
| Network and telephony | Fixed overhead | Region placement, persistent connections |
Where a tool call is genuinely slow, the agent says something natural while it waits rather than leaving silence. That single behaviour changes the perceived quality of the system more than any improvement in the model.
What we build into a voice agent
Turn-taking that feels human
Knowing when someone has finished speaking is harder than it sounds: people pause mid-sentence, think aloud, and read numbers slowly. We tune endpointing on your real calls rather than on a default, because a system that interrupts callers is worse than one that is half a second slower.
Barge-in as a first-class feature
The caller must be able to interrupt at any moment, and the agent must stop immediately, discard what it was going to say and listen. Systems that talk over people get abandoned within one call, and retrofitting proper barge-in is considerably harder than designing for it.
Design for transcription error
Names, postcodes, reference numbers and accented speech will be transcribed imperfectly. The agent confirms critical values explicitly, uses phonetic confirmation where it matters, validates against your systems rather than trusting the transcript, and never takes an irreversible action on an unconfirmed value.
Actions with the same controls as any agent
A voice agent that can look things up and make changes is an agent, and it inherits the same requirements: scoped tools, idempotent writes, bounded loops and full tracing, as described under custom AI agent development. The audio channel does not lower the bar.
Escalation that respects the caller
A clear path to a person on request, on repeated failure, on detected distress and on any topic outside scope. The handover carries the transcript and the context so the caller does not repeat themselves, which is the single most common complaint about automated phone systems and the easiest to fix.
Callers forgive a slower agent, not a rude one
In review after review, the behaviours that sink a voice deployment are interrupting, ignoring a request for a human, and asking for information the caller has already given. None of those are model quality problems, and all of them are design decisions.
Compliance and consent
Voice carries obligations that text does not, and they are not optional:
- Disclosure that the caller is speaking to an automated system, at the start, in plain language.
- Recording and transcription consent handled to the rules of each jurisdiction you operate in.
- Retention and deletion policy for audio and transcripts, applied automatically rather than by intention.
- Accessibility, including the ability to reach a person without navigating a maze first.
- Where a voice is cloned or licensed, the rights to use it, documented.
We build these in and hand over the evidence, because they are the questions your legal team will ask before launch rather than after.
How the engagement runs
Latency and turn-taking are proven on real calls before the conversation design is finished.
Call analysis and scope
Real call recordings analysed, the intents in scope chosen by volume and containment potential, and the latency budget set.
Real-time pipeline
Streaming transcription, endpointing tuned on your calls, streaming synthesis and barge-in, measured against the budget.
Conversation and actions
Dialogue design, confirmation behaviour for critical values, scoped tools for lookups and changes, and the escalation path.
Pilot on live traffic
A limited share of real calls, with transcript review, containment and satisfaction measured against the baseline.
Rollout and handover
Graduated traffic increase, monitoring, and handover of the pipeline, evaluation sets and runbooks.
What you receive
A voice agent measured on containment and caller experience, not on how impressive the demo sounded.
Voice agent
The deployed real-time pipeline with telephony integration, deployed as infrastructure as code.
Latency report
Measured time per stage against the budget, on your infrastructure and your call paths.
Conversation design
Intents in scope, confirmation behaviour, failure phrasing and the escalation rules.
Action tool layer
Scoped lookups and writes with idempotency, validation and full tracing.
Compliance pack
Disclosure, consent, retention and accessibility behaviour, with the evidence for review.
Quality dashboard
Containment, escalation reasons, average handle time and caller satisfaction against the baseline.
Is this the right engagement?
Worth being direct. Voice AI Agents is the wrong spend in some situations, and those are listed rather than buried.
Good fit if
- Call volume is high and dominated by a small number of repetitive intents.
- Callers currently wait, or abandon, at times you can identify.
- The information the agent needs is reachable from your systems in real time.
- You accept a graduated pilot on a share of live calls.
- A fast, respectful path to a human is acceptable and expected.
Choose something else if
- Calls are emotionally sensitive or high-stakes enough that automation is inappropriate.
- The intents are highly varied with no dominant patterns.
- The systems the agent would need cannot answer within the latency budget.
- What you need is chat rather than voice, which is a considerably simpler build.
Frequently asked questions
Marked up with FAQPage schema so these answers can surface directly in search results and inside AI assistant responses.
How fast does a voice AI agent need to respond?
Well under a second from the caller finishing to the first audio, and consistently. Beyond that the conversation starts to feel wrong even when the answers are right, which is why we treat the latency budget as the architecture rather than as a tuning exercise at the end.
Can a voice agent take actions, not just answer questions?
Yes, and that is where most of the value sits: checking an order, rescheduling, updating a detail, raising a case. Those actions carry the same requirements as any agent, plus explicit confirmation of critical values because transcription is never perfect.
What happens when the caller asks for a human?
They get one, immediately, with the transcript and context passed across so they do not repeat themselves. We also escalate automatically on repeated failure, on detected distress and on any topic outside scope. Trapping callers is the fastest way to lose the goodwill the deployment depends on.
How do you handle names, postcodes and reference numbers?
By assuming the transcript is wrong until confirmed. Critical values are read back, validated against your systems, and confirmed phonetically where the stakes justify it. No irreversible action is taken on an unconfirmed value.
Do we have to tell callers they are speaking to an AI?
In most jurisdictions yes, and we recommend it regardless. Disclosure at the start costs nothing in containment, and the alternative is a complaint and a regulatory question you do not want. Consent, recording and retention are handled per jurisdiction and documented for your legal team.
Often paired with this
Most clients combine two or three engagements from the AI Agents & Agentic Automation pillar. These are the ones that most often run immediately before or after.
AI Customer Support Agents
Support agents that resolve tickets end to end, with real actions, grounded answers and honest escalation.
Read more →Custom AI Agent Development
An agent that completes work in your systems, with scoped tools, trajectory evaluation and a circuit breaker.
Read more →AgentOps: Monitoring, Evaluation and Guardrails
The operations layer for production agents: trajectory tracing, evaluation, spend ceilings and a tested kill switch.
Read more →Is this the right engagement?
Tell us what you are trying to build. If a different service fits better, or if you do not need us at all, we will say so.