Latency is the whole experience in voice AI
A one-second pause feels broken on a phone call and fine in chat. Where the delay comes from, what is fixable, and what you trade away to fix it.
In chat, a two-second wait is invisible. On a phone call it is a held breath, and by three seconds the person has said "hello?" and started talking over you. Latency is not a performance metric in voice AI. It is the experience.
Where the delay comes from
Every turn passes through four stages, and they add up in series:
- Endpointing — deciding the caller has finished speaking. Frequently the largest single contributor, and the one nobody talks about.
- Transcription — audio to text.
- The model — thinking, plus any function call. A lookup against your own slow API lands here, and it is often your fault rather than the vendor's.
- Speech synthesis — text back to audio.
Ask a vendor for the breakdown, not the total. A total tells you nothing about what you can change.
The one nobody expects: endpointing
The system has to decide when you have stopped talking. Wait too long and it feels sluggish. Wait too little and it interrupts you mid-sentence, which is far more annoying.
This gets harder with exactly the callers you care about — people who pause to think, who say "umm", who read a number off a card. Tune it too aggressively for snappiness and your agent starts talking over an elderly customer reading their order number. That trade is real and it does not have a setting that is right for everyone.
What you can actually change
- Shorten the first response. Streaming means the caller hears the opening words while the rest is still generating. A reply that starts "Sure —" buys you most of a second.
- Shorten the prompt. Long system prompts are re-processed every turn. Move facts into the knowledge base — structured for retrieval.
- Make your functions fast. If your order-lookup API takes 1.8 seconds, no vendor can fix that. Cache it, or acknowledge before calling it.
- Fill the gap out loud. "Let me check that for you" before a slow lookup turns dead air into normal conversation. This is the cheapest and most effective fix available.
- Right-size the model. Your most capable model is rarely worth a second of silence for an hours-and-directions question.
Barge-in is not optional
People interrupt. If the caller starts talking, the agent must stop immediately — mid-word, not mid-sentence. An agent that finishes its paragraph while the caller is talking over it reads as rude in a way that no amount of voice quality repairs.
Test this specifically. It is the single most common thing that works in a demo and fails on a real call, because demo callers are polite and real callers are not.
How to test it honestly
Not on your office wifi with the vendor on the line. Test:
- From a mobile on a normal cellular connection.
- With background noise — a street, a shop, a car.
- With someone who pauses mid-sentence.
- With an interruption three words into the agent's reply.
- On the function that hits your slowest internal system.
Then listen to the recordings rather than reading the transcripts. Transcripts hide every timing problem there is — the text of a badly-timed conversation looks perfect.
A slightly worse answer delivered promptly beats a better one after two seconds of silence. On a phone call, hesitation reads as incompetence.
Voice quality is a plan tier in Convarza — standard through ultra-realistic — and the model is set per bot, so you can put a fast, cheap agent on the hours-and-directions line and a better one where the conversation earns it. How that is configured.