How voice-to-voice latency works in voice agents
How voice-to-voice latency works in voice agents
Learn what voice-to-voice latency measures and why capable models respond more slowly.
Voice-to-voice latency is the gap between when a caller finishes speaking and when your assistant starts responding. It is a major factor in whether a conversation feels natural or frustrating. This page explains how Vapi measures latency and how to compare models.
Voice-to-voice latency sequence
Voice-to-voice latency combines five stages that run in sequence. Each stage begins after the previous one.
- Transport: The caller’s audio travels over the network for processing.
- Endpointing: Vapi detects that the caller has finished speaking.
- Transcription: The transcriber converts the caller’s speech to text.
- Model response: The model reads the transcript and begins generating a reply. Vapi measures this stage as time to first token.
- Speech synthesis: The voice provider turns the reply into audio. Vapi measures this stage as time to first audio.
A caller’s total voice-to-voice latency is roughly the sum of all five stages.
What the displayed latency measures
Performance metrics show latency for each component. The displayed total adds the median latency for the transcriber, model, and voice. This should be treated as a rough estimate.
Vapi calculates each median from production calls, which reflect a typical call - half of calls are faster and half are slower. Individual calls can vary.
The displayed total excludes endpointing and transport time. Both contribute to the delay a caller experiences. Endpointing can account for a meaningful share of that delay. Improving endpointing can make an assistant feel more responsive even though it is not included in the displayed total.
Model capability and latency
More capable models usually respond more slowly. A model’s time to first token is often the largest component of voice-to-voice latency. More capable models tend to be larger and take longer to begin generating a response.
Ultra Fast uses speed-optimized models for lower latency. High Intelligence accepts higher latency for greater capability. Neither preset is always better than the other; choose based on whether your use case needs faster responses or stronger reasoning.
How to judge latency
Use the displayed latency to compare models and then test the assistant in a real conversation.
- Lower latency improves the experience only until callers stop noticing the difference.
- People naturally pause between turns. A response can feel natural without being instant.
- Endpointing, turn-taking, and silence handling all affect how latency feels.
Testing the assistant in a real conversation provides the most reliable evaluation. A model with slightly higher latency can still feel natural. Choosing the lowest number may sacrifice capability that your use case needs.
Other latency figures may use ideal lab conditions or measure only one component, while Vapi reports medians from actual production calls. Compare figures only when their scope and conditions match.