How voice-to-voice latency works in voice agents
Learn what voice-to-voice latency measures and why capable models respond more slowly
Voice-to-voice latency is the gap between when a caller finishes speaking and when your assistant starts responding. It is a major factor in whether a conversation feels natural or frustrating. This page explains how Vapi measures latency and how to compare models.
Voice-to-voice latency sequence
Voice-to-voice latency combines five stages. After the caller stops speaking, each stage begins when the previous one ends.
- Transport: The caller’s audio travels over the network for processing.
- Transcription: The transcriber finishes converting the caller’s last words to text.
- Endpointing: Vapi decides that the caller has finished their turn and sends the transcript to the model.
- Model response: The model reads the transcript and begins generating a reply. Vapi measures this stage as time to first token.
- Speech synthesis: The voice provider turns the reply into audio. Vapi measures this stage as time to first audio.
A caller’s total voice-to-voice latency is roughly the sum of all five stages, plus the time the reply takes to reach the caller.
Per-turn latency on a call
Each call records one latency entry per turn in artifact.performanceMetrics.turnLatencies. The call’s Latency Summary tab, the Traffic tab, and simulation latency limits are all built from these entries. GPT-Live assistants don’t record these entries yet; measure their latency from the call’s timestamps, as described in testing GPT-Live.
Keep these points in mind when you read the numbers:
-
The fields don’t add up to
turnLatency. TheturnLatencyfield istranscriberLatencyplusendpointingLatencyplus the time from when Vapi decides the turn is over to the start of the reply. Model and voice latency happen inside that last stretch, alongside other work such as handing the turn to the model and buffering the reply audio before it plays. -
Model and voice latency include more than the provider. The
modelLatencyfield includes knowledge base retrieval and request routing before the provider request. ThevoiceLatencyfield includes the time spent collecting enough of the reply to send the first voice request. TheminCharactersandpunctuationBoundariessettings invoice.chunkPlancontrol that wait. The field also includes any sample-rate conversion after the provider returns audio. -
Vapi measures on its servers. None of the fields include network time to or from the caller, or playback on the caller’s device.
-
Only replies to the caller are recorded. Turns where the assistant didn’t speak, for example because the caller interrupted first, have no entry. When the assistant speaks first, its first message has no entry either. When it waits for the caller (
firstMessageModeset toassistant-waits-for-user), the first message is recorded as a reply, withendpointingLatencyas0. -
The
unmeasuredlist tells a missing value from a real0. A stage that didn’t run or wasn’t measured on a turn is recorded as0, and the entry’sunmeasuredarray names it. A0for a component that isn’t listed is a real 0 ms. The array is left out when every component was measured. Entries recorded beforeunmeasuredexisted don’t have it, so a0there can mean either. A component is listed when:transcriber: voice activity detection saw no speech for the turn, more than 10 seconds passed between the caller’s last detected speech and the transcript, or the speech that detection last saw had already been answered by an earlier turn. Those last two usually mean detection missed this speech. It’s also listed when the transcript’s text was first seen before the caller started this speech, or when detection still hadn’t confirmed the end of speech when the reply started. That last case includes a caller who kept talking after a short pause. In that case,transcriberLatencyis measured from when the speech started or from the pause they talked through, so it can be above0.endpointing: no caller transcript was sent to the model that turn, for example the first message or a reply to a tool result, or the transcript’s text was first seen before the caller started this speech.model: the model sent no output token that turn, for example when the first audio is a static tool message or a guardrail held the reply.voice: there was no model output to time the voice from.
When
transcriberLatencyis unmeasured,turnLatencyisn’t a real measurement either: it leaves that interval out, or counts it from when the speech started. -
Averages count every turn. The call’s
modelLatencyAverage,voiceLatencyAverage, and other averages divide by every entry, including those zeros. Simulation latency limits leave out turns that list the metric inunmeasured, so a simulation’s result can be higher than the call’s average for the same call. Theartifact.performanceMetrics.turnLatencyUnmeasuredTrackedfield istrueon calls recorded withunmeasured. On those calls, an entry without theunmeasuredlist had every component measured. On older calls, which don’t haveturnLatencyUnmeasuredTracked, simulation latency limits drop a0formodelLatencyorvoiceLatencyand take notranscriberLatencyorendpointingLatencyvalues at all.
What the displayed latency measures
Performance metrics show latency for each component. The displayed total adds the median latency for the transcriber, model, and voice. This should be treated as a rough estimate.
Vapi calculates each median from production calls, which reflect a typical call - half of calls are faster and half are slower. Individual calls can vary.
The displayed total excludes endpointing and transport time. Both contribute to the delay a caller experiences. Endpointing can account for a meaningful share of that delay. Improving endpointing can make an assistant feel more responsive even though it is not included in the displayed total.
The displayed total is a sum of component medians across many calls, so it isn’t comparable to a call’s turnLatency. A call measures turnLatency end to end on each turn, including endpointing. For more information, see per-turn latency on a call.
Model capability and latency
More capable models usually respond more slowly. A model’s time to first token is often the largest component of voice-to-voice latency. More capable models tend to be larger and take longer to begin generating a response.
Ultra Fast uses speed-optimized models for lower latency. High Intelligence accepts higher latency for greater capability. Neither preset is always better than the other; choose based on whether your use case needs faster responses or stronger reasoning.
How to judge latency
Use the displayed latency to compare models and then test the assistant in a real conversation.
- Lower latency improves the experience only until callers stop noticing the difference.
- People naturally pause between turns. A response can feel natural without being instant.
- Endpointing, turn-taking, and silence handling all affect how latency feels.
Testing the assistant in a real conversation provides the most reliable evaluation. A model with slightly higher latency can still feel natural. Choosing the lowest number may sacrifice capability that your use case needs.
Other latency figures may use ideal lab conditions or measure only one component, while Vapi reports medians from actual production calls. Compare figures only when their scope and conditions match.