Skip to navigation

How voice-to-voice latency works in voice agents

Learn what voice-to-voice latency measures and why capable models respond more slowly

Voice-to-voice latency is the gap between when a caller finishes speaking and when your assistant starts responding. It is a major factor in whether a conversation feels natural or frustrating. This page explains how Vapi measures latency and how to compare models.

Voice-to-voice latency sequence

Voice-to-voice latency combines five stages. After the caller stops speaking, each stage begins when the previous one ends.

  1. Transport: The caller’s audio travels over the network for processing.
  2. Transcription: The transcriber finishes converting the caller’s last words to text.
  3. Endpointing: Vapi decides that the caller has finished their turn and sends the transcript to the model.
  4. Model response: The model reads the transcript and begins generating a reply. Vapi measures this stage as time to first token.
  5. Speech synthesis: The voice provider turns the reply into audio. Vapi measures this stage as time to first audio.

A caller’s total voice-to-voice latency is roughly the sum of all five stages, plus the time the reply takes to reach the caller.

Per-turn latency on a call

Each call records one latency entry per turn in artifact.performanceMetrics.turnLatencies. The call’s Latency Summary tab, the Traffic tab, and simulation latency limits are all built from these entries. GPT-Live assistants don’t record these entries yet; measure their latency from the call’s timestamps, as described in testing GPT-Live.

FieldStarts whenEnds when
transcriberLatencyThe caller’s last voice activity, normally when they stop speakingThe transcriber returns the words of that speech
endpointingLatencyThe transcriber returns the wordsVapi decides the turn is over and sends it to the model
modelLatencyThe turn’s model stage starts, before knowledge base retrieval and request routingThe model returns its first output token
voiceLatencyThe model returns its first output tokenThe first reply audio reaches Vapi’s turn-taking buffer, after any sample-rate conversion
turnLatencyThe caller’s last voice activityVapi starts sending the reply audio to the caller

Keep these points in mind when you read the numbers:

  • The fields don’t add up to turnLatency. The turnLatency field is transcriberLatency plus endpointingLatency plus the time from when Vapi decides the turn is over to the start of the reply. Model and voice latency happen inside that last stretch, alongside other work such as handing the turn to the model and buffering the reply audio before it plays.

  • Model and voice latency include more than the provider. The modelLatency field includes knowledge base retrieval and request routing before the provider request. The voiceLatency field includes the time spent collecting enough of the reply to send the first voice request. The minCharacters and punctuationBoundaries settings in voice.chunkPlan control that wait. The field also includes any sample-rate conversion after the provider returns audio.

  • Vapi measures on its servers. None of the fields include network time to or from the caller, or playback on the caller’s device.

  • Only replies to the caller are recorded. Turns where the assistant didn’t speak, for example because the caller interrupted first, have no entry. When the assistant speaks first, its first message has no entry either. When it waits for the caller (firstMessageMode set to assistant-waits-for-user), the first message is recorded as a reply, with endpointingLatency as 0.

  • The unmeasured list tells a missing value from a real 0. A stage that didn’t run or wasn’t measured on a turn is recorded as 0, and the entry’s unmeasured array names it. A 0 for a component that isn’t listed is a real 0 ms. The array is left out when every component was measured. Entries recorded before unmeasured existed don’t have it, so a 0 there can mean either. A component is listed when:

    • transcriber: voice activity detection saw no speech for the turn, more than 10 seconds passed between the caller’s last detected speech and the transcript, or the speech that detection last saw had already been answered by an earlier turn. Those last two usually mean detection missed this speech. It’s also listed when the transcript’s text was first seen before the caller started this speech, or when detection still hadn’t confirmed the end of speech when the reply started. That last case includes a caller who kept talking after a short pause. In that case, transcriberLatency is measured from when the speech started or from the pause they talked through, so it can be above 0.
    • endpointing: no caller transcript was sent to the model that turn, for example the first message or a reply to a tool result, or the transcript’s text was first seen before the caller started this speech.
    • model: the model sent no output token that turn, for example when the first audio is a static tool message or a guardrail held the reply.
    • voice: there was no model output to time the voice from.

    When transcriberLatency is unmeasured, turnLatency isn’t a real measurement either: it leaves that interval out, or counts it from when the speech started.

  • Averages count every turn. The call’s modelLatencyAverage, voiceLatencyAverage, and other averages divide by every entry, including those zeros. Simulation latency limits leave out turns that list the metric in unmeasured, so a simulation’s result can be higher than the call’s average for the same call. The artifact.performanceMetrics.turnLatencyUnmeasuredTracked field is true on calls recorded with unmeasured. On those calls, an entry without the unmeasured list had every component measured. On older calls, which don’t have turnLatencyUnmeasuredTracked, simulation latency limits drop a 0 for modelLatency or voiceLatency and take no transcriberLatency or endpointingLatency values at all.

What the displayed latency measures

Performance metrics show latency for each component. The displayed total adds the median latency for the transcriber, model, and voice. This should be treated as a rough estimate.

Vapi calculates each median from production calls, which reflect a typical call - half of calls are faster and half are slower. Individual calls can vary.

The displayed total excludes endpointing and transport time. Both contribute to the delay a caller experiences. Endpointing can account for a meaningful share of that delay. Improving endpointing can make an assistant feel more responsive even though it is not included in the displayed total.

The displayed total is a sum of component medians across many calls, so it isn’t comparable to a call’s turnLatency. A call measures turnLatency end to end on each turn, including endpointing. For more information, see per-turn latency on a call.

Model capability and latency

More capable models usually respond more slowly. A model’s time to first token is often the largest component of voice-to-voice latency. More capable models tend to be larger and take longer to begin generating a response.

Ultra Fast uses speed-optimized models for lower latency. High Intelligence accepts higher latency for greater capability. Neither preset is always better than the other; choose based on whether your use case needs faster responses or stronger reasoning.

How to judge latency

Use the displayed latency to compare models and then test the assistant in a real conversation.

  • Lower latency improves the experience only until callers stop noticing the difference.
  • People naturally pause between turns. A response can feel natural without being instant.
  • Endpointing, turn-taking, and silence handling all affect how latency feels.

Testing the assistant in a real conversation provides the most reliable evaluation. A model with slightly higher latency can still feel natural. Choosing the lowest number may sacrifice capability that your use case needs.

Other latency figures may use ideal lab conditions or measure only one component, while Vapi reports medians from actual production calls. Compare figures only when their scope and conditions match.