Simulations advanced

Advanced simulation options let you set variable values, mock tool responses, trigger lifecycle webhooks, set latency limits, and reuse structured outputs. Use them after you complete the Simulations quickstart and need more control over behavior or test conditions.

How it works

Advanced options belong to each simulation in a suite. They control which values the assistant or squad receives, what mocked tools return, and which lifecycle events send webhooks.

Open a suite and select Edit, then Next. The review step contains the Success criteria, Variables, and Tool mocks & webhooks tabs.

Mock tool responses

During a simulation, the assistant or squad under test runs its real tools. Mock a tool to return fixed text instead, so that tool never calls its live service or API. A mock intercepts only the tool it names; every other tool the assistant or squad calls still runs for real.

Dashboard

cURL

On the review step, open Tool mocks & webhooks. Under Tool mocks, select Add, then configure each mock with three fields:

• Tool: Select one of the tools configured on the assistant or squad. Only its configured tools, including Function tools, can be mocked.
• Mock result: Enter the string returned in place of the tool’s live output.
• Enabled: Toggle the mock on or off. A disabled mock is ignored, and the tool runs live.

The mock result is always passed to the assistant or squad as a string. In the Dashboard, enter plain text or JSON-formatted text directly. In a cURL JSON body, escape the quotation marks inside JSON-formatted text, as shown in the example. Use tool mocks to keep tool behavior deterministic, avoid side effects, and steer the conversation down a specific path without depending on a live service.

Common mock patterns

Choose a result that leads the assistant or squad through the behavior you want to test. These examples show how the result field appears in a cURL JSON body:

{
  "result": "{\"status\": \"success\", \"confirmationId\": \"APT-12345\"}"
}

These strings replace the selected tool’s output. They do not produce an HTTP error, add network latency, or cause a network timeout.

Handoffs and transfers in simulations

In a simulation, a handoff with a fixed list of destinations runs only when the model picks one of them. If the model names no destination, asks for dynamic, or names an assistant that isn’t on the list, the handoff is refused. The assistant doesn’t switch, and the model is asked to try again with a listed destination. Your server isn’t asked for a destination. On a live call, a missing or dynamic destination can instead trigger a request to your server.

A dynamic handoff runs as it does on a live call, so it asks your server for a destination unless you mock it.

In a suite you build yourself, transferCall isn’t mocked automatically. In a voice simulation, an external transfer ends the simulation without dialing the phone number or SIP destination, so it doesn’t test that destination. If the tool asks your server to choose a destination, that lookup can still run. Mock the tool to avoid that request, and use a real test call to test the external transfer itself.

Gaps in generated suites

When Composer generates a suite, it lists each handoff or transfer the suite doesn’t exercise as Not tested, with the reason:

Other supported handoffs and assistant-to-assistant transfers run normally. Cover the paths above with test calls against sandbox systems.

Hooks and call endings

Simulations run the hooks on the assistant or squad under test, just like a real call. For example, a call.ending hook runs in a voice simulation. The exception is customer.speech.timeout. Those hooks are removed from the assistant and every squad member so the assistant doesn’t prompt the AI tester when the tester pauses. To test re-engagement, set targetOverrides.hooks on the scenario. Those hooks are used exactly as set.

If the AI tester hangs up first in a voice simulation, the call ends with customer-ended-call, just like a real caller hanging up. Anything that depends on a caller hang-up runs normally.

Send webhooks on run start and end

Turn on webhooks to notify your server when a simulation iteration starts or ends. Each webhook sends a POST request to the URL you configure.

Dashboard

cURL

On the Tool mocks & webhooks tab, under Webhooks, toggle Simulation start and Simulation end.

Vapi sends simulation.run.started when a simulation iteration begins. The example below is a voice payload. A chat payload omits the calls object:

{
  "type": "simulation.run.started",
  "simulationId": "8f2a9c1e-3b4d-4a7e-9c2f-1d6b0a5e7f34",
  "runId": "b7d4e2a0-6c9f-4e13-8a52-2f7c9b1d4e60",
  "simulationRunItemId": "c3e8f1a2-9d47-4b6c-8e30-5a1f2b9c7d84",
  "iterationNumber": 1,
  "isSimulation": true,
  "calls": {
    "testerCallId": "a1b2c3d4-...",
    "targetCallId": "e5f6a7b8-...",
    "listenUrl": "wss://..."
  }
}

Vapi sends simulation.run.ended when the simulation iteration finishes. The example below is a voice payload. A chat payload omits calls, endedReason, and recordingUrl:

{
  "type": "simulation.run.ended",
  "simulationId": "8f2a9c1e-3b4d-4a7e-9c2f-1d6b0a5e7f34",
  "runId": "b7d4e2a0-6c9f-4e13-8a52-2f7c9b1d4e60",
  "simulationRunItemId": "c3e8f1a2-9d47-4b6c-8e30-5a1f2b9c7d84",
  "iterationNumber": 1,
  "isSimulation": true,
  "calls": { "testerCallId": "a1b2c3d4-...", "targetCallId": "e5f6a7b8-...", "listenUrl": "wss://..." },
  "endedReason": "...",
  "canceled": false,
  "failureReason": "...",
  "startedAt": "2026-07-28T17:10:05.000Z",
  "endedAt": "2026-07-28T17:12:30.000Z",
  "transcript": "...",
  "messages": [],
  "recordingUrl": "https://..."
}

Both chat and voice payloads include simulationId, runId, simulationRunItemId, iterationNumber, and isSimulation. The ended event adds startedAt and endedAt, plus transcript and messages when you include them. The call-specific fields appear for voice simulations only: calls, endedReason, and recordingUrl. The calls object includes calls.listenUrl, a WebSocket URL for monitoring the run in real time. For the complete list of endedReason values, see Call ended reasons.

Reuse structured outputs

Each evaluation references a structured output. Reuse a structured output when several simulations should apply the same evaluation criterion. For more patterns, see structured output examples.

Set latency limits

A latency limit fails a voice simulation when the assistant or squad under test responds too slowly. Vapi measures latency on each turn of the call, combines the turns into one number, and passes the limit when that number is at or below the limit you set.

Latency limits work alongside structured-output evaluations. A scenario still needs at least one evaluation, and a required limit that fails marks the simulation as failed even when every evaluation passes.

Metrics and aggregations

Each metric reads one field of the call’s per-turn latency, measured the same way as on a live call. See per-turn latency on a call for exactly where each one starts and ends.

Model and voice latency happen inside turn latency, so the model and voice metrics don’t add up to turn. A turn that recorded 0 for model or voice didn’t measure that stage, so the metric leaves it out.

The AI tester speaks with clean synthetic audio, so transcription, endpointing, and turn latency can run faster than with real callers.

The aggregation reduces the call’s turns to one value: mean, median, p95, or max. p95 uses the nearest-rank method, so on a call with fewer than 20 turns it equals max. Measured values are rounded to the nearest millisecond before they are compared with the limit.

When a limit is skipped

Skipped limits don’t affect the result.

Review latency results

Open a run and select a simulation. The Latency section shows average turn latency when at least one turn was measured. Each limit shows its threshold and result. When at least one turn measured that limit’s metric, the result also shows the measured value and number of turns. Optional and skipped limits are labeled. When a simulation fails, the count in the results list, such as 1 of 3 evaluations failed, includes its latency limits.

Through the API, each run item’s results.latencyEvaluations holds one entry per limit with actualMs, thresholdMs, sampleCount, passed, required, and, for skipped limits, isSkipped and skipReason. results.passed accounts for required latency limits.

Set variable values for the assistant or squad

Variables provide values for the dynamic variables used by the assistant or squad during the simulation. On the Variables tab, add Name and Value pairs. Each name must match a {{variable}} placeholder in the assistant’s prompt, and the value is substituted during the run.

Use variables to test specific inputs, such as a customer name or account tier, without editing the assistant or squad.

Simulate a call from a specific phone number

Assistants that identify callers by phone number, for example by passing {{customer.number}} to an account lookup tool, need a known caller number to test against. Set customer.number in the scenario’s targetOverrides.variableValues. The Variables tab only accepts flat names, so set this value through the API:

curl -X PATCH "https://api.vapi.ai/eval/simulation/scenario/<scenario-id>" \
  -H "Authorization: Bearer $VAPI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "targetOverrides": {
      "variableValues": {
        "customer": { "number": "+14155550123" }
      }
    }
  }'

For a transient scenario, set the same targetOverrides on the inline scenario in POST /eval/simulation/run.

• The value replaces the whole customer variable, so also set any other customer fields the assistant references, such as customer.name.
• Only {{customer.number}} changes. The call.customer object in server messages does not, so a tool that reads the caller from the message payload won’t see the override.
• To test the conversation without a real lookup, mock the tool instead.

Troubleshooting

Next steps