Simulations advanced
Advanced simulation options let you set variable values, mock tool responses, trigger lifecycle webhooks, set latency limits, and reuse structured outputs. Use them after you complete the Simulations quickstart and need more control over behavior or test conditions.
How it works
Advanced options belong to each simulation in a suite. They control which values the assistant or squad receives, what mocked tools return, and which lifecycle events send webhooks.
Open a suite and select Edit, then Next. The review step contains the Success criteria, Variables, and Tool mocks & webhooks tabs.
Mock tool responses
During a simulation, the assistant or squad under test runs its real tools. Mock a tool to return fixed text instead, so that tool never calls its live service or API. A mock intercepts only the tool it names; every other tool the assistant or squad calls still runs for real.
Dashboard
cURL
On the review step, open Tool mocks & webhooks. Under Tool mocks, select Add, then configure each mock with three fields:
- Tool: Select one of the tools configured on the assistant or squad. Only its configured tools, including Function tools, can be mocked.
- Mock result: Enter the string returned in place of the tool’s live output.
- Enabled: Toggle the mock on or off. A disabled mock is ignored, and the tool runs live.
The mock result is always passed to the assistant or squad as a string. In the Dashboard, enter plain text or JSON-formatted text directly. In a cURL JSON body, escape the quotation marks inside JSON-formatted text, as shown in the example. Use tool mocks to keep tool behavior deterministic, avoid side effects, and steer the conversation down a specific path without depending on a live service.
Common mock patterns
Choose a result that leads the assistant or squad through the behavior you want to test. These examples show how the result field appears in a cURL JSON body:
Success
Handled error
Unavailable service
Return the data that the tool normally provides after a successful action:
These strings replace the selected tool’s output. They do not produce an HTTP error, add network latency, or cause a network timeout.
Send webhooks on run start and end
Turn on webhooks to notify your server when a simulation iteration starts or ends. Each webhook sends a POST request to the URL you configure.
Dashboard
cURL
On the Tool mocks & webhooks tab, under Webhooks, toggle Simulation start and Simulation end.
Vapi sends simulation.run.started when a simulation iteration begins. The example below is a voice payload. A chat payload omits the calls object:
Vapi sends simulation.run.ended when the simulation iteration finishes. The example below is a voice payload. A chat payload omits calls, endedReason, and recordingUrl:
Both chat and voice payloads include simulationId, runId, simulationRunItemId, iterationNumber, and isSimulation. The ended event adds startedAt and endedAt, plus transcript and messages when you include them. The call-specific fields appear for voice simulations only: calls, endedReason, and recordingUrl. The calls object includes calls.listenUrl, a WebSocket URL for monitoring the run in real time. For the complete list of endedReason values, see Call ended reasons.
Reuse structured outputs
Each evaluation references a structured output. Reuse a structured output when several simulations should apply the same evaluation criterion. For more patterns, see structured output examples.
Dashboard
cURL
Reuse an existing structured output across simulations to keep the same criterion consistent, or create a new one inline from the Success criteria tab.
Set latency limits
A latency limit fails a voice simulation when the assistant or squad under test responds too slowly. Vapi measures latency on each turn of the call, combines the turns into one number, and passes the limit when that number is at or below the limit you set.
Latency limits work alongside structured-output evaluations. A scenario still needs at least one evaluation, and a required limit that fails marks the simulation as failed even when every evaluation passes.
Dashboard
cURL
On the Success criteria tab, under Latency expectations, select Add latency expectation. Configure each limit with four fields:
- Aggregation: How the call’s turns are combined: Median, Mean, P95, or Max.
- Metric: Turn latency, Model latency, or Voice latency.
- Limit: The maximum in milliseconds, as a whole number from 1 to 60,000.
- Required: Turn off to make the limit informational. An optional limit is reported but never fails the simulation.
A new limit starts as a required median turn latency of 1,200 ms. Each simulation can have up to 20 limits.
Metrics and aggregations
The aggregation reduces the call’s turns to one value: mean, median, p95, or max. p95 uses the nearest-rank method, so on a call with fewer than 20 turns it equals max. Measured values are rounded to the nearest millisecond before they are compared with the limit.
Gate on median turn latency. A short simulation has only a few turns, so p95 and max can fail on a single slow turn. To watch the slowest turns without failing runs, add an optional p95 or max limit next to a required median limit on the same metric.
When a limit is skipped
Skipped limits don’t affect the result.
Review latency results
Open a run and select a simulation. The Latency section shows average turn latency when at least one turn was measured. Each limit shows its threshold and result. When at least one turn measured that limit’s metric, the result also shows the measured value and number of turns. Optional and skipped limits are labeled. When a simulation fails, the count in the results list, such as 1 of 3 evaluations failed, includes its latency limits.
Through the API, each run item’s results.latencyEvaluations holds one entry per limit with actualMs, thresholdMs, sampleCount, passed, required, and, for skipped limits, isSkipped and skipReason. results.passed accounts for required latency limits.
Set variable values for the assistant or squad
Variables provide values for the dynamic variables used by the assistant or squad during the simulation. On the Variables tab, add Name and Value pairs. Each name must match a {{variable}} placeholder in the assistant’s prompt, and the value is substituted during the run.
Use variables to test specific inputs, such as a customer name or account tier, without editing the assistant or squad.
Simulate a call from a specific phone number
Assistants that identify callers by phone number, for example by passing {{customer.number}} to an account lookup tool, need a known caller number to test against. Set customer.number in the scenario’s targetOverrides.variableValues. The Variables tab only accepts flat names, so set this value through the API:
For a transient scenario, set the same targetOverrides on the inline scenario in POST /eval/simulation/run.
- The value replaces the whole
customervariable, so also set any othercustomerfields the assistant references, such ascustomer.name. - Only
{{customer.number}}changes. Thecall.customerobject in server messages does not, so a tool that reads the caller from the message payload won’t see the override. - To test the conversation without a real lookup, mock the tool instead.