> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.vapi.ai/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.vapi.ai/_mcp/server.

# Test outcomes with Simulations

> Write Simulation scenarios that give the AI tester a goal, set success criteria two reviewers would score alike, and cover recovery, guardrail, and voice paths.

## Overview

Simulations answer: **Can the agent guide the caller to the right result?**

An AI tester follows a goal and adapts during a complete conversation with your assistant or squad. Score the final result and important guardrails, not one exact route through the conversation. This keeps the test useful when the caller phrases something differently or the agent asks an unexpected but reasonable question.

Use [**Test decisions with Evals**](/test/evals-best-practices) when you need to test the agent's next decision at a specific point in a conversation.

## Start with the customer outcome

Write down what must be true when the conversation ends. Use evidence that distinguishes a completed task from a confident-sounding promise.

For an appointment change, avoid a test that requires a fixed sequence: greet the caller, ask for their name, offer Tuesday at 3:00 PM, call the scheduling tool, then read a confirmation.

Instead, test the durable requirements:

* The scheduling tool reports that the appointment changed to the time the caller accepts.
* The agent confirms the change only after the scheduling result reports success.
* The agent gives the caller the final date and time clearly.
* If the change fails, the agent explains the next safe option instead of claiming success.

The agent can then recover from clarifications or conversational detours without failing a valid test. Require a specific step only when the step is itself a business, safety, or compliance requirement.

## Give the AI tester a goal, not a script

The **scenario** should describe the caller's goal and relevant facts. The **personality** should describe how the caller behaves.

| Include in the scenario                 | Include in the personality                        |
| --------------------------------------- | ------------------------------------------------- |
| What the caller wants                   | Tone and patience                                 |
| Facts the caller knows                  | How concise or talkative they are                 |
| Information they may provide            | Whether they interrupt or ask follow-up questions |
| Constraints and acceptable alternatives | How they respond to unclear answers               |
| When to end the conversation            | Voice, transcriber, and turn-taking settings      |

For example:

**`Scenario`**

```text title="Scenario" wordWrap
You need to move your appointment because of a conflict. Ask for Tuesday at
3:00 PM. If that time is unavailable, accept Wednesday at 10:00 AM. Provide
your name and confirmation number when asked. End after the agent confirms the
new appointment or gives a clear next step because it cannot be changed.
```

Do not tell the tester every sentence to say or how the agent must respond. A realistic tester should be able to clarify, react, and continue toward the goal.

## Write focused success criteria

Each criterion should answer one question that two reviewers could score the same way. For example:

| Criterion               | Type             | Role     | Use it to check                                                               |
| ----------------------- | ---------------- | -------- | ----------------------------------------------------------------------------- |
| `appointment_changed`   | Yes/no (Boolean) | Required | The scheduling tool reported a successful change to the agreed date and time. |
| `no_false_confirmation` | Yes/no (Boolean) | Required | The agent confirmed success only after the scheduling tool reported success.  |
| `final_details_clear`   | Yes/no (Boolean) | Optional | The agent clearly summarized the final date and time.                         |

Success criteria can return a Boolean, string, or number. Choose the simplest type that represents the result. Use **required** criteria only for conditions that must pass. Use **optional** criteria to observe quality without failing the whole simulation. Separate independent requirements so a failure shows what needs attention.

The structured-output judge can assess tool results in the recorded messages.
It does not independently check your scheduling system or database. If you need
to verify that a change was saved, check that system separately. With
[mock tool responses](/observability/simulations-advanced#mock-tool-responses),
a passing test shows how the agent handled the supplied result; it does not
prove a real booking was made.

An AI judge can make mistakes. Check its scores against examples your team agrees
should pass or fail before relying on it for release decisions.

Good criteria:

* Describe an observable result.
* State exactly what counts as success.
* Allow different valid wording and conversational paths.
* Check both task completion and important guardrails.

Avoid criteria such as "The call went well" or "The agent was helpful." If interaction quality matters, define it more clearly, such as "The agent acknowledged the caller's concern without blaming them and explained the next step."

## Cover realistic paths

Start with each core reason a customer calls. Add depth according to the frequency and cost of failure.

| Path                               | Example                                           | What it proves                                            |
| ---------------------------------- | ------------------------------------------------- | --------------------------------------------------------- |
| **Core outcome**                   | The requested appointment is available.           | The main workflow succeeds.                               |
| **Natural variation**              | The caller gives the date before their name.      | The agent does not depend on one order of information.    |
| **Missing or unclear information** | The caller says "next Friday" without a timezone. | The agent asks before acting.                             |
| **Recovery**                       | The scheduling tool reports an error.             | The agent does not invent success and offers a next step. |
| **Guardrail**                      | The caller asks to bypass identity verification.  | The agent protects the account even under pressure.       |
| **Safe alternative**               | The requested time is unavailable.                | The agent offers valid choices without losing the goal.   |

Reuse a scenario across personalities when the outcome stays the same. Create
scenario variants with different variable values when the test data changes.
This creates useful variation without changing the business requirement.

## Choose chat or voice deliberately

| Mode      | Best for                                                                                                        |
| --------- | --------------------------------------------------------------------------------------------------------------- |
| **Chat**  | Fast iteration on prompts, tools, conversation logic, and success criteria.                                     |
| **Voice** | Final validation of the complete audio experience, including transcription, speaking, and turn-taking behavior. |

Build and debug a scenario in chat mode, then run important paths in voice mode before launch. Also use voice mode after changing the transcriber, voice, or turn-taking configuration.

> **Note**
>
> Synthetic callers do not fully represent real callers, devices, or environments.
> Supplement voice Simulations with manual calls and production-call review for
> background noise, degraded audio, varied accents, silences, and interruptions
> that your Simulations cannot reliably reproduce.

## Repeat critical tests and review the conversation

Agent behavior can vary between runs. One pass shows that the agent **can** succeed; repeated passes provide stronger evidence that it succeeds consistently. Use more iterations for outcomes where a failure would harm a customer or the business.

Read every failed transcript and sample passed transcripts. Ask:

* Did the agent make a real mistake?
* Did the AI tester follow the scenario?
* Did the criterion reject a valid outcome?
* Was the run affected by a tool or configuration error?

Keep the scenario and criteria stable when comparing agent changes. Update the test only when the expected customer outcome changes.

## Avoid common pitfalls

| Pitfall                                              | Better approach                                                                                              |
| ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| Prescribing every turn                               | Give the tester a goal, facts, constraints, and a stopping condition.                                        |
| Passing because the agent said the task was complete | Judge the recorded tool result and confirmed details. Verify external system changes separately when needed. |
| Combining many requirements into one criterion       | Use one criterion for each independent result or guardrail.                                                  |
| Testing only the happy path                          | Add ambiguity, failure, recovery, and refusal cases.                                                         |
| Treating one successful run as proof of reliability  | Repeat business-critical simulations.                                                                        |
| Editing a regression test until a change passes      | Keep the original expectation unless the business requirement changed.                                       |

## Next steps

#### [Simulations quickstart](/observability/simulations-quickstart)

Create and run a simulation suite.

#### [Configure an AI tester](/observability/simulations-configure-ai-tester)

Configure scenarios, personalities, and voice behavior.

#### [Simulations advanced](/observability/simulations-advanced)

Use variables, tool mocks, and reusable structured outputs.

#### [Run and maintain tests](/test/run-and-maintain-tests)

Review results, repeat critical tests, and keep coverage current.