Simulations overview

Simulations are automated tests that run an AI tester through a real conversation with your assistant or squad and score the outcome. They matter because they catch broken flows, bad handoffs, and voice issues before the assistant or squad handles production traffic. This page explains what Simulations are and when to use them.

Key concepts

How it works

A simulation recreates a real conversation instead of replaying a fixed script. The AI tester follows the scenario’s intent, talks to your assistant or squad over voice or chat, and the success criteria score whether the assistant or squad behaved correctly.

You start from a simulation suite. A suite contains one or more simulations and can run against one or more assistants or squads.

Each simulation combines a scenario and a personality. The scenario sets what the AI tester is trying to accomplish and the criteria for success. The personality sets how the AI tester behaves and speaks. A suite can include multiple simulations, each with its own scenario and personality.

Voice and chat modes

Simulations support two conversation modes:

Both modes run the same AI tester conversation and score it the same way. Chat mode simply removes the audio layer: the AI tester and the assistant or squad exchange text directly instead of synthesized speech and transcription.

When you run the suite, Vapi runs each simulation for the number of iterations you select. During each iteration, the AI tester holds a live conversation with the assistant or squad. Vapi then evaluates each conversation with structured outputs and reports which criteria passed or failed.

When to use Simulations

Use Simulations to confirm that an assistant or squad behaves correctly before or after a change reaches production:

• Pre-launch validation: Confirm a new or updated assistant or squad handles a scenario correctly before it takes real traffic.
• Regression testing: After changing a prompt, tool, or configuration, rerun a suite to catch behavior that broke.
• Edge cases and difficult behavior: Give the AI tester different personalities to see how the assistant or squad handles impatient, confused, or interrupting behavior.
• Guardrail validation: Confirm the assistant or squad stays within its boundaries, such as never promising a refund it can’t authorize, and keep the results as evidence.
• Tool-call validation: Confirm the assistant or squad calls tools with the right arguments and handles failures gracefully, such as surfacing a booking error instead of falsely confirming it.

Simulations compared with Evals

Use Evals to test the logic of an assistant or squad. Use Simulations to test how that assistant or squad handles a real conversation. Both are testing tools in Vapi, and they complement each other.

Evals verify specific, expected behavior. You script a mock conversation with user turns and expected responses, then check the result with exact match, regex, an AI judge, or tool-call checks. Evals are precise and repeatable, which makes them a good fit for unit-style checks.

Simulations test live interactions. An AI tester with a defined personality and intent improvises a conversation with your assistant or squad over voice or chat. Evaluations score the outcome with structured outputs, showing how the assistant or squad handles a realistic AI tester rather than a fixed script.

Choose Evals when you want to lock down a specific response or verify a tool call’s arguments with fast, repeatable checks.

Choose Simulations when you want to see how your assistant or squad performs against an AI tester with different personalities, multi-turn flows, squad handoffs, or voice.

Next steps