Simulations overview

Simulations are automated tests that run an AI tester through a real conversation with your assistant or squad and score the outcome. They matter because they catch broken flows, bad handoffs, and voice issues before the assistant or squad handles production traffic. This page explains what Simulations are and when to use them.

Key concepts

Term	Definition
Simulation suite	A simulation suite groups one or more simulations that you can run against one or more assistants or squads.
Simulation	A simulation pairs a scenario with a personality to test one situation.
Scenario	A scenario defines the AI tester’s intent and the success criteria that determine the result.
Personality	A personality defines how the AI tester behaves, including its model, transcriber, and voice.
AI tester	An AI tester drives the simulated conversation according to a scenario and personality.

How it works

A simulation recreates a real conversation instead of replaying a fixed script. The AI tester follows the scenario’s intent, talks to your assistant or squad over voice or chat, and the success criteria score whether the assistant or squad behaved correctly.

You start from a simulation suite. A suite contains one or more simulations and can run against one or more assistants or squads.

Each simulation combines a scenario and a personality. The scenario sets what the AI tester is trying to accomplish and the criteria for success. The personality sets how the AI tester behaves and speaks. A suite can include multiple simulations, each with its own scenario and personality.

Voice and chat modes

Simulations support two conversation modes:

Voice mode

Runs a full voice conversation with audio.
Tests the model, transcriber, and voice together.
Supports end-to-end voice validation.

Chat mode

Runs the same conversation as voice, exchanging text instead of audio.
Finishes faster and costs less because it skips speech synthesis and transcription.
Supports rapid iteration during development.

Both modes run the same scenario. Chat mode removes the audio layer: the AI tester and the assistant or squad exchange text directly instead of synthesized speech and transcription.

Use chat mode for rapid iteration during development, then switch to voice mode for final validation.

When you run the suite, Vapi runs each simulation for the number of iterations you select. During each iteration, the AI tester holds a live conversation with the assistant or squad. Vapi then evaluates each conversation with structured outputs and reports which criteria passed or failed.

When to use Simulations

Use Simulations to confirm that an assistant or squad behaves correctly before or after a change reaches production:

Pre-launch validation: Confirm a new or updated assistant or squad handles a scenario correctly before it takes real traffic.
Regression testing: After changing a prompt, tool, or configuration, rerun a suite to catch behavior that broke.
Edge cases and difficult behavior: Give the AI tester different personalities to see how the assistant or squad handles impatient, confused, or interrupting behavior.
Guardrail validation: Confirm the assistant or squad stays within its boundaries, such as never promising a refund it can’t authorize, and keep the results as evidence.
Tool-backed outcomes: Confirm the assistant or squad completes a workflow and handles tool failures gracefully, such as surfacing a booking error instead of falsely confirming it. Use Evals for exact tool and argument checks.

Simulations compared with Evals

Use Evals to test decisions at specific points in a conversation. Use Simulations to test outcomes across a complete conversation. Both are testing tools in Vapi, and they complement each other.

Evals verify specific, expected behavior. You script a mock conversation with user turns and expected responses, then check the result with exact match, regex, an AI judge, or tool-call checks. Their fixed context makes them a good fit for focused, unit-style checks.

Simulations test live interactions. An AI tester with a defined personality and intent improvises a conversation with your assistant or squad over voice or chat. Evaluations score the outcome with structured outputs, showing how the assistant or squad handles a realistic AI tester rather than a fixed script.

Comparison	Evals	Simulations
What you’re testing	A decision at a specific conversation point	The outcome of a complete conversation
What you provide	A scripted conversation with expected responses	An AI tester personality and an intent
How it runs	Fixed-context, turn-by-turn checks	Dynamic; the AI tester improvises the conversation
Transport	Chat (mock conversations)	Voice, or text-only
Evaluation	Exact match, regex, AI judge, tool-call checks	Structured outputs on the conversation outcome
Reach for it when	You need focused checks at a known point	You need realistic end-to-end or voice behavior

Choose Evals when you want to lock down a specific response or verify a tool call’s arguments with fast, rerunnable checks.

Choose Simulations when you want to see how your assistant or squad performs against an AI tester with different personalities, multi-turn flows, squad handoffs, or voice.

See testing voice agents for guidance on combining both tools into one coverage plan.

Next steps

Simulations quickstart

Run your first simulation against an assistant or squad.

Configure an AI tester

Define the AI tester’s scenario, behavior, model, transcriber, and voice.

Evals quickstart

Test assistant or squad logic with scripted mock conversations.

Manage simulations

Edit suites, review and rerun results, and maintain test coverage.