Simulations quickstart

In about 10 minutes, you’ll run an AI tester through an appointment-booking conversation with Riley and get a pass or fail result. You’ll also have a reusable simulation suite that you can run again after any change.

What you’ll build

A reusable simulation suite that asks an AI tester to book an appointment with Riley, Vapi’s appointment-scheduler template assistant. The first criterion checks whether Riley reports the requested date and time with a confirmation number. This is a conversation-only demonstration, not proof that an appointment exists in a calendar.

Prerequisites

A Vapi account.

You create Riley and the structured output during this quickstart.

Simulations run unmocked tools for real. Before running, inspect Riley’s tools and use sandbox integrations or mock every tool that could create a booking, send a message, or affect a real customer.

Build and run a simulation

Dashboard

cURL

Create Riley from the appointment-scheduler template

In the Vapi Dashboard, go to Assistants. Open the arrow next to Create Assistant, then select Appointment Scheduler. Vapi creates Riley immediately.

Create a simulation suite

Open Simulations, then select Suites. Select Create suite, give the suite a name, add Riley under Assistants or squads, then select Next.

A suite opens in view mode. To change it, open the suite and select Edit in the top-right corner.

Create a new simulation suite dialog with a suite name and Riley selected under Assistants or squads

Configure the scenario

The suite opens with its first simulation. In the Scenario tab, enter Book an appointment as the Scenario name. Paste the following text into Intent:

You are Alex Taylor, calling to book an appointment for January 20, 2027
at 2pm in America/Los_Angeles. Your test email is alex@example.com.
Provide these details when asked. Don't invent other account information.
End the call once you receive a confirmation number.

A suite can contain multiple simulations, each with its own scenario and personality. Select Add simulation in the suite editor to add another simulation.

Scenario tab with the Book an appointment scenario name and appointment-booking intent

Choose a personality

Open the Personality tab and select a built-in personality marked (Default). Keep its behavior and advanced settings unchanged for this first simulation.

To customize what the AI tester does and how it behaves, see Configure an AI tester.

Personality tab with the Skeptical Sam default personality selected

Add success criteria

Select Next. On the Success criteria tab, select Create structured output and configure this structured output:

Setting	Value
Name	`appointment_details_reported`
Type	Boolean
Description	Return `true` only if Riley tells the caller the appointment is confirmed for January 20, 2027 at 2pm Pacific time and provides a confirmation number. Otherwise return `false`. This checks the reported details, not calendar state.
Comparator	`equals (=)`
Expected value	`true`

You must add at least one structured output before you can run a simulation.

Save and run

Select Save & run. In Run suite, confirm Riley, choose a mode, and set Iterations to 1:

Chat

Runs without speech generation or transcription, so it is faster and costs less.

Voice

Exercises speech, transcription, and turn-taking with a synthetic caller. Listen to the recording to assess the audio experience.

Select Confirm to start the run.

Run suite dialog with Riley selected, Chat mode selected, and Iterations set to 1

Review results

Wait for the run to move through queued, running, and ended, then select View results to see what passed or failed.

To find the run later, open Simulations, then select Runs. Your run appears with its Status and Simulation result, along with the suite name, iterations, Riley, and run date. Filter by time range, status, or assistant or squad, then select the suite name to open the results.

The header shows Riley, the mode, run date, and overall result (Passed or a count such as 1/1 failed). The list on the left shows one result for each simulation and iteration. Select a result to review it:

The Success criteria panel appears next to the Transcript. Review whether appointment_details_reported agrees with what Riley actually said. The transcript lists the conversation turn by turn between the AI tester and Riley. For voice runs, also listen to the recording when available; the criterion doesn’t grade pronunciation, pauses, or interruptions.

Passed simulation run with Success criteria beside the Transcript

Select Rerun to run the suite again.

Next steps

Use one iteration to check the setup, then repeat critical journeys before release. For a tool-backed booking test, add separate criteria for the requested tool arguments and for reporting its result accurately. Include a tool-error case where Riley must not claim success. A mocked success proves behavior under the supplied result, not an actual booking; verify calendar state separately with sandbox records. See writing observable success criteria.

Simulations advanced

Configure variables, tool mocks, webhooks, and structured outputs.

Configure an AI tester

Define the AI tester’s scenario, behavior, model, transcriber, and voice.

Simulations overview

What Simulations are and when to use them over Evals.

Structured outputs quickstart

Define the structured outputs your evaluations check against.

Manage simulations

Edit suites, review and rerun results, and maintain test coverage.