> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.vapi.ai/llms.txt. > For full documentation content, see https://docs.vapi.ai/llms-full.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.vapi.ai/_mcp/server. # Simulations quickstart ## Overview This quickstart guide will help you test your AI assistants and squads using realistic, AI-powered callers. In just a few minutes, you'll create test scenarios, define evaluation criteria, and validate your agents work correctly under different conditions. ### What are Simulations? Simulations is Vapi's voice agent testing framework that enables you to systematically test assistants and squads using AI-powered callers that follow defined instructions and evaluate outcomes using structured outputs. Instead of relying on manual testing or rigid scripts, Simulations recreate real conversations and measure whether your assistant behaves correctly. Test your agents by: 1. **Creating personalities** - Define a full assistant configuration for the AI tester (voice, model, system prompt) 2. **Defining scenarios** - Specify instructions for the tester and evaluations using structured outputs 3. **Creating simulations** - Pair scenarios with personalities 4. **Running simulations** - Execute tests against your assistant or squad in voice or chat mode 5. **Reviewing results** - Analyze pass/fail outcomes based on structured output evaluations ### When are Simulations useful? Simulations help you maintain quality and catch issues early: * **Pre-deployment testing** - Validate new assistant configurations before going live * **Regression testing** - Ensure prompt or tool changes don't break existing behaviors * **Conversation flow validation** - Test multi-turn interactions and complex scenarios * **Personality-based testing** - Verify your agent handles different caller types appropriately * **Squad handoff testing** - Ensure smooth transitions between squad members * **Performance monitoring** - Track success rates over time and identify regressions ### Voice vs Chat mode Simulations support two transport modes: * Full voice simulation with audio * Realistic end-to-end testing * Tests speech recognition and synthesis * Produces call recordings - Text-based chat simulation - Faster execution - Lower cost (no audio processing) - Ideal for rapid iteration Use **chat mode** during development for quick iteration, then switch to **voice mode** for final validation before deployment. ### What you'll build A simulation suite for an appointment booking assistant that tests: * Different caller personalities (confused user, impatient customer) * Evaluation criteria using structured outputs with comparators * Real-time monitoring of test runs * Both voice and chat mode execution ## Prerequisites Sign up at [dashboard.vapi.ai](https://dashboard.vapi.ai) Get your API key from **API Keys** in sidebar You'll also need an existing assistant or squad to test. You can create one in the Dashboard or use the API. ## Step 1: Create a personality Personalities define how the AI tester behaves during a simulation. A personality is a full assistant configuration that controls the tester's voice, model, and behavior via system prompt. 1. Log in to [dashboard.vapi.ai](https://dashboard.vapi.ai) 2. Click on **Simulations** in the left sidebar 3. Click the **Personalities** tab 1) Click **Create Personality** 2) **Name**: Enter "Impatient Customer" 3) **Assistant Configuration**: Configure the tester assistant: * **Model**: Select your preferred LLM (e.g., GPT-4o) * **System Prompt**: Define the personality behavior: ``` You are an impatient customer who wants quick answers. Speak directly and may interrupt if responses are too long. You expect immediate solutions to your problems. ``` * **Voice**: Select a voice for the tester (optional for chat mode) 4) Click **Save** Start with the built-in default personalities to get familiar with the system before creating custom ones. ```bash curl -X POST "https://api.vapi.ai/eval/simulation/personality" \ -H "Authorization: Bearer $VAPI_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Impatient Customer", "assistant": { "model": { "provider": "openai", "model": "gpt-4o", "messages": [ { "role": "system", "content": "You are an impatient customer who wants quick answers. Speak directly and may interrupt if responses are too long. You expect immediate solutions to your problems." } ] }, "voice": { "provider": "cartesia", "voiceId": "sonic-english" } } }' ``` **Response:** ```json { "id": "550e8400-e29b-41d4-a716-446655440001", "orgId": "org-456", "name": "Impatient Customer", "assistant": { "model": { "provider": "openai", "model": "gpt-4o", "messages": [...] }, "voice": { "provider": "cartesia", "voiceId": "sonic-english" } }, "createdAt": "2024-01-15T09:30:00Z", "updatedAt": "2024-01-15T09:30:00Z" } ``` Save the returned `id` - you'll need it when creating simulations. **Personality types:** Consider creating personalities for different customer types you encounter: decisive buyers, confused users, detail-oriented customers, or frustrated callers. ## Step 2: Create a scenario Scenarios define what the test is evaluating. A scenario contains: * **Instructions**: What the tester should do during the call * **Evaluations**: Structured outputs with expected values to validate outcomes 1. In **Simulations**, click the **Scenarios** tab 2. Click **Create Scenario** 1) **Name**: Enter "Book Appointment" 2) **Instructions**: Define what the tester should do: ``` You are calling to book an appointment for next Monday at 2pm. Confirm your identity when asked and provide any required information. End the call once you receive a confirmation number. ``` Evaluations use structured outputs to extract data from the conversation and compare against expected values. 1. Click **Add Evaluation** 2. Create or select a structured output: * **Name**: "appointment\_booked" * **Schema Type**: boolean 3. Set the **Comparator**: `=` 4. Set the **Expected Value**: `true` 5. Mark as **Required**: Yes 6. Add another evaluation for confirmation number: * **Name**: "confirmation\_provided" * **Schema Type**: boolean * **Comparator**: `=` * **Expected Value**: `true` 7. Click **Save Scenario** ```bash curl -X POST "https://api.vapi.ai/eval/simulation/scenario" \ -H "Authorization: Bearer $VAPI_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Book Appointment", "instructions": "You are calling to book an appointment for next Monday at 2pm. Confirm your identity when asked and provide any required information. End the call once you receive a confirmation number.", "evaluations": [ { "structuredOutput": { "name": "appointment_booked", "schema": { "type": "boolean", "description": "Whether an appointment was successfully booked" } }, "comparator": "=", "value": true, "required": true }, { "structuredOutput": { "name": "confirmation_provided", "schema": { "type": "boolean", "description": "Whether a confirmation number was provided" } }, "comparator": "=", "value": true, "required": true } ] }' ``` **Response:** ```json { "id": "550e8400-e29b-41d4-a716-446655440002", "orgId": "org-456", "name": "Book Appointment", "instructions": "You are calling to book an appointment for next Monday at 2pm...", "evaluations": [ { "structuredOutput": { "name": "appointment_booked", "schema": { "type": "boolean", "description": "..." } }, "comparator": "=", "value": true, "required": true }, { "structuredOutput": { "name": "confirmation_provided", "schema": { "type": "boolean", "description": "..." } }, "comparator": "=", "value": true, "required": true } ], "createdAt": "2024-01-15T09:35:00Z", "updatedAt": "2024-01-15T09:35:00Z" } ``` Save the returned `id` - you'll need it when creating simulations. ### Evaluation structure Each evaluation consists of: | Field | Description | | -------------------- | --------------------------------------------------------------------------------------- | | `structuredOutputId` | Reference to an existing structured output (mutually exclusive with `structuredOutput`) | | `structuredOutput` | Inline structured output definition (mutually exclusive with `structuredOutputId`) | | `comparator` | Comparison operator: `=`, `!=`, `>`, `<`, `>=`, `<=` | | `value` | Expected value (string, number, or boolean) | | `required` | Whether this evaluation must pass for the simulation to pass (default: `true`) | **Schema type restrictions:** Evaluations only support primitive schema types: `string`, `number`, `integer`, `boolean`. Objects and arrays are not supported. ### Comparator options | Comparator | Description | Supported Types | | ---------- | --------------------- | -------------------------------- | | `=` | Equals | string, number, integer, boolean | | `!=` | Not equals | string, number, integer, boolean | | `>` | Greater than | number, integer | | `<` | Less than | number, integer | | `>=` | Greater than or equal | number, integer | | `<=` | Less than or equal | number, integer | **Evaluation tips:** Use boolean structured outputs for pass/fail checks like "appointment\_booked" or "issue\_resolved". Use numeric outputs with comparators for metrics like "satisfaction\_score >= 4". ## Step 3: Create a simulation Simulations pair a scenario with a personality. The target assistant or squad is specified when you run the simulation. 1. In **Simulations**, click the **Simulations** tab 2. Click **Create Simulation** 1) **Name**: Enter "Appointment Booking - Impatient Customer" (optional) 2) **Scenario**: Select "Book Appointment" from the dropdown 3) **Personality**: Select "Impatient Customer" from the dropdown 4) Click **Save Simulation** ```bash curl -X POST "https://api.vapi.ai/eval/simulation" \ -H "Authorization: Bearer $VAPI_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Appointment Booking - Impatient Customer", "scenarioId": "550e8400-e29b-41d4-a716-446655440002", "personalityId": "550e8400-e29b-41d4-a716-446655440001" }' ``` **Response:** ```json { "id": "550e8400-e29b-41d4-a716-446655440003", "orgId": "org-456", "name": "Appointment Booking - Impatient Customer", "scenarioId": "550e8400-e29b-41d4-a716-446655440002", "personalityId": "550e8400-e29b-41d4-a716-446655440001", "createdAt": "2024-01-15T09:40:00Z", "updatedAt": "2024-01-15T09:40:00Z" } ``` Save the returned `id` - you'll need it when running simulations. **Multiple simulations:** Create several simulations with different personality and scenario combinations to thoroughly test your assistant across various conditions. ## Step 4: Create a simulation suite (optional) Simulation suites group multiple simulations into a single batch that runs together. 1. In **Simulations**, click the **Suites** tab 2. Click **Create Suite** 1) **Name**: Enter "Appointment Booking Regression Suite" 2) Click **Add Simulations** 3) Select the simulations you want to include: * "Appointment Booking - Impatient Customer" * "Appointment Booking - Confused User" * "Appointment Booking - Decisive Customer" 4) Click **Save Suite** ```bash curl -X POST "https://api.vapi.ai/eval/simulation/suite" \ -H "Authorization: Bearer $VAPI_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Appointment Booking Regression Suite", "simulationIds": [ "550e8400-e29b-41d4-a716-446655440003", "550e8400-e29b-41d4-a716-446655440004", "550e8400-e29b-41d4-a716-446655440005" ] }' ``` **Response:** ```json { "id": "550e8400-e29b-41d4-a716-446655440006", "orgId": "org-456", "name": "Appointment Booking Regression Suite", "simulationIds": [ "550e8400-e29b-41d4-a716-446655440003", "550e8400-e29b-41d4-a716-446655440004", "550e8400-e29b-41d4-a716-446655440005" ], "createdAt": "2024-01-15T09:45:00Z", "updatedAt": "2024-01-15T09:45:00Z" } ``` Save the returned `id` - you'll need it to run the suite. **Suite organization:** Group related simulations together. For example, create separate suites for "Booking Tests", "Cancellation Tests", and "Rescheduling Tests". ## Step 5: Run a simulation Execute simulations against your assistant or squad. You can run individual simulations or entire suites. 1. Navigate to your simulation or suite 2. Click **Run** 3. Select the **Target**: * Choose **Assistant** or **Squad** * Select from the dropdown 4. Configure **Transport** (optional): * **Voice**: `vapi.websocket` (default) * **Chat**: `vapi.webchat` (faster, no audio) 5. Set **Iterations** (optional): Number of times to run each simulation 6. Click **Start Run** 1) Click the **Runs** tab to see live status updates 2) Watch as each simulation progresses: * **Queued** - Waiting to start * **Running** - Test in progress * **Ended** - Test finished 3) For voice mode, click **Listen** on any running test to hear the call live **Run a single simulation in voice mode:** ```bash curl -X POST "https://api.vapi.ai/eval/simulation/run" \ -H "Authorization: Bearer $VAPI_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "simulations": [ { "type": "simulation", "simulationId": "550e8400-e29b-41d4-a716-446655440003" } ], "target": { "type": "assistant", "assistantId": "your-assistant-id" }, "transport": { "provider": "vapi.websocket" } }' ``` **Run a simulation in chat mode (faster, no audio):** ```bash curl -X POST "https://api.vapi.ai/eval/simulation/run" \ -H "Authorization: Bearer $VAPI_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "simulations": [ { "type": "simulation", "simulationId": "550e8400-e29b-41d4-a716-446655440003" } ], "target": { "type": "assistant", "assistantId": "your-assistant-id" }, "transport": { "provider": "vapi.webchat" } }' ``` **Run a suite with multiple iterations:** ```bash curl -X POST "https://api.vapi.ai/eval/simulation/run" \ -H "Authorization: Bearer $VAPI_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "simulations": [ { "type": "simulationSuite", "simulationSuiteId": "550e8400-e29b-41d4-a716-446655440006" } ], "target": { "type": "assistant", "assistantId": "your-assistant-id" }, "iterations": 3 }' ``` **Response:** ```json { "id": "550e8400-e29b-41d4-a716-446655440007", "orgId": "org-456", "status": "queued", "simulations": [ { "type": "simulation", "simulationId": "550e8400-e29b-41d4-a716-446655440003" } ], "target": { "type": "assistant", "assistantId": "your-assistant-id" }, "transport": { "provider": "vapi.websocket" }, "queuedAt": "2024-01-15T09:50:00Z", "createdAt": "2024-01-15T09:50:00Z", "updatedAt": "2024-01-15T09:50:00Z" } ``` **Check run status:** ```bash curl -X GET "https://api.vapi.ai/eval/simulation/run/550e8400-e29b-41d4-a716-446655440007" \ -H "Authorization: Bearer $VAPI_API_KEY" ``` ## Step 6: Review results Analyze the results of your simulation runs to understand how your assistant performed. ### Successful run When all evaluations pass, you'll see: ```json { "id": "550e8400-e29b-41d4-a716-446655440007", "status": "ended", "itemCounts": { "total": 3, "passed": 3, "failed": 0, "running": 0, "queued": 0, "canceled": 0 }, "startedAt": "2024-01-15T09:50:05Z", "endedAt": "2024-01-15T09:52:30Z" } ``` **Pass criteria:** * `status` is "ended" * `itemCounts.passed` equals `itemCounts.total` * All required evaluations show `passed: true` ### Failed run When evaluation fails, you'll see details about what went wrong: ```json { "id": "550e8400-e29b-41d4-a716-446655440008", "status": "ended", "itemCounts": { "total": 3, "passed": 2, "failed": 1, "running": 0, "queued": 0, "canceled": 0 } } ``` **Failure indicators:** * `itemCounts.failed` > 0 * Individual run items show which evaluations failed and why 1. Navigate to the **Runs** tab 2. Click on a completed run to see details 3. View the summary showing pass/fail counts 1) Click on any failed simulation 2) Review the **Conversation** to see the full transcript 3) Check which evaluations failed and their actual vs expected values 4) For voice mode, click **Listen to Recording** to hear the full call 1. Go to the main **Simulations** page 2. View historical runs and their pass rates 3. Monitor trends to identify regressions **List all runs:** ```bash curl -X GET "https://api.vapi.ai/eval/simulation/run" \ -H "Authorization: Bearer $VAPI_API_KEY" ``` **Get detailed results for a specific run:** ```bash curl -X GET "https://api.vapi.ai/eval/simulation/run/550e8400-e29b-41d4-a716-446655440007" \ -H "Authorization: Bearer $VAPI_API_KEY" ``` **Filter runs by status:** ```bash curl -X GET "https://api.vapi.ai/eval/simulation/run?status=ended" \ -H "Authorization: Bearer $VAPI_API_KEY" ``` Full conversation transcripts are available for all simulation runs, making it easy to understand exactly what happened during each test. ## Next steps Learn about tool mocks, hooks, CI/CD integration, and testing strategies Create and configure assistants to test Learn about chat-based testing with mock conversations Learn how to define structured outputs for evaluations ## Tips for success **Best practices for effective simulation testing:** * **Start with chat mode** - Use `vapi.webchat` for rapid iteration, then validate with voice * **Use realistic personalities** - Model your test callers after actual customer types * **Define clear evaluations** - Use specific, measurable structured outputs * **Group related tests** - Organize suites by feature or user flow * **Monitor trends** - Track pass rates over time to catch regressions early * **Test after changes** - Run your simulation suites after updating prompts or tools * **Listen to recordings** - Audio recordings reveal issues that metrics alone miss * **Iterate on failures** - Use failed tests to improve both your assistant and test design ## Frequently asked questions Simulation concurrency follows your organization's call concurrency limits. Each voice simulation uses 2 concurrent call slots (one for the AI tester, one for your assistant being tested). Chat mode simulations are more efficient since they don't require audio processing. If you need higher concurrency limits, contact support. Simulations use AI-powered testers that have actual conversations with your assistant, producing real call recordings and transcripts. Evals use mock conversations with predefined messages and judge the responses. Use Simulations for realistic end-to-end testing; use Evals for faster, more controlled validation. Yes! You can either define inline structured outputs in your scenario evaluations, or reference existing structured outputs by ID using the `structuredOutputId` field. Create a simulation that targets a squad instead of an assistant. Use the `target.type: "squad"` and `target.squadId` fields when creating a run. ## Get help Need assistance? We're here to help: * [Discord Community](https://discord.gg/pUFNcf2WmH) * [Support](mailto:support@vapi.ai)