Simulations advanced

Overview

This guide covers advanced simulation strategies, testing patterns, and best practices for building robust test suites that ensure your AI voice agents work reliably in production.

You’ll learn:

• Advanced scenario configuration (tool mocks, hooks)
• Strategic testing approaches (smoke, regression, edge cases)
• Performance optimization techniques
• CI/CD integration strategies
• Maintenance and troubleshooting methods

Advanced scenario configuration

Tool mocks

Mock tool call responses at the scenario level to test specific paths without calling real APIs. This is useful for:

• Testing error handling paths
• Simulating unavailable services
• Deterministic test results
• Faster test execution (no real API calls)

Common tool mock patterns:

1
{
2
  "toolName": "bookAppointment",
3
  "result": "{\"status\": \"success\", \"confirmationId\": \"APT-12345\", \"datetime\": \"2024-01-20T14:00:00Z\"}",
4
  "enabled": true
5
}

1
{
2
  "toolName": "bookAppointment",
3
  "result": "{\"error\": \"Time slot no longer available\", \"availableSlots\": [\"14:30\", \"15:00\", \"15:30\"]}",
4
  "enabled": true
5
}

1
{
2
  "toolName": "checkInventory",
3
  "result": "{\"error\": \"Request timeout\", \"code\": \"ETIMEDOUT\"}",
4
  "enabled": true
5
}

1
{
2
  "toolName": "processOrder",
3
  "result": "{\"status\": \"partial\", \"itemsProcessed\": 2, \"itemsFailed\": 1, \"failedReason\": \"Item out of stock\"}",
4
  "enabled": true
5
}

Simulation hooks

Trigger actions on simulation lifecycle events. Hooks are useful for:

• Notifying external systems when tests start/end
• Logging test execution to your own systems
• Triggering follow-up workflows
• Custom analytics and reporting

Webhook payload examples:

1
// simulation.run.started webhook payload
2
{
3
  "event": "simulation.run.started",
4
  "simulationId": "550e8400-e29b-41d4-a716-446655440003",
5
  "runId": "550e8400-e29b-41d4-a716-446655440007",
6
  "timestamp": "2024-01-15T09:50:05Z"
7
}
8
9
// simulation.run.ended webhook payload
10
{
11
  "event": "simulation.run.ended",
12
  "simulationId": "550e8400-e29b-41d4-a716-446655440003",
13
  "runId": "550e8400-e29b-41d4-a716-446655440007",
14
  "timestamp": "2024-01-15T09:52:30Z",
15
  "duration": 145,
16
  "status": "passed",
17
  "transcript": "...",          // if include.transcript = true
18
  "messages": [...],            // if include.messages = true
19
  "recordingUrl": "https://..." // if include.recordingUrl = true
20
}

Using existing structured outputs

Instead of defining inline structured outputs in each scenario, you can reference structured outputs you’ve already created. This provides:

• Reusability across multiple scenarios
• Centralized management of evaluation criteria
• Consistency in how data is extracted

Testing strategies

Smoke tests

Quick validation that core functionality works. Run these first to catch obvious issues.

Purpose: Verify your assistant responds and basic conversation flow works before running comprehensive tests.

1
{
2
  "name": "Smoke Test - Basic Response",
3
  "instructions": "Say hello and ask if the assistant can hear you.",
4
  "evaluations": [
5
    {
6
      "structuredOutput": {
7
        "name": "assistant_responded",
8
        "schema": {
9
          "type": "boolean",
10
          "description": "Whether the assistant provided any response"
11
        }
12
      },
13
      "comparator": "=",
14
      "value": true
15
    }
16
  ]
17
}

Characteristics:

• Minimal evaluation criteria (just check for any response)
• Fast execution (simple instructions)
• Run before detailed tests
• Use chat mode for speed

When to use:

• Before running expensive voice test suites
• After deploying configuration changes
• As health checks in monitoring
• Quick validation during development

Regression tests

Ensure fixes and updates don’t break existing functionality.

Purpose: Validate that known issues stay fixed and features keep working.

Best practices:

• Name tests after bugs they prevent
• Include ticket/issue numbers
• Add regression tests when fixing bugs
• Run full regression suite before major releases

Edge case testing

Test boundary conditions and unusual inputs your assistant might encounter.

Common edge cases to test:

1
{
2
  "name": "Edge Case - Ambiguous Request",
3
  "instructions": "Make a vague, unclear request like 'I need something done' without specifying what you want.",
4
  "evaluations": [
5
    {
6
      "structuredOutput": {
7
        "name": "asked_for_clarification",
8
        "schema": {
9
          "type": "boolean",
10
          "description": "Whether the assistant asked for more details"
11
        }
12
      },
13
      "comparator": "=",
14
      "value": true
15
    }
16
  ]
17
}

1
{
2
  "name": "Edge Case - Topic Switch",
3
  "instructions": "Start asking about booking an appointment, then suddenly switch to asking about cancellation policies mid-conversation.",
4
  "evaluations": [
5
    {
6
      "structuredOutput": {
7
        "name": "handled_topic_switch",
8
        "schema": {
9
          "type": "boolean",
10
          "description": "Whether the assistant smoothly transitioned to the new topic"
11
        }
12
      },
13
      "comparator": "=",
14
      "value": true
15
    }
16
  ]
17
}

1
{
2
  "name": "Edge Case - Interruption Handling",
3
  "instructions": "Interrupt the assistant mid-sentence with a new question. See if it handles the interruption gracefully.",
4
  "evaluations": [
5
    {
6
      "structuredOutput": {
7
        "name": "handled_interruption",
8
        "schema": {
9
          "type": "boolean",
10
          "description": "Whether the assistant stopped and addressed the interruption"
11
        }
12
      },
13
      "comparator": "=",
14
      "value": true
15
    }
16
  ]
17
}

1
{
2
  "name": "Edge Case - Invalid Date",
3
  "instructions": "Try to book an appointment for 'the 45th of Octember' - an obviously invalid date.",
4
  "evaluations": [
5
    {
6
      "structuredOutput": {
7
        "name": "handled_invalid_date",
8
        "schema": {
9
          "type": "boolean",
10
          "description": "Whether the assistant politely asked for a valid date"
11
        }
12
      },
13
      "comparator": "=",
14
      "value": true
15
    }
16
  ]
17
}

Edge case categories to cover:

• Input boundaries: Empty, maximum length, special characters
• Data formats: Invalid dates, malformed phone numbers, unusual names
• Conversation patterns: Interruptions, topic changes, contradictions
• Emotional scenarios: Frustrated caller, confused caller, impatient caller

Best practices

Evaluation design principles

Choosing voice vs chat mode

CI/CD integration

Automate simulation runs in your deployment pipeline.

Basic workflow

1
# .github/workflows/test-assistant.yml
2
name: Test Assistant Changes
3
4
on:
5
  pull_request:
6
    paths:
7
      - 'assistants/**'
8
      - 'prompts/**'
9
10
jobs:
11
  run-simulations:
12
    runs-on: ubuntu-latest
13
    steps:
14
      - name: Run smoke tests (chat mode)
15
        run: |
16
          # Create a simulation run
17
          RUN_ID=$(curl -s -X POST "https://api.vapi.ai/eval/simulation/run" \
18
            -H "Authorization: Bearer ${{ secrets.VAPI_API_KEY }}" \
19
            -H "Content-Type: application/json" \
20
            -d '{
21
              "simulations": [{"type": "simulationSuite", "simulationSuiteId": "${{ vars.SMOKE_TEST_SUITE_ID }}"}],
22
              "target": {"type": "assistant", "assistantId": "${{ vars.STAGING_ASSISTANT_ID }}"},
23
              "transport": {"provider": "vapi.webchat"}
24
            }' | jq -r '.id')
25
26
          echo "Run ID: $RUN_ID"
27
28
          # Poll for completion
29
          while true; do
30
            STATUS=$(curl -s "https://api.vapi.ai/eval/simulation/run/$RUN_ID" \
31
              -H "Authorization: Bearer ${{ secrets.VAPI_API_KEY }}" | jq -r '.status')
32
33
            if [ "$STATUS" = "ended" ]; then
34
              break
35
            fi
36
37
            sleep 10
38
          done
39
40
          # Check results
41
          RESULT=$(curl -s "https://api.vapi.ai/eval/simulation/run/$RUN_ID" \
42
            -H "Authorization: Bearer ${{ secrets.VAPI_API_KEY }}")
43
44
          PASSED=$(echo $RESULT | jq '.itemCounts.passed')
45
          FAILED=$(echo $RESULT | jq '.itemCounts.failed')
46
47
          if [ "$FAILED" -gt 0 ]; then
48
            echo "Simulations failed: $FAILED"
49
            exit 1
50
          fi
51
52
          echo "All simulations passed: $PASSED"

Advanced patterns

$
# Run comprehensive tests against staging
$
./scripts/run-simulation-suite.sh \
>
  --suite-id "$REGRESSION_SUITE_ID" \
>
  --target-assistant "$STAGING_ASSISTANT_ID" \
>
  --transport "vapi.websocket" \
>
  --iterations 3
$
$
# Only deploy to production if all pass
$
if [ $? -eq 0 ]; then
$
  ./scripts/deploy-to-production.sh
$
fi

1
# .github/workflows/nightly-regression.yml
2
on:
3
  schedule:
4
    - cron: '0 2 * * *'  # 2 AM daily
5
6
jobs:
7
  regression-suite:
8
    runs-on: ubuntu-latest
9
    steps:
10
      - name: Run full regression (voice mode)
11
        run: ./scripts/run-simulation-suite.sh --full-regression
12
13
      - name: Notify on failures
14
        if: failure()
15
        run: |
16
          # Send Slack notification
17
          curl -X POST $SLACK_WEBHOOK_URL \
18
            -d '{"text": "Nightly simulation regression failed!"}'

$
RESULT=$(curl -s "https://api.vapi.ai/eval/simulation/run/$RUN_ID" \
>
  -H "Authorization: Bearer $VAPI_API_KEY")
$
$
TOTAL=$(echo $RESULT | jq '.itemCounts.total')
$
PASSED=$(echo $RESULT | jq '.itemCounts.passed')
$
$
PASS_RATE=$((PASSED * 100 / TOTAL))
$
$
if [ $PASS_RATE -lt 95 ]; then
$
  echo "Pass rate $PASS_RATE% below threshold 95%"
$
  exit 1
$
fi

Maintenance strategies

Regular review cycle

When to update simulations

Troubleshooting

Common issues

Debugging tips

$
curl "https://api.vapi.ai/eval/simulation/run/$RUN_ID" \
>
  -H "Authorization: Bearer $VAPI_API_KEY" | jq '.status, .endedReason'

$
curl "https://api.vapi.ai/eval/simulation/run/$RUN_ID/item" \
>
  -H "Authorization: Bearer $VAPI_API_KEY" | jq '.[].status'

Getting help

Include these details when reporting issues:

• Simulation run ID
• Scenario and personality IDs
• Transport mode used (voice/chat)
• Expected vs actual behavior
• Assistant configuration

Resources:

• Simulations Quickstart
• Discord Community
• Support

Next steps

Summary