Generate a simulation suite with the API

The Simulations onboarding API builds a simulation suite for an assistant from its prompt, tools, and handoffs. It runs the same flow as Composer. Call these three endpoints from a script, a CI job, or a coding agent:

You then run the suite with the existing POST /eval/simulation/run endpoint.

Prerequisites

• An assistant to test. Copy its ID from its page in the Dashboard.
• A private API key in VAPI_API_KEY. The key needs permission to create and update simulation suites.
• An HTTP client with a long timeout. Suite generation can take up to 3 minutes, so set a client timeout of at least 200 seconds.

Generate and run a suite

curl --fail-with-body -sS -X POST "https://api.vapi.ai/eval/simulation/onboarding/requirements" \
  --max-time 200 \
  -H "Authorization: Bearer $VAPI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "assistantId": "<assistant-id>" }' \
  > draft.json

{
  "version": 1,
  "sources": ["System prompt", "Tools: bookAppointment, checkAvailability"],
  "requirements": [
    { "id": "R1", "text": "Confirms the date, time, and time zone before booking." },
    { "id": "R2", "text": "Offers another slot when the requested time is unavailable." },
    {
      "id": "R3",
      "text": "Does not book outside office hours.",
      "tag": "assumption",
      "note": "Office hours are not in the prompt; assumed Monday to Friday, 9am to 5pm."
    }
  ],
  "openQuestions": [
    {
      "id": "Q1",
      "text": "What are the office hours?",
      "why": "R3 depends on them."
    }
  ]
}

jq -n --slurpfile draft draft.json '{
  assistantId: "<assistant-id>",
  previousDraft: $draft[0],
  feedback: "Office hours are Monday to Friday, 8am to 6pm. Drop R2: we never offer another slot."
}' > revise.json
curl --fail-with-body -sS -X POST "https://api.vapi.ai/eval/simulation/onboarding/requirements" \
  --max-time 200 \
  -H "Authorization: Bearer $VAPI_API_KEY" \
  -H "Content-Type: application/json" \
  -d @revise.json \
  > draft.json

jq -n --slurpfile draft draft.json '{
  assistantId: "<assistant-id>",
  idempotencyKey: "booking-agent-draft-2",
  requirements: $draft[0].requirements
}' > suite.json
curl --fail-with-body -sS -X POST "https://api.vapi.ai/eval/simulation/onboarding/suite" \
  --max-time 200 \
  -H "Authorization: Bearer $VAPI_API_KEY" \
  -H "Content-Type: application/json" \
  -d @suite.json

{
  "suite": { "id": "<suite-id>", "name": "Riley: core behaviors" },
  "assistantId": "<assistant-id>",
  "simulations": [
    {
      "id": "<simulation-id>",
      "scenarioId": "<scenario-id>",
      "name": "Books a slot inside office hours",
      "personalityName": "Skeptical Sam",
      "successCriteria": ["Did the assistant confirm the date, time, and time zone before booking?"],
      "requirementIds": ["R1", "R3"],
      "mockedTools": ["bookAppointment", "checkAvailability"]
    }
  ],
  "coverage": {
    "covered": 2,
    "total": 3,
    "gaps": [
      { "requirementId": "R2", "reason": "budget", "detail": "It did not fit in this generation." }
    ],
    "retryable": 1
  },
  "requirements": [{ "id": "R1", "text": "Confirms the date, time, and time zone before booking." }],
  "docs": { "bestPractices": "https://docs.vapi.ai/...", "toolMocks": "https://docs.vapi.ai/..." },
  "reused": false
}

curl --fail-with-body -sS -X PATCH "https://api.vapi.ai/eval/simulation/onboarding/suite/<suite-id>" \
  --max-time 200 \
  -H "Authorization: Bearer $VAPI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "patchKey": "booking-agent-edit-1",
    "edits": [
      {
        "simulationId": "<simulation-id>",
        "instruction": "Make the caller ask for 7pm first, then accept 5pm."
      }
    ],
    "retryFailed": true
  }'

curl --fail-with-body -sS -X POST "https://api.vapi.ai/eval/simulation/run" \
  -H "Authorization: Bearer $VAPI_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: booking-agent-run-1" \
  -d '{
    "simulations": [{ "type": "simulationSuite", "simulationSuiteId": "<suite-id>" }],
    "target": { "type": "assistant", "assistantId": "<assistant-id>" },
    "transport": { "provider": "vapi.webchat" },
    "iterations": 5
  }'

Request fields

Draft requirements

Generate a suite

Edit a suite

The suiteId in the path must be the ID of a suite generated by POST /eval/simulation/onboarding/suite (or by Composer) in your organization.

Response fields

Requirements draft

Suite

Generating and editing a suite both return this object.

Coverage gaps

A requirement with no simulation covering it appears in coverage.gaps with one of these reasons:

Tool gaps

The toolGaps array lists tool paths the simulations don’t exercise as configured. Most are mocked handoffs or transfers. Running them for real could reach your server, a phone number, or a destination that couldn’t be checked safely. A mocked handoff or transfer answers as if it succeeded, so the conversation continues, but the destination isn’t tested. New reasons may be added.

To test a mocked path, run a separate simulation against the destination assistant, or replace the mock with a sandbox integration. See Mock tool responses.

Idempotency and retries

Suite generation. Send one idempotencyKey per agreed draft, and retry with the same key:

• On a timeout or a 502, 504, or 524, retry with the same key. You get the saved suite, a 409 while it is still generating, or a fresh generation if the first one never saved.
• On a 409 with SIMULATION_ONBOARDING_GENERATION_IN_PROGRESS, the first request is still generating. Wait Retry-After seconds (15) and retry with the same key until you get the suite.
• On a 409 with SIMULATION_ONBOARDING_IDEMPOTENCY_KEY_REUSED, the key was already used for a different assistant or different requirements. Don’t retry with it; send a new key.

A retry that returns a saved suite sets reused to true and doesn’t count toward your limits. To generate a new suite from a revised draft, use a new key.

Edits. Send a new patchKey for each edit. To retry an edit after a timeout or a 5xx, send the same key again. If the suite already applied that key, nothing changes. You get the current suite, with failedEditSimulationIds listing what that edit couldn’t rewrite.

A suite remembers only the last 50 patch keys it applied. If you reuse one of those keys for a different edit, the new edit is ignored and you get the current suite. Once a key drops out of the last 50, a request with it can apply again. Never reuse a key for a different edit.

Drafts. Drafting has no key, so each retry requests a new draft. A request counts toward your daily limit once generation starts, even if it later fails. Requests refused before generation starts, such as a 404 or a 429, don’t count.

Limits

Each organization has a free daily allowance, shared by the API, the Dashboard, and Composer. Limits reset at 00:00 UTC.

An organization can also have at most 2 suite generations running at once.

These don’t count toward the daily limits:

• A retry that returns a saved suite, or a replayed patchKey.
• An edit that only removes simulations, or retryFailed when nothing is left to retry.
• A request refused before generation starts, such as a 404 or a 422.

A request counts once generation starts, even if it later fails with a 502 or times out.

The same limits apply to every plan, including Enterprise. No request field raises them.

When you reach a limit, the API returns 429:

Errors

The 409, 429, and quota-unavailable 503 responses carry a JSON body and a Retry-After header when waiting helps:

{
  "code": "SIMULATION_ONBOARDING_QUOTA_EXCEEDED",
  "message": "This org has used its 20 free suite generations for today. The limit resets at 00:00 UTC.",
  "retryAfterSeconds": 41230,
  "limit": 20
}

Next steps