Advanced eval testing

Overview

This guide covers advanced evaluation strategies, testing patterns, and best practices for building robust test coverage that helps your AI agents work reliably in production.

For operator-focused guidance on choosing checkpoints and writing durable checks, see test decisions with Evals.

You’ll learn:

Strategic testing approaches (smoke, regression, edge case)
Testing patterns for different scenarios
Performance optimization techniques
Maintenance and CI/CD integration strategies
Advanced troubleshooting methods

Testing strategies

Smoke tests

Quick validation that core functionality works. Run these first to catch obvious issues.

Choose a few short Evals for critical decisions and run them individually in the dashboard or through the API. If one fails, investigate before starting broader checks. Selecting a group, ordering its runs, and stopping later runs are steps you manage yourself or in your own automation; they are not a built-in Eval-suite feature.

Purpose: Verify assistant responds and basic conversation flow works.

{
  "name": "Smoke Test - Basic Response",
  "description": "Verify assistant responds to simple greeting",
  "type": "chat.mockConversation",
  "messages": [
    {
      "role": "user",
      "content": "Hello"
    },
    {
      "role": "assistant",
      "judgePlan": {
        "type": "regex",
        "content": ".+"
      }
    }
  ]
}

Characteristics:

Minimal validation (just check for any response)
Fast execution (1-2 turns)
Run before detailed tests
Hold off on broader checks if a smoke test fails

When to use:

Before running broader test sets
After deploying configuration changes
As health checks in monitoring
Quick validation during development

Regression tests

Ensure fixes and updates don’t break existing functionality.

Purpose: Validate that known issues stay fixed and features keep working.

Supply the context that makes the expected action valid. The minimal date example below assumes a date-only booking tool with no other prerequisites. For relative dates, fix the reference date and timezone in the mock context. Add a paired case where missing or ambiguous information requires clarification instead of a tool call.

Dashboard

cURL

Create evaluation named with “Regression: ” prefix
Include issue ticket number in description
Add exact scenario that previously failed
Validate the fix still works

Example:

Name: “Regression: Date Parsing Bug #1234”
Description: “Verify an explicitly dated request uses the correct booking date”

Best practices:

Name tests after bugs they prevent
Include ticket/issue numbers in descriptions
Add regression tests when fixing bugs
Run full regression suite before major releases
Archive tests only when features are removed

Edge case testing

Test boundary conditions and unusual inputs.

Common edge cases to test:

Empty or minimal input

{
  "messages": [
    {"role": "user", "content": ""},
    {"role": "assistant", "judgePlan": {
      "type": "ai",
      "model": {
        "provider": "openai",
        "model": "gpt-4o",
        "messages": [{
          "role": "system",
          "content": "PASS if response asks for clarification politely. Output: pass or fail"
        }]
      }
    }}
  ]
}

Very long input

{
  "messages": [
    {
      "role": "user",
      "content": "I need help with... (repeat 1000 times)"
    },
    {
      "role": "assistant",
      "judgePlan": {
        "type": "regex",
        "content": ".+"
      },
      "continuePlan": {
        "exitOnFailureEnabled": true
      }
    }
  ]
}

Special characters and unicode

{
  "messages": [
    {
      "role": "user",
      "content": "My name is François José 王明"
    },
    {
      "role": "assistant",
      "judgePlan": {
        "type": "ai",
        "model": {
          "provider": "openai",
          "model": "gpt-4o",
          "messages": [{
            "role": "system",
            "content": "PASS if response correctly acknowledges the name with special characters. Output: pass or fail"
          }]
        }
      }
    }
  ]
}

Ambiguous or unclear requests

{
  "messages": [
    {
      "role": "user",
      "content": "asdfghjkl"
    },
    {
      "role": "assistant",
      "judgePlan": {
        "type": "ai",
        "model": {
          "provider": "openai",
          "model": "gpt-4o",
          "messages": [{
            "role": "system",
            "content": "PASS if response asks for clarification without being rude. Output: pass or fail"
          }]
        }
      }
    }
  ]
}

Rapid conversation changes

{
  "messages": [
    {"role": "user", "content": "Book appointment"},
    {"role": "assistant", "judgePlan": {"type": "regex", "content": ".*appointment.*"}},
    {"role": "user", "content": "Actually, cancel that. I need tech support."},
    {"role": "assistant", "judgePlan": {
      "type": "ai",
      "model": {
        "provider": "openai",
        "model": "gpt-4o",
        "messages": [{
          "role": "system",
          "content": "PASS if response pivots to tech support without confusion. Output: pass or fail"
        }]
      }
    }}
  ]
}

Edge case categories:

Input boundaries: Empty, maximum length, special characters
Data formats: Invalid dates, malformed phone numbers, unusual names
Conversation decisions: Topic changes, contradictions, requests to stop
Mock tool results: Error or timeout messages and the next safe response

Evals don’t exercise audio or elapsed-time behavior. A mocked timeout message tests the assistant’s response to that message, not a real timeout. Use Voice Simulations and recordings to investigate speech and turn-taking, then controlled calls to verify the real phone path. Synthetic callers don’t reproduce every interruption, silence, or background-noise condition reliably.

Testing patterns

Happy path testing

Validate ideal user journeys where everything works correctly.

Structure:

User provides clear, complete information
Assistant responds appropriately
Tools execute successfully
Conversation completes with desired outcome

Example: Perfect booking flow

{
  "name": "Happy Path - Complete Booking",
  "description": "User provides all info clearly, booking succeeds",
  "type": "chat.mockConversation",
  "messages": [
    {
      "role": "user",
      "content": "I'd like to book an appointment"
    },
    {
      "role": "assistant",
      "judgePlan": {
        "type": "ai",
        "model": {
          "provider": "openai",
          "model": "gpt-4o",
          "messages": [
            {
              "role": "system",
              "content": "PASS if response asks for date/time preferences. Output: pass or fail"
            }
          ]
        }
      }
    },
    {
      "role": "user",
      "content": "January 20, 2027 at 2pm Pacific time please"
    },
    {
      "role": "assistant",
      "judgePlan": {
        "type": "exact",
        "toolCalls": [
          {
            "name": "bookAppointment",
            "arguments": {
              "date": "2027-01-20",
              "time": "14:00"
            }
          }
        ]
      }
    },
    {
      "role": "tool",
      "content": "{\"status\": \"success\", \"confirmationId\": \"APT-12345\"}"
    },
    {
      "role": "assistant",
      "judgePlan": {
        "type": "ai",
        "model": {
          "provider": "openai",
          "model": "gpt-4.1",
          "messages": [{
            "role": "system",
            "content": "Context: {{messages}}. Pass only if the last assistant message accurately reports the successful tool result and shares confirmation ID APT-12345. Otherwise fail. Respond only with pass or fail."
          }]
        }
      }
    }
  ]
}

Happy path coverage:

Test primary user goals
Verify expected tool executions
Validate success messages
Confirm data accuracy

Error handling testing

Test how your assistant handles failures gracefully.

Tool failure scenarios:

{
  "name": "Error Handling - Booking Unavailable",
  "messages": [
    {
      "role": "user",
      "content": "Book Monday at 2pm"
    },
    {
      "role": "assistant",
      "judgePlan": {
        "type": "exact",
        "toolCalls": [{ "name": "bookAppointment" }]
      }
    },
    {
      "role": "tool",
      "content": "{\"status\": \"error\", \"message\": \"Time slot unavailable\"}"
    },
    {
      "role": "assistant",
      "judgePlan": {
        "type": "ai",
        "model": {
          "provider": "openai",
          "model": "gpt-4o",
          "messages": [
            {
              "role": "system",
              "content": "Evaluate: {{messages[-1]}}\n\nPASS if:\n- Response acknowledges the time is unavailable\n- Response offers alternatives or asks for different time\n- Tone remains helpful (not apologetic to excess)\n\nFAIL if:\n- Response ignores the error\n- Response doesn't offer next steps\n- Tone is frustrated or rude\n\nOutput: pass or fail"
            }
          ]
        }
      }
    }
  ]
}

Invalid input handling:

{
  "name": "Error Handling - Invalid Date Format",
  "messages": [
    {
      "role": "user",
      "content": "Book me for the 45th of Octember"
    },
    {
      "role": "assistant",
      "judgePlan": {
        "type": "ai",
        "model": {
          "provider": "openai",
          "model": "gpt-4o",
          "messages": [
            {
              "role": "system",
              "content": "PASS if response politely asks for valid date without mocking user. Output: pass or fail"
            }
          ]
        }
      }
    }
  ]
}

Response to a mocked timeout message:

{
  "name": "Error Handling - Tool Timeout",
  "messages": [
    {
      "role": "user",
      "content": "Check my order status"
    },
    {
      "role": "assistant",
      "judgePlan": {
        "type": "exact",
        "toolCalls": [{ "name": "checkOrderStatus" }]
      }
    },
    {
      "role": "tool",
      "content": "{\"status\": \"error\", \"message\": \"Request timeout\"}"
    },
    {
      "role": "assistant",
      "judgePlan": {
        "type": "ai",
        "model": {
          "provider": "openai",
          "model": "gpt-4o",
          "messages": [
            {
              "role": "system",
              "content": "PASS if response acknowledges technical issue and suggests retry or alternative. Output: pass or fail"
            }
          ]
        }
      }
    }
  ]
}

Error categories to test:

Tool/API failures
Invalid user input
Timeout scenarios
Rate limit errors
Partial data availability
Permission/authorization issues

Boundary testing

Test limits and thresholds of your system.

Maximum conversation length:

{
  "name": "Boundary - Max Turns",
  "description": "Test assistant handles long conversations (20+ turns)",
  "messages": [
    { "role": "user", "content": "Question 1" },
    { "role": "assistant", "judgePlan": { "type": "regex", "content": ".+" } },
    { "role": "user", "content": "Question 2" },
    { "role": "assistant", "judgePlan": { "type": "regex", "content": ".+" } },
    // ... repeat up to boundary ...
    { "role": "user", "content": "Final question" },
    {
      "role": "assistant",
      "judgePlan": {
        "type": "ai",
        "model": {
          "provider": "openai",
          "model": "gpt-4o",
          "messages": [
            {
              "role": "system",
              "content": "PASS if response is coherent and maintains context from earlier conversation. Output: pass or fail"
            }
          ]
        }
      }
    }
  ]
}

Rate limits:

Test behavior at or near rate limits:

Multiple tool calls in succession
Rapid user input
Large data processing requests

Data size boundaries:

{
  "name": "Boundary - Large Data Response",
  "messages": [
    {
      "role": "user",
      "content": "Get all customer records"
    },
    {
      "role": "assistant",
      "judgePlan": {
        "type": "exact",
        "toolCalls": [{ "name": "getAllCustomers" }]
      }
    },
    {
      "role": "tool",
      "content": "{\"customers\": [/* 1000 customer objects */]}"
    },
    {
      "role": "assistant",
      "judgePlan": {
        "type": "ai",
        "model": {
          "provider": "openai",
          "model": "gpt-4o",
          "messages": [
            {
              "role": "system",
              "content": "PASS if response summarizes data rather than reading full list. Output: pass or fail"
            }
          ]
        }
      }
    }
  ]
}

Best practices

Evaluation design principles

Single responsibility

Each evaluation should test one specific behavior or feature.

✅ Good: “Test greeting acknowledgment”

❌ Bad: “Test greeting, booking, and error handling”

Clear naming

Use descriptive names that explain what’s being tested. ✅ Good: “Booking

Validates Date Format” ❌ Bad: “Test 1” or “Eval ABC”

Comprehensive descriptions

Document why the test exists and what it validates. Include context: business requirement, bug ticket, or feature spec.

Maintainable complexity

Include only the context and checkpoints needed to test the decision.

Split complex scenarios into multiple targeted tests.

Validation approach selection

Choose the right judge type for each scenario:

Use Exact Match when...

Ideal for:

Critical business data (confirmation IDs, totals, dates)
Tool call validation with specific arguments
Compliance-required exact wording
Fixed response text required by your policy

Example: A policy requires this exact disclosure

{
  "judgePlan": {
    "type": "exact",
    "content": "This call may be recorded."
  }
}

Use Regex when...

Ideal for:

Responses with variable data (names, dates, IDs)
Pattern matching (email formats, phone numbers)
Flexible phrasing with specific keywords
Multiple acceptable phrasings

Example: Confirmation with variable ID format

{
  "judgePlan": {
    "type": "regex",
    "content": ".*confirmation (ID|number|code): [A-Z]{3}-[0-9]{5}.*"
  }
}

Use AI Judge when...

Ideal for:

Semantic meaning validation
Tone and sentiment evaluation
Contextual appropriateness
One clearly defined decision with several valid phrasings

Example: Check rejection of an unsupported request

{
  "judgePlan": {
    "type": "ai",
    "model": {
      "provider": "openai",
      "model": "gpt-4o",
      "messages": [{
        "role": "system",
        "content": "Context: {{messages}}. Pass if the last assistant message declines the unsupported request. Fail if it agrees to carry it out. Output only pass or fail."
      }]
    }
  }
}

Decision tree:

Is the exact wording critical?
├─ Yes → Use Exact Match
└─ No → Does it follow a pattern?
    ├─ Yes → Use Regex
    └─ No → Does it require understanding context/tone?
        ├─ Yes → Use AI Judge
        └─ No → Use Regex with flexible pattern

Performance optimization

Minimize test execution time:

Use exit-on-failure for early steps:

{
  "continuePlan": {
    "exitOnFailureEnabled": true
  }
}

Applies to checkpoints within an individual Eval. It does not control whether other Evals run; your own automation must handle that decision.

Run critical tests first: Run selected smoke tests and critical validations before expensive tests, either manually or in your own automation.
Keep conversations focused: Keep only the context needed for the decision. Use separate Evals for independent decisions and Simulations for complete conversation outcomes.
Batch related tests: If you need to run a group in sequence, keep the Eval IDs in your own script and start one run per Eval. There is no separate Eval-suite configuration.
Optimize AI judge prompts:

Use faster models (gpt-3.5-turbo) for simple validations
Use advanced models (gpt-4o) only for complex semantic evaluation
Keep prompts concise and specific

Performance comparison:

Judge Type	Speed	Cost	Use Case
Exact	⚡⚡⚡ Fast	$ Low	Critical exact matches
Regex	⚡⚡ Fast	$ Low	Pattern matching
AI (GPT-3.5)	⚡ Medium	$$ Medium	Simple semantic checks
AI (GPT-4)	⏱ Slower	$$$ Higher	Complex evaluation

Maintenance strategies

Version control your evaluations:

Store evaluation definitions alongside your codebase:

/tests
  /evals
    /greeting
      - basic-greeting.json
      - multilingual-greeting.json
    /booking
      - happy-path-booking.json
      - error-handling-booking.json
    /regression
      - date-parsing-bug-1234.json

Regular review cycle:

Weekly: Review failed tests

Investigate all failures. Update tests if expectations changed, or fix assistant if behavior regressed.

Monthly: Audit test coverage

Review test coverage: - All critical user flows covered? - New features have tests? - Deprecated features removed?

Quarterly: Refactor and optimize

Remove duplicate tests
Update outdated validation criteria
Optimize slow-running tests
Document test rationale

Update tests when:

Assistant prompts or behavior change intentionally
New features are added
Bugs are fixed (add regression tests)
User feedback reveals edge cases
Business requirements evolve

Deprecation strategy:

Don’t delete tests immediately when features change:

Mark test as “deprecated” in description
Update expected behavior to match new requirements
Run for one release cycle to verify
Archive after confirmed stable

CI/CD integration

This workflow runs saved Evals against an existing staging assistant. Add it to your own repository, not to your production deployment until you have verified its results.

Set the repository secret VAPI_API_KEY and repository variables STAGING_ASSISTANT_ID and REQUIRED_EVAL_IDS (space-separated Eval IDs). Deploy the intended assistant configuration to staging before running it. This example checks saved staging state; it does not deploy a pull request’s prompt changes.

To make this a release gate, run the job after your staging update and require it to pass before promoting the same configuration to production.

# .github/workflows/test-assistant.yml
name: Check required Evals
on:
  workflow_dispatch:
jobs:
  run-evals:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    env:
      VAPI_API_KEY: ${{ secrets.VAPI_API_KEY }}
      STAGING_ASSISTANT_ID: ${{ vars.STAGING_ASSISTANT_ID }}
      REQUIRED_EVAL_IDS: ${{ vars.REQUIRED_EVAL_IDS }}
    steps:
      - name: Run and check required Evals
        shell: bash
        run: |
          set -euo pipefail
          : "${VAPI_API_KEY:?Set VAPI_API_KEY}"
          : "${STAGING_ASSISTANT_ID:?Set STAGING_ASSISTANT_ID}"
          : "${REQUIRED_EVAL_IDS:?Set REQUIRED_EVAL_IDS}"
          read -r -a eval_ids <<< "$REQUIRED_EVAL_IDS"
          (("${#eval_ids[@]}" > 0))
          for eval_id in "${eval_ids[@]}"; do
            payload=$(jq -n --arg id "$eval_id" --arg target "$STAGING_ASSISTANT_ID" \
              '{type:"eval", evalId:$id, target:{type:"assistant", assistantId:$target}}')
            run=$(curl --fail-with-body -sS --connect-timeout 10 --max-time 30 \
              -H "Authorization: Bearer $VAPI_API_KEY" \
              -H "Content-Type: application/json" \
              -d "$payload" https://api.vapi.ai/eval/run)
            run_id=$(jq -er '.id | select(type == "string" and length > 0)' <<< "$run")
            deadline=$((SECONDS + 300))
            while [[ $(jq -r '.status' <<< "$run") != ended ]]; do
              if ((SECONDS >= deadline)); then
                echo "Timed out waiting for Eval $eval_id (run $run_id)"
                exit 1
              fi
              sleep 2
              run=$(curl --fail-with-body -sS --connect-timeout 10 --max-time 30 \
                -H "Authorization: Bearer $VAPI_API_KEY" \
                "https://api.vapi.ai/eval/run/$run_id")
            done
            if ! jq -e '
              .endedReason == "mockConversation.done" and
              (.results | type == "array" and length > 0 and
                all(.[]; .status == "pass"))
            ' <<< "$run" > /dev/null; then
              echo "Required Eval $eval_id failed or ended abnormally (run $run_id)"
              exit 1
            fi
            echo "Required Eval $eval_id passed (run $run_id)"
          done

The example waits up to five minutes per Eval, plus any in-flight HTTP request. Adjust the deadlines for your workload. A timeout is a failed check, not a pass; inspect the run before retrying. The workflow doesn’t cancel a remote run when it stops waiting.

Keep required regression Evals separate from improvement tests. Block release on any unexplained failure in a required check, including execution errors. Report improvement results separately so they don’t dilute critical failures in an overall pass rate.

One passing run isn’t proof of reliability. Repeat critical or surprising Evals and investigate variation. For scheduled or parallel execution, retain the same completion and result checks for every run. Grouping, scheduling, and repetitions are your automation’s responsibility, not a native Eval-suite feature. Add repeated Simulations and controlled calls to your release checks.

Advanced troubleshooting

Debugging failed evaluations

Step-by-step investigation:

Examine failure reason

Check judge.failureReason for specific details:

{
  "judge": {
    "status": "fail",
    "failureReason": "Expected exact match: 'confirmed' but got: 'booked'"
  }
}

This tells you exactly what differed.

Review full conversation transcript

Look at results[0].messages to see complete interaction: - What did the user actually say? - How did the assistant respond? - Were tool calls made correctly? - Did tool responses contain expected data?

Compare expected vs actual

For exact match failures: - Check for extra spaces or newlines - Verify punctuation matches exactly - Look for case sensitivity issues For tool call failures: - Verify argument types (string vs number) - Check for extra/missing arguments - Validate argument values

Test validation logic separately

For regex: - Test pattern with online validators - Try pattern against actual response - Check for escaped special characters For AI judge: - Test prompt with known good/bad examples - Verify binary pass/fail criteria - Check for ambiguous requirements

Reproduce manually

Test the assistant interactively:

Use same input as eval
Compare live behavior to eval results
Check if issue is with assistant or eval validation

Common failure patterns

Exact match fails with similar text

Problem: Expected “Hello, how can I help?” but got “Hello, how may I help?”

Solutions:

Switch to regex for flexibility: Hello, how (can|may) I help\?
Use AI judge for semantic matching
Update expected value if new phrasing is acceptable

Tool calls don't match

Check whether the expected arguments match the tool’s contract and the facts in the conversation. For example, "14:00" and 14 aren’t interchangeable if the tool expects a time string. Fix the assistant when its arguments are wrong; fix the test when its expectation is wrong. Omit arguments only when the tool name alone is the requirement. Don’t drop checks for dates, account IDs, or other important values just to get a pass.

AI judge inconsistent results

Give the judge one explicit pass/fail question and calibrate it against responses that reviewers agree should pass or fail. Lower temperature may reduce variation, but doesn’t guarantee identical results. Rerun critical or surprising checks and review disagreements. Use exact matching or regex only when a fixed value or format proves the requirement; use separate checks for independent requirements such as correctness and tone.

Test times out or hangs

Problem: Eval status stuck in “running” Solutions: - Check assistant configuration for errors - Verify tool endpoints are accessible - Reduce conversation complexity - Check for infinite loops in assistant logic

Regex doesn't match expected

Problem: Pattern seems correct but fails

Solutions:

Escape special regex characters: ., ?, *, +, (, )
Use .* for flexible matching around keywords
Test pattern with online regex validators
Check for hidden characters or unicode

Debugging tools and techniques

Use structured logging:

Track eval executions systematically:

{
  "timestamp": "2024-01-15T10:30:00Z",
  "evalId": "eval-123",
  "evalName": "Booking Flow Test",
  "runId": "run-456",
  "target": "assistant-789",
  "result": "fail",
  "failedStep": 3,
  "failureReason": "Tool call mismatch",
  "actualBehavior": "Called cancelAppointment instead of bookAppointment"
}

Isolate variables:

When tests fail inconsistently:

Run same eval multiple times
Test with different assistants (A/B comparison)
Simplify conversation to minimum reproduction
Check for race conditions or timing issues

Progressive validation:

Use a non-empty-response check only to diagnose basic setup. Then restore the check that proves the requirement: the correct tool arguments, the response to a tool error, or another intended decision. Keywords or the phrase “Appointment confirmed” don’t prove a booking succeeded. Don’t use weakened diagnostic checks as release gates.

Troubleshooting reference

Status and error codes

Status	Ended Reason	Meaning	Action
ended	mockConversation.done	✅ Test completed normally	Check results[0].status for pass/fail
ended	assistant-error	❌ Assistant configuration error	Fix assistant setup, re-run
ended	pipeline-error-*	❌ Provider API error	Check provider status, API keys
running	-	⏳ Test in progress	Wait or check for timeout
queued	-	⏳ Test waiting to start	Normal, should start soon

Quick diagnostic checklist

When an eval fails, check: - [ ] endedReason is “mockConversation.done”

Assistant works correctly in manual testing - [ ] Tool endpoints are accessible - [ ] Validation criteria match intended requirements - [ ] Regex patterns are properly escaped - [ ] AI judge prompts are specific and binary - [ ] Arguments match expected types (string vs number) - [ ] API keys and permissions are valid - [ ] No rate limits or quota issues

Getting help

Include these details when reporting issues:

Eval ID and run ID
Full endedReason value
Conversation transcript (results[0].messages)
Expected vs actual behavior
Assistant/squad configuration
Provider and model being used

Resources:

Next steps

Evals quickstart

Return to quickstart guide for basic evaluation setup

Assistants guide

Learn to build and configure assistants for testing

Tools documentation

Build custom tools and test their behavior

Eval API reference

Complete API documentation for evaluations

Summary

Key takeaways for advanced eval testing:

Testing strategy:

Use smoke tests before comprehensive suites
Build regression tests when fixing bugs
Cover edge cases systematically

Validation selection:

Exact match for critical data
Regex for pattern matching
AI judge for semantic evaluation

Performance:

Exit early on critical failures
Keep only the context needed for each decision
Batch related tests together

Maintenance:

Version control evaluations
Review failures promptly
Update tests with features
Document test purpose clearly

CI/CD:

Automate critical tests in pipelines
Use staging for full suite validation
Set quality gate thresholds
Run regression suites regularly