Testing voice agents

Overview

Voice agents combine speech recognition, language models, voice generation, and tools. A small change to any part can improve one conversation and break another. Because the agent can respond differently to the same caller, one successful test call is not enough.

Calling the agent yourself is a useful starting point. As you add customer flows, checking them all by hand before every change becomes difficult to sustain.

Systematic testing turns your expectations into checks you can rerun after you change a prompt, model, voice, transcriber, tool, or configuration. It helps your team agree on what the agent must do and find regressions before callers do.

Why testing pays off

Testing requires time up front, but its value grows as your agent changes. A useful test set helps you:

• Define success clearly. Product, operations, and support teams agree on the expected behavior before launch.
• Make changes with confidence. Rerun the same checks after changing a prompt, tool, model, voice, or transcriber.
• Separate regressions from normal variation. Compare repeated results instead of relying on how one call felt.
• Learn from real failures. Turn support issues and production mistakes into tests that prevent the same problem from returning.
• Evaluate upgrades faster. Compare a new model or provider against the expectations you already defined.

Use Evals for decisions and Simulations for outcomes

The difference starts with what you provide:

• For an Eval, provide the conversation up to a decision. The agent generates its next response, which is checked using exact matching, a pattern, or an AI judge. You can rerun the same context, but the response and AI judgment may vary.
• For a Simulation, provide the caller’s goal and relevant facts. An AI tester acts as the caller and adapts during a complete conversation. Define the outcome and guardrails that should hold across different valid paths.

Use an Eval when you need to check a decision at a known point. Use a Simulation when you need to check the result of a complete conversation. Most production voice agents need both.

For example, use an Eval to check that an agent calls the rescheduling tool only after it verifies the caller. Use a Simulation to check that a caller can reschedule, hear the correct new time, and finish the conversation successfully. The Simulation should pass even if the agent asks the questions in a different valid order.

Read the Evals quickstart for specific decision checks. Read the Simulations overview for complete conversation testing.

Test in layers

Plan coverage from the customer outcomes you need. When you run the tests, start with narrow decision checks for fast feedback, then expand to complete chat and voice conversations. Each layer catches a different kind of problem.

Know what automated tests cover

Evals test text-based decisions. Voice Simulations exercise speech recognition, voice output, and conversation timing, but synthetic callers do not fully represent real callers and their environments. Use manual calls and production recordings to test background noise, degraded audio, a range of accents, silences, and interruptions that your Simulations cannot reliably reproduce.

Simulation scores also depend on the evidence available to the judge. A tool reporting success does not independently prove that an external system changed. See how to write success criteria for guidance on tool results and mocked responses.

What good testing looks like

A healthy test set is tied to real requirements and real caller behavior. It covers:

• The main outcomes callers expect
• The critical decisions the agent makes along the way
• Both when an action should happen and when it should not
• Common failure and recovery paths
• High-risk privacy, compliance, financial, and handoff behavior
• Representative caller styles and important voice-specific cases
• Every important issue you have already fixed

Do not aim to script every possible conversation. Prioritize the behavior with the greatest customer or business impact, then expand coverage as you learn from test runs and production calls. See Plan test coverage for a practical framework.

Next steps