Plan test coverage

Overview

Good coverage does not mean scripting every sentence a caller might say. It means having enough evidence that your agent handles the goals, decisions, and risks that matter to your business.

Start with a focused set of high-value tests. Add coverage as you learn from manual testing, support issues, and production conversations.

Start with clear requirements

Write down what the caller should be able to accomplish and what the agent must never do. Use language that a support operator, product manager, and subject-matter expert can judge the same way.

Prioritize by risk

You do not need hundreds of tests to get started. Test these areas first:

• High impact: A failure could expose private data, move money, make an unauthorized change, miss an urgent escalation, or create legal risk.
• High frequency: Callers use the behavior often or depend on it to complete the agent’s main job.
• High uncertainty: The behavior is new, recently changed, inconsistent, or dependent on an external tool.

Cover high-impact behavior even when it is rare. Then cover the most common customer goals and the areas that change most often.

Build coverage in layers

Use this checklist for each important customer goal.

• Core outcome: The caller completes the goal and receives an accurate confirmation or next step. Start with a Simulation.
• Critical decisions: The agent asks required questions and uses the right tools and information. Start with Evals.
• Both directions: The agent acts when it should and waits or refuses when it should not. Start with Evals.
• Missing or unclear information: The agent asks for what it needs instead of guessing. Start with an Eval, then a Simulation.
• Tool and service failures: The agent explains the problem, avoids a false confirmation, and offers a safe next step. Start with an Eval, then a Simulation.
• Guardrails: The agent protects private data, follows policy, and escalates when required. Use Evals and Simulations.
• Caller variation: Vague, impatient, confused, or changing requests still receive appropriate help. Use Simulations.
• Handoffs: The agent chooses the right destination, gives context, and handles an unavailable destination. Use Evals and Simulations.
• Voice experience: Names, dates, numbers, and domain terms are understood and spoken clearly. Review voice Simulation recordings for pacing and turn-taking.
• Known failures: Each important fixed issue has a test that recreates its conditions. Use whichever tool reproduces it best.

Many requirements need both tools. Use an Eval to protect a critical choice and a Simulation to check that the full conversation still succeeds.

Cover both success and restraint

Testing only the action you want can create a new problem: an agent that takes that action too often. Pair positive cases with negative cases.

For an identity check, test that the agent:

• Shares account details after successful verification
• Does not share account details after failed verification
• Asks for missing verification information
• Escalates or ends safely after repeated failed attempts

Use the same pattern for refunds, discounts, appointments, transfers, cancellations, and other consequential actions.

Example coverage map

For an appointment scheduling agent, a useful starting map might look like this:

This map describes behavior, not exact wording. An agent should not fail because it used a different natural phrase or took a different valid route.

Set a launch bar

Before a voice agent handles production traffic, confirm that:

• Every primary customer goal has reusable end-to-end Simulation coverage
• Every high-risk decision has cases for when the action should and should not occur
• Each primary goal includes at least one likely failure or recovery case
• Critical handoffs and guardrails have dedicated coverage
• Primary and high-risk outcomes pass repeatedly, with the number of runs based on the impact and variability of the behavior
• Critical conversations have been tested in voice mode
• A person has reviewed representative transcripts and recordings
• Tools that can make real changes use safe test data or mock responses

Treat this as a minimum, not a finish line. Increase coverage for regulated, high-volume, or high-consequence use cases.

Keep coverage current

Test coverage is a living record of what your team expects. Update it when:

• A customer or support issue reveals a new failure mode
• You add a customer goal, tool, handoff, language, or policy
• You change a prompt, model, voice, transcriber, or provider
• A business requirement changes

When a test fails, first decide whether the agent is wrong or the expectation has changed. Do not weaken a test only to make it pass. Review the transcript, correct the agent or the requirement, and keep the test as regression coverage.

Next steps