Voice AI Testing

Overview

Voice Testing is an end-to-end feature that automates testing of your AI voice agents. Our platform simulates a call from an AI tester that interacts with your voice agent by following a pre-defined call script. After the call, the transcript is sent to a language model (LLM) along with your evaluation rubric. The LLM then determines if the call met the defined objectives.

Creating a Test Suite

Begin by creating a Test Suite that organizes and executes multiple test cases.

Step 1: Create a New Test Suite

Navigate to the Test tab in your dashboard and select Test Suites.
Click the Create Test Suite button.

Step 2: Define Test Suite Details

Enter a title for your Test Suite.
Select a phone number from your organization using the dropdown.
Make sure the phone number has an assistant assigned to it (if not, navigate to Phone Numbers tab to complete that action).

Step 3: Add Test Cases

Once your Test Suite is created, you will see a table where you can add test cases.
Click Add Test to add a new test case (up to 50 can be added).

Step 4: Configure Each Test Case

Script: Define how the testing agent should behave, including a detailed multi-step prompt to simulate how the customer should behave on the call.
Rubric List one or more questions that an LLM will use to evaluate if the call was successful.
Attempts: Choose the number of times (up to 5) the test case should be executed each time the Test Suite is run.

Step 5: Run and Review Tests

Click Run Tests to execute all test cases one by one.
While tests are running, you will see a loading state.
Upon completion, a table displays the outcomes with check marks (success) or x-marks (failure).
Click on a test row to view detailed results: a dropdown shows each attempt, the LLM’s reasoning, the transcript of the call, the defined script, and the success rubric.

Test Execution and Evaluation

When you run a Test Suite, the following steps occur:

Call Simulation: An AI voice agent dials your voice agent, executing the pre-defined script.
Transcript Capture: The entire conversation is transcribed, capturing both the caller’s behavior and your voice agent’s responses.
Automated Evaluation: The transcript, along with your Success Criteria, is processed by an LLM to determine if the call was successful.
Results Display: Each test case outcome is shown with details. Clicking on a test case reveals:
- The number of attempts made.
- The LLM’s reasoning for each attempt.
- The complete call transcript.
- The configured script and rubric.

Example Test Cases

Below are three example test cases to illustrate how you can configure detailed simulation scripts and evaluation rubrics.

Example 1: Billing Support

In this example, we will simulate a customer who is frustrated and calling about a billing discrepancy.

Script:

Script

1 1. Express anger over an unexpected charge and the current bill appearing unusually high.
2 2. Try to get a detailed explanation, confirming whether an overcharge occurred, and understanding the steps for resolution.
3 3. End the call.

Rubric:

Rubric

1 The voice agent acknowledges the billing discrepancy respectfully without dismissing the concern.

Example 2: Account Inquiry

Unlike in the previous example, this time we will provide a more free-form script for the test agent to follow.

Script:

Script

1 Simulate a customer inquiring about their account status with growing concern as unexplained charges appear in their statement.  
2 
3 Your primary objective is to clarify several unexplained charges by requesting a detailed breakdown of your recent transactions and ensuring your account balance is accurate.
4 
5 Begin the call by stating your name and expressing concern over unexpected charges. Ask straightforward questions and press for more details if the explanation is not satisfactory.

Rubric:

Rubric

1 1. The voice agent clearly presents the current account balance.
2 2. The voice agent provides a detailed breakdown of recent transactions.
3 3. The response addresses the customer's concerns in a calm and informative manner.

Example 3: Appointment Scheduling

This time, we will spin up an even more detailed personality for the test agent. By showing these varied styles of scripts, we hope to show the flexibility of the Voice Testing feature and how you can use it to meet your testing needs.

Script:

Script

1 Simulate a customer trying to schedule an appointment with a hint of urgency due to previous delays.  
2 
3 [Identity]
4 You are an organized customer who values efficiency and punctuality.
5 
6 [Personality]
7 While generally courteous and friendly, you are anxious due to previous delays in scheduling appointments, and your tone conveys urgency.
8 
9 [Goals]
10 Your goal is to secure an appointment at your preferred time, while remaining flexible enough to consider alternative timings if your desired slot is unavailable.
11 
12 [Interaction Style]
13 Begin the call by stating your need for an appointment, specifying a preferred date and time (e.g., next Monday at 3 PM). Request clear confirmation of your slot, and if unavailable, ask for suitable alternatives.

Rubric:

Rubric

1 1. The voice agent confirms the requested appointment time clearly and accurately.
2 2. The agent reiterates the appointment details to ensure clarity.
3 3. The scheduling process ends with a definitive confirmation message of the booked appointment.

1	1. Express anger over an unexpected charge and the current bill appearing unusually high.
2	2. Try to get a detailed explanation, confirming whether an overcharge occurred, and understanding the steps for resolution.
3	3. End the call.

1	Simulate a customer inquiring about their account status with growing concern as unexplained charges appear in their statement.
2
3	Your primary objective is to clarify several unexplained charges by requesting a detailed breakdown of your recent transactions and ensuring your account balance is accurate.
4
5	Begin the call by stating your name and expressing concern over unexpected charges. Ask straightforward questions and press for more details if the explanation is not satisfactory.

1	1. The voice agent clearly presents the current account balance.
2	2. The voice agent provides a detailed breakdown of recent transactions.
3	3. The response addresses the customer's concerns in a calm and informative manner.

1	Simulate a customer trying to schedule an appointment with a hint of urgency due to previous delays.
2
3	[Identity]
4	You are an organized customer who values efficiency and punctuality.
5
6	[Personality]
7	While generally courteous and friendly, you are anxious due to previous delays in scheduling appointments, and your tone conveys urgency.
8
9	[Goals]
10	Your goal is to secure an appointment at your preferred time, while remaining flexible enough to consider alternative timings if your desired slot is unavailable.
11
12	[Interaction Style]
13	Begin the call by stating your need for an appointment, specifying a preferred date and time (e.g., next Monday at 3 PM). Request clear confirmation of your slot, and if unavailable, ask for suitable alternatives.

1	1. The voice agent confirms the requested appointment time clearly and accurately.
2	2. The agent reiterates the appointment details to ensure clarity.
3	3. The scheduling process ends with a definitive confirmation message of the booked appointment.