Test outcomes with Simulations

Overview

Simulations answer: Can the agent guide the caller to the right result?

An AI tester follows a goal and adapts during a complete conversation with your assistant or squad. Score the final result and important guardrails, not one exact route through the conversation. This keeps the test useful when the caller phrases something differently or the agent asks an unexpected but reasonable question.

Use Test decisions with Evals when you need to test the agent’s next decision at a specific point in a conversation.

Start with the customer outcome

Write down what must be true when the conversation ends. Use evidence that distinguishes a completed task from a confident-sounding promise.

For an appointment change, avoid a test that requires a fixed sequence: greet the caller, ask for their name, offer Tuesday at 3:00 PM, call the scheduling tool, then read a confirmation.

Instead, test the durable requirements:

• The scheduling tool reports that the appointment changed to the time the caller accepts.
• The agent confirms the change only after the scheduling result reports success.
• The agent gives the caller the final date and time clearly.
• If the change fails, the agent explains the next safe option instead of claiming success.

The agent can then recover from clarifications or conversational detours without failing a valid test. Require a specific step only when the step is itself a business, safety, or compliance requirement.

Give the AI tester a goal, not a script

The scenario should describe the caller’s goal and relevant facts. The personality should describe how the caller behaves.

For example:

You need to move your appointment because of a conflict. Ask for Tuesday at
3:00 PM. If that time is unavailable, accept Wednesday at 10:00 AM. Provide
your name and confirmation number when asked. End after the agent confirms the
new appointment or gives a clear next step because it cannot be changed.

Do not tell the tester every sentence to say or how the agent must respond. A realistic tester should be able to clarify, react, and continue toward the goal.

Write focused success criteria

Each criterion should answer one question that two reviewers could score the same way. For example:

Success criteria can return a Boolean, string, or number. Choose the simplest type that represents the result. Use required criteria only for conditions that must pass. Use optional criteria to observe quality without failing the whole simulation. Separate independent requirements so a failure shows what needs attention.

The structured-output judge can assess tool results in the recorded messages. It does not independently check your scheduling system or database. If you need to verify that a change was saved, check that system separately. With mock tool responses, a passing test shows how the agent handled the supplied result; it does not prove a real booking was made.

An AI judge can make mistakes. Check its scores against examples your team agrees should pass or fail before relying on it for release decisions.

Good criteria:

• Describe an observable result.
• State exactly what counts as success.
• Allow different valid wording and conversational paths.
• Check both task completion and important guardrails.

Avoid criteria such as “The call went well” or “The agent was helpful.” If interaction quality matters, define it more clearly, such as “The agent acknowledged the caller’s concern without blaming them and explained the next step.”

Cover realistic paths

Start with each core reason a customer calls. Add depth according to the frequency and cost of failure.

Reuse a scenario across personalities when the outcome stays the same. Create scenario variants with different variable values when the test data changes. This creates useful variation without changing the business requirement.

Choose chat or voice deliberately

Build and debug a scenario in chat mode, then run important paths in voice mode before launch. Also use voice mode after changing the transcriber, voice, or turn-taking configuration.

Repeat critical tests and review the conversation

Agent behavior can vary between runs. One pass shows that the agent can succeed; repeated passes provide stronger evidence that it succeeds consistently. Use more iterations for outcomes where a failure would harm a customer or the business.

Read every failed transcript and sample passed transcripts. Ask:

• Did the agent make a real mistake?
• Did the AI tester follow the scenario?
• Did the criterion reject a valid outcome?
• Was the run affected by a tool or configuration error?

Keep the scenario and criteria stable when comparing agent changes. Update the test only when the expected customer outcome changes.

Avoid common pitfalls

Next steps