Changelog

Evaluation System Foundation

1. Evaluation Framework: You can now systematically test your Vapi voice assistants with the new Eval system. Create comprehensive test scenarios to validate assistant behavior, conversation flow, and tool usage through mock conversations.

2. Mock Conversation Builder: Design test conversations using Eval.messages with support for multiple message types:

   • ChatEvalUserMessageMock: Simulate user inputs and questions
   • ChatEvalSystemMessageMock: Inject system messages mid-conversation
   • ChatEvalToolResponseMessageMock: Mock tool responses for consistent testing
   • ChatEvalAssistantMessageEvaluation: Define evaluation checkpoints

3. Evaluation Types: Currently focused on chat.mockConversation type evaluations, with the framework designed to support additional evaluation methods in future releases.

4. Evaluation Management: Organize your tests with CreateEvalDTO and UpdateEvalDTO:

   • name: Descriptive names up to 80 characters (e.g., “Customer Support Flow Validation”)
   • description: Detailed descriptions up to 500 characters explaining the test purpose
   • messages: The complete mock conversation flow

5. Evaluation Endpoints: Access your evaluations through the new /eval endpoint family:

   • GET /eval: List all evaluations with pagination support
   • POST /eval: Create new evaluations
   • GET /eval/{id}: Retrieve specific evaluation details
   • PUT /eval/{id}: Update existing evaluations

6. Judge Plan Architecture: Define how assistant responses are validated using AssistantMessageJudgePlan with three evaluation methods:

   • Exact Match: AssistantMessageJudgePlanExact for precise content and tool call validation
   • Regex Pattern: AssistantMessageJudgePlanRegex for flexible pattern-based evaluation
   • AI Judge: AssistantMessageJudgePlanAI for intelligent evaluation using LLM-as-a-judge

Testing Capabilities