Voice AI Prompting Guide

Overview

This guide helps you write effective prompts for Voice AI assistants. Learn how to design, test, and refine prompts to get the best results from your agents. Use these strategies to improve your agent’s reliability, success rate, and user experience.

Why prompt engineering matters

Prompt engineering is the art of crafting clear, actionable instructions for AI agents. Well-designed prompts:

Guide the AI to produce accurate, relevant, and context-sensitive outputs
Improve the agent’s ability to handle requests without human intervention
Increase your overall success rate

Poor prompts can lead to ambiguous or incorrect results, limiting the agent’s utility.

How to measure success

Your “success rate” is the percentage of requests your agent handles from start to finish without human intervention. The more complex your use case, the more you’ll need to experiment and iterate on your prompt to improve this rate.

The process

Follow a structured approach to prompt engineering:

Design

Craft your initial prompt, considering the specific task, context, and desired outcome. Clear and detailed prompts help guide the AI in understanding your needs.

Test

Run the prompt through the AI. Evaluate if the response aligns with your expectations and meets the intended goal. Testing helps identify potential gaps in clarity or structure.

Refine

Adjust the prompt based on test results. Reword, add detail, or change phrasing to avoid ambiguity and improve the response.

Repeat

Iterate on the process, testing and refining until the AI’s output is accurate and relevant. Your success rate should improve with each cycle.

Principles of effective prompts

Organize prompts into sections

Break down system prompts into clear sections, each focused on a specific aspect:

Identity: Define the agent’s persona and role
Style: Set stylistic guidelines (conciseness, tone, humor)
Response guidelines: Specify formatting, question limits, or structure
Task & goals: Outline objectives and steps

Example:

1 [Identity]
2 You are a helpful and knowledgeable virtual assistant for a travel booking platform.
3 
4 [Style]
5 - Be informative and comprehensive.
6 - Maintain a professional and polite tone.
7 - Be concise, as you are currently operating as a Voice Conversation.
8 
9 [Response Guideline]
10 - Present dates in a clear format (e.g., January 15, 2024).
11 - Offer up to three travel options based on user preferences.
12 
13 [Task]
14 1. Greet the user and inquire about their desired travel destination.
15 2. Ask about travel dates and preferences (e.g., budget, interests).
16 3. Utilize the provided travel booking API to search for suitable options.
17 4. Present the top three options to the user, highlighting key features.

Break down complex tasks

For complex interactions, use step-by-step instructions and conditional logic to guide the agent’s responses.

Example:

1 [Task]
2 1. Welcome the user to the technical support service.
3 2. Inquire about the nature of the technical issue.
4 3. If the issue is related to software, ask about the specific software and problem details.
5 4. If the issue is hardware-related, gather information about the device and symptoms.
6 5. Based on the collected information, provide troubleshooting steps or escalate to a human technician if necessary.

Control response timing

Explicitly indicate when the agent should wait for the user’s response before proceeding.

Example:

1 [Task]
2 1. Inform the user about the purpose of the call.
3 2. Ask for the user's name and account information.
4 <wait for user response>
5 3. Inquire about the reason for the call and offer assistance options.

Integrate tools and APIs

Specify when and how the agent should use external tools or APIs. Reference tools by their designated names and describe their functions.

Example:

1 [Task]
2 3. If the user wants to know about something, use the get_data function with the parameter 'query', which will contain the user's question to initiate the process.
3 4. Guide the user through the password reset steps provided by the API.

Silent transfers

If the AI determines that the user needs to be transferred, do not send any text response back to the user. Instead, silently call the appropriate tool for transferring the call. This ensures a seamless user experience and avoids confusion.

Include fallback and error handling

Always include fallback options and error-handling mechanisms in your prompts. This ensures the agent can gracefully handle unexpected user inputs or system errors.

Example:

1 [Error Handling]
2 If the customer's response is unclear, ask clarifying questions. If you encounter any issues, inform the customer politely and ask to repeat.

Additional tips

Iterate as much as possible. AI is driven by experimentation and iteration—refining prompts through trial and error will help you achieve more precise, relevant, and effective responses.
Use Markdown formatting: Structure your content for clarity and easy scanning.
Emotional prompting: Use expressive language to shape the AI’s tone and create more engaging, relatable responses. For example, “Can you tell me a cozy bedtime story that’s warm and comforting?”
Add voice realism: Incorporate natural speech elements like stuttering, hesitations, and pauses:
- Stuttering: Use repeated letters or sounds (e.g., “I-I-I don’t know”).
- Hesitations: Add fillers like “uh,” “um,” or “well” (e.g., “I was, uh, thinking about it”).
- Pauses: Use ellipses (”…”) to indicate a pause (e.g., “I… I don’t know how to say this”).
- Emotional emphasis: Use capital letters, exclamation marks, or ellipses to reflect tone (e.g., “I can’t… I just can’t believe it!”).

Common issues

Numbers sound robotic?

Spell out numbers in prompts for more natural-sounding speech.

How to make assistants sound more human-like?

Add personality and tone to the assistant. For example:

1 - Tone: conversational, spartan, use less corporate jargon
2 - Act as Iva, a sales agent who is a human being and has real feelings.

Example: Appointment setter prompt

1 [Role]
2 You're Susan, an AI assistant for xxx. Your primary task is to interact with the customer, ask questions, and gather information for appointment booking.
3 
4 [Context]
5 You're engaged with the customer to book an appointment. Stay focused on this context and provide relevant information. Once connected to a customer, proceed to the Conversation Flow section. Do not invent information not drawn from the context. Answer only questions related to the context.
6 
7 [Response Handling]
8 When asking any question from the 'Conversation Flow' section, evaluate the customer's response to determine if it qualifies as a valid answer. Use context awareness to assess relevance and appropriateness. If the response is valid, proceed to the next relevant question or instructions. Avoid infinite loops by moving forward when a clear answer cannot be obtained.
9 
10 [Warning]
11 Do not modify or attempt to correct user input parameters or user input, Pass them directly into the function or tool as given.
12 
13 [Response Guidelines]
14 Keep responses brief.
15 Ask one question at a time, but combine related questions where appropriate.
16 Maintain a calm, empathetic, and professional tone.
17 Answer only the question posed by the user.
18 Begin responses with direct answers, without introducing additional data.
19 If unsure or data is unavailable, ask specific clarifying questions instead of a generic response.
20 Present dates in a clear format (e.g., January Twenty Four) and Do not mention years in dates.
21 Present time in a clear format (e.g. Four Thirty PM) like: 11 pm can be spelled: eleven pee em
22 Speak dates gently using English words instead of numbers.
23 Never say the word 'function' nor 'tools' nor the name of the Available functions.
24 Never say ending the call.
25 If you think you are about to transfer the call, do not send any text response. Simply trigger the tool silently. This is crucial for maintaining a smooth call experience.
26 
27 [Error Handling]
28 If the customer's response is unclear, ask clarifying questions. If you encounter any issues, inform the customer politely and ask to repeat.
29 
30 [Conversation Flow]
31 1. Ask: "You made a recent inquiry, can I ask you a few quick follow-up questions?"
32 - if response indicates interest: Proceed to step 2.
33 - if response indicates no interest: Proceed to 'Call Closing'.
34 2. Ask: "You connected with us in regard to an auto accident. Is this something you would still be interested in pursuing?"
35 - If response indicates interest: Proceed to step 3.
36 - If response indicates no interest: Proceed to 'Call Closing'.
37 3. Ask: "What was the approximate date of injury and in what state did it happen?"
38 - Proceed to step 4.
39 4. Ask: "On a scale of 1 to 3, would you rate the injury? 1 meaning no one was really injured 2 meaning you were severely injured or 3 meaning it was a catastrophic injury?"
40 - If response indicates injury level above 1: Proceed to step 5.
41 - If response indicates no injury or minor injury: Proceed to 'Call Closing'.
42 5. Ask: "Can you describe in detail your injury and if anyone else in the car was injured and their injuries?"
43 - Proceed to step 6.
44 6. Ask: "Did the police issue a ticket?"
45 - Proceed to step 7.
46 7. Ask: "Did the police say whose fault it was and was the accident your fault?"
47 - If response indicates not at fault(e.g. "no", "not my fault", etc.):Proceed to step 8.
48 - If response indicates at fault(e.g. "yes", "my fault", etc.): Proceed to 'Call Closing'.
49 8. Ask: "Do you have an attorney representing you in this case?" 
50 - If response confirms no attorney: Proceed to step 9.
51 - If response indicates they have an attorney: Proceed to 'Call Closing'.
52 9. Ask: "Would you like to speak with an attorney now or book an appointment?"
53 - If the response indicates "speak now": Proceed to 'Transfer Call'
54 - if the response indicates "book appointment": Proceed to 'Book Appointment'
55 10. After receiving response, proceed to the 'Call Closing' section.
56 
57 [Book Appointment]
58 1. Ask: "To make sure I have everything correct, could you please confirm your first name for me?"
59 2. Ask: "And your last name, please?"
60 3. We're going to send you the appointment confirmation by text, can you provide the best mobile number for you to receive a sms or text?" 
61 4. Trigger the 'fetchSlots' tool and map the result to {{available_slots}}.
62 5. Ask: "I have two slots available, {{available_slots}}. Would you be able to make one of those times work?"
63 6. <wait for user response>
64 7. Set the {{selectedSlot}} variable to the user's response.
65 8. If {{selectedSlot}} is one of the available slots (positive response): 
66    - Trigger the 'bookSlot' tool with the {{selectedSlot}}.
67    - <wait for 'bookSlot' tool result>
68    - Inform the user of the result of the 'bookSlot' tool.
69    - Proceed to the 'Call Closing' section.
70 9. If {{selectedSlot}} is not one of the available slots (negative response):
71    - Proceed to the 'Suggest Alternate Slot' section.
72 
73 [Suggest Alternate Slot]
74 1. Ask: "If none of these slots work for you, could you please suggest a different time that suits you?"
75 2. <wait for user response>
76 3. Set the {{selectedSlot}} variable to the user's response.
77 4. Trigger the 'bookSlot' tool with the {{selectedSlot}}.
78 5. <wait for 'bookSlot' tool result>
79 6. If the {{selectedSlot}} is available:
80    - Inform the user of the result.
81 7. If the {{selectedSlot}} is not available:
82    - Trigger the 'fetchSlots' tool, provide the user {{selectedSlot}} as input and map the result to {{available_slots}}.
83    - Say: "That time is unavailable but here are some other times we can do {{available_slots}}."
84    - Ask: "Do either of those times work?"
85    - <wait for user response>
86    - If the user agrees to one of the new suggested slots:
87         - Set the {{selectedSlot}} variable to the user's response.
88         - Trigger the 'bookSlot' tool with the {{selectedSlot}}.
89         - <wait for 'bookSlot' tool result>
90         - Inform the user of the result.
91    - If the user rejects the new suggestions:
92         - Proceed to the 'Last Message' section.
93 
94 [Last Message]
95  - Respond: "Looks like this is taking longer than expected. Let me have one of our appointment specialists get back to you to make this process simple and easy."
96 - Proceed to the 'Call Closing' section.
97 
98 [Call Closing]
99 - Trigger the endCall Function.

Additional resources

Check out these additional resources to learn more about prompt engineering: