OpenAI Realtime

Overview

OpenAI’s Realtime API enables developers to use a native speech-to-speech model. Unlike other Vapi configurations which orchestrate a transcriber, model and voice API to simulate speech-to-speech, OpenAI’s Realtime API natively processes audio in and audio out.

In this guide, you’ll learn to:

• Choose the right realtime model for your use case
• Configure voice assistants with realtime capabilities
• Implement best practices for production deployments
• Optimize prompts specifically for realtime models

Available models

OpenAI offers three realtime models, each with different capabilities and cost/performance trade-offs:

Voice options

Realtime models support a specific set of OpenAI voices optimized for speech-to-speech:

Configuration

Basic setup

Configure a realtime assistant with function calling:

1
{
2
  "model": {
3
    "provider": "openai",
4
    "model": "gpt-realtime-2025-08-28",
5
    "messages": [
6
      {
7
        "role": "system",
8
        "content": "You are a helpful assistant. Be concise and friendly."
9
      }
10
    ],
11
    "temperature": 0.7,
12
    "maxTokens": 250,
13
    "tools": [
14
      {
15
        "type": "function",
16
        "function": {
17
          "name": "getWeather",
18
          "description": "Get the current weather",
19
          "parameters": {
20
            "type": "object",
21
            "properties": {
22
              "location": {
23
                "type": "string",
24
                "description": "The city name"
25
              }
26
            },
27
            "required": ["location"]
28
          }
29
        }
30
      }
31
    ]
32
  },
33
  "voice": {
34
    "provider": "openai",
35
    "voiceId": "alloy"
36
  }
37
}

Using realtime-exclusive voices

To use the enhanced voices only available with realtime models:

1
{
2
  "voice": {
3
    "provider": "openai",
4
    "voiceId": "marin"  // or "cedar"
5
  }
6
}

Handling instructions

The system message in your model configuration is automatically optimized for realtime processing:

1. System messages are converted to session instructions
2. Instructions are sent during WebSocket session initialization
3. The instructions field supports the same prompting strategies as system messages

Prompting best practices

General tips

• Iterate relentlessly: Small wording changes can significantly impact behavior
• Use bullet points over paragraphs: Clear, short bullets outperform long text blocks
• Guide with examples: The model closely follows sample phrases you provide
• Be precise: Ambiguity or conflicting instructions degrade performance
• Control language: Pin output to a target language to prevent unwanted switching
• Reduce repetition: Add variety rules to avoid robotic phrasing
• Capitalize for emphasis: Use CAPS for key rules to make them stand out

Prompt structure

Organize your prompts with clear sections for better model comprehension:

# Role & Objective
You are a customer service agent for Acme Corp. Your goal is to resolve issues quickly.
# Personality & Tone  
- Friendly, professional, and empathetic
- Speak naturally at a moderate pace
- Keep responses to 2-3 sentences
# Instructions
- Greet callers warmly
- Ask clarifying questions before offering solutions
- Always confirm understanding before proceeding
# Tools
Use the available tools to look up account information and process requests.
# Safety
If a caller becomes aggressive or requests something outside your scope, 
politely offer to transfer them to a specialist.

Realtime-specific techniques

## Pacing
- Deliver responses at a natural, conversational speed
- Do not rush through information
- Pause briefly between key points

Migration guide

Transitioning from standard STT/TTS to realtime models:

1
{
2
  "model": {
3
    "provider": "openai",
4
    "model": "gpt-realtime-2025-08-28"  // Changed from gpt-4
5
  }
6
}

Best practices

Model selection strategy

Performance optimization

• Temperature settings: Use 0.5-0.7 for consistent yet natural responses
• Max tokens: Set appropriate limits (200-300) for conversational responses
• Voice selection: Test different voices to match your brand personality
• Function design: Keep function schemas simple for faster execution

Error handling

Handle edge cases gracefully:

1
{
2
  "messages": [{
3
    "role": "system", 
4
    "content": "If you don't understand the user, politely ask them to repeat. Never make assumptions about unclear requests."
5
  }]
6
}

Current limitations

• Knowledge Bases are not currently supported with the Realtime API
• Endpointing and Interruption models are managed by Vapi’s orchestration layer
• Custom voice cloning is not available for realtime models
• Some OpenAI voices (ash, ballad, coral, fable, onyx, nova) are incompatible
• Transcripts may have slight differences from traditional STT output

Additional resources

• OpenAI Realtime Documentation
• Realtime Prompting Guide
• Prompting Cookbook
• Vapi Discord Community

Next steps

Now that you understand OpenAI Realtime models:

• Phone Calling Guide: Set up inbound and outbound calling
• Assistant Hooks: Add custom logic to your conversations
• Voice Providers: Explore other voice options