Custom LLM

Vapi supports using any OpenAI-compatible endpoint as the LLM. This includes services like OpenRouter, AnyScale, Together AI, or your own server.

​

Using an LLM provider

You’ll first want to POST your API key via the /credential endpoint:

{
  "provider": "openrouter",
  "apiKey": "<YOUR OPENROUTER KEY>"
}

Then, you can create an assistant with the model provider:

{
  "name": "My Assistant",
  "model": {
    "provider": "openrouter",
    "model": "cognitivecomputations/dolphin-mixtral-8x7b",
    "messages": [
      {
        "role": "system",
        "content": "You are an assistant."
      }
    ],
    "temperature": 0.7
  }
}

​

Using Fine-Tuned OpenAI Models

To set up your OpenAI Fine-Tuned model, you need to follow these steps:

1. Set the custom llm URL to https://api.openai.com/v1.
2. Assign the custom llm key to the OpenAI key.
3. Update the model to their model.
4. Execute a PATCH request to the /assistant endpoint and ensure that model.urlRequestMetadataEnabled is set to false.

​

Using your server

To set up your server to act as the LLM, you’ll need to create an endpoint that is compatible with the OpenAI Client. For best results, your endpoint should also support streaming completions.

If your server is making calls to an OpenAI compatble API, you can pipe the requests directly back in your response to Vapi.

If you’d like your OpenAI-compatible endpoint to be authenticated, you can POST your server’s API key and URL via the /credential endpoint:

{
  "provider": "custom-llm",
  "apiKey": "<YOUR SERVER API KEY>"
}

If your server isn’t authenticated, you can skip this step.

Then, you can create an assistant with the custom-llm model provider:

{
  "name": "My Assistant",
  "model": {
    "provider": "custom-llm",
    "url": "<YOUR OPENAI COMPATIBLE ENDPOINT BASE URL>",
    "model": "my-cool-model",
    "messages": [
      {
        "role": "system",
        "content": "You are an assistant."
      }
    ],
    "temperature": 0.7
  }
}