Ollama

Portkey provides a robust and secure gateway to facilitate the integration of various Large Language Models (LLMs) into your applications, including your locally hosted models through Ollama.

Provider Slug: ollama

Portkey SDK Integration with Ollama Models

Portkey provides a consistent API to interact with models from various providers. To integrate Ollama with Portkey:

1. Install the Portkey SDK

Install the Portkey SDK in your application to interact with your Ollama API through Portkey.

NodeJS

npm install --save portkey-ai

Python

pip install portkey-ai

2. Initialize Portkey with Ollama URL

Instantiate the Portkey client by adding your Ollama publicly-exposed URL to the customHost property.

NodeJS SDK

import Portkey from 'portkey-ai'
 
const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY", // defaults to process.env["PORTKEY_API_KEY"]
    provider: "ollama",
    customHost: "https://7cc4-3-235-157-146.ngrok-free.app" // Your Ollama ngrok URL
})

Python SDK

from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",  # Replace with your Portkey API key
    provider="ollama",
    custom_host="https://7cc4-3-235-157-146.ngrok-free.app" # Your Ollama ngrok URL    
)

For the Ollama integration, you only need to pass the base URL to customHost without the version identifier (such as /v1) - Portkey takes care of it for Ollama.

Requests made to your localhost Ollama endpoints will fail. To integrate with Portkey and to observe your requests, you'll need to expose your localhost URLs publicly via a service like ngrok.

3. Invoke Chat Completions with Ollama

Use the Portkey SDK to invoke chat completions from your Ollama model, just as you would with any other provider.

NodeJS SDK

const chatCompletion = await portkey.chat.completions.create({
    messages: [{ role: 'user', content: 'Say this is a test' }],
    model: 'llama3',
});

console.log(chatCompletion.choices);

Python SDK

completion = portkey.chat.completions.create(
    messages= [{ "role": 'user', "content": 'Say this is a test' }],
    model= 'llama3'
)

print(completion)

Next Steps

Explore the complete list of features supported in the SDK:

SDK

---

You'll find more information in the relevant sections:

1. Add metadata to your requests
2. Add gateway configs to your Ollama requests
3. Tracing Ollama requests
4. Setup a fallback from OpenAI to Ollama APIs