Ollama

Portkey provides a robust and secure gateway to facilitate the integration of various Large Language Models (LLMs) into your applications, including your locally hosted models through Ollama.

Provider Slug: ollama

Portkey SDK Integration with Ollama Models

Portkey provides a consistent API to interact with models from various providers. To integrate Ollama with Portkey:

1. Install the Portkey SDK

Install the Portkey SDK in your application to interact with your Ollama API through Portkey.

npm install --save portkey-ai

pip install portkey-ai

2. Initialize Portkey with Ollama URL

Instantiate the Portkey client by adding your Ollama publicly-exposed URL to the customHost property.

import Portkey from 'portkey-ai'
 
const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY", // defaults to process.env["PORTKEY_API_KEY"]
    provider: "ollama",
    customHost: "https://7cc4-3-235-157-146.ngrok-free.app" // Your Ollama ngrok URL
})

from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",  # Replace with your Portkey API key
    provider="ollama",
    custom_host="https://7cc4-3-235-157-146.ngrok-free.app" # Your Ollama ngrok URL    
)

For the Ollama integration, you only need to pass the base URL to customHost without the version identifier (such as /v1) - Portkey takes care of it for Ollama.

Requests made to your localhost Ollama endpoints will fail. To integrate with Portkey and to observe your requests, you'll need to expose your localhost URLs publicly via a service like ngrok.

3. Invoke Chat Completions with Ollama

Use the Portkey SDK to invoke chat completions from your Ollama model, just as you would with any other provider.

const chatCompletion = await portkey.chat.completions.create({
    messages: [{ role: 'user', content: 'Say this is a test' }],
    model: 'llama3',
});

console.log(chatCompletion.choices);

completion = portkey.chat.completions.create(
    messages= [{ "role": 'user', "content": 'Say this is a test' }],
    model= 'llama3'
)

print(completion)

Next Steps

Explore the complete list of features supported in the SDK:

You'll find more information in the relevant sections:

PreviousNovita AI NextOpenRouter

Last updated 1 year ago

Was this helpful?

Portkey SDK Integration with Ollama Models

Portkey provides a consistent API to interact with models from various providers. To integrate Ollama with Portkey:

1. Install the Portkey SDK

Install the Portkey SDK in your application to interact with your Ollama API through Portkey.

npm install --save portkey-ai

pip install portkey-ai

2. Initialize Portkey with Ollama URL

Instantiate the Portkey client by adding your Ollama publicly-exposed URL to the customHost property.

import Portkey from 'portkey-ai'
 
const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY", // defaults to process.env["PORTKEY_API_KEY"]
    provider: "ollama",
    customHost: "https://7cc4-3-235-157-146.ngrok-free.app" // Your Ollama ngrok URL
})

from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",  # Replace with your Portkey API key
    provider="ollama",
    custom_host="https://7cc4-3-235-157-146.ngrok-free.app" # Your Ollama ngrok URL    
)

For the Ollama integration, you only need to pass the base URL to customHost without the version identifier (such as /v1) - Portkey takes care of it for Ollama.

Requests made to your localhost Ollama endpoints will fail. To integrate with Portkey and to observe your requests, you'll need to expose your localhost URLs publicly via a service like ngrok.

3. Invoke Chat Completions with Ollama

Use the Portkey SDK to invoke chat completions from your Ollama model, just as you would with any other provider.

const chatCompletion = await portkey.chat.completions.create({
    messages: [{ role: 'user', content: 'Say this is a test' }],
    model: 'llama3',
});

console.log(chatCompletion.choices);

completion = portkey.chat.completions.create(
    messages= [{ "role": 'user', "content": 'Say this is a test' }],
    model= 'llama3'
)

print(completion)