December '23 Migration

Date: 8th Dec, 2023

This December, we're pushing out some exciting new updates to Portkey's SDKs, APIs, and Configs.

are upped to major version 1.0 bringing parity with the new OpenAI SDK structure and adding Portkey production features to it. We are also bringing native Langchain & Llamaindex integrations inside the SDK. This is a Breaking Change that Requires Migration.

are upgraded with new endpoints, making it simpler to do /chat/completions and /completions calls and adding Portkey's production functionalities to them.

This is a Breaking Change that Requires Migration.

are upgraded to version 2.0, bringing nested gateway strategies with granular handling. For Configs saved in the Portkey dashboard, this is NOT a Breaking Change and we will Auto Migrate your old Configs. For Configs directly defined at the time of making a call, through the old SDKs or old APIS, they will fail on the new APIs & SDKs and require migration.

Compatibility & Deprecation List

List

Compatibility

Deprecation Date

API (Old) /v1/proxy /v1/complete /v1/chatComplete /v1/embed /v1/prompts/ID/generate

✅ SDK (Old) ❌ SDK (New) ✅ Configs (Old) ✅ Configs (New)

Q2 '24

API (New) /v1 /v1/completions /v1/chat/completions /v1/embeddings /v1/prompts/ID/completions

❌ SDK (Old) ✅ SDK (New) ❌ Configs (Old) ✅ Configs (New)

SDK Version < 1 (Old)

✅ API (Old) ❌ API (New) ✅ Configs (Old) ✅ Configs (new)

Q2 '24

SDK Version = 1 (New)

❌ API (Old) ✅ API (New) ✅ Configs (Old) ✅ Configs (new)

Configs 1.0 (Old)

✅ API (Old)

❌ API (New) ✅ SDK (Old) ❌ SDK (new) === Configs saved through the Portkey UI will be auto migrated.

Q2 '24

Configs 2.0 (New)

✅ API (Old)

✅ API (New) ✅ SDK (Old) ✅ SDK (new)

We recommend upgrading to these new versions promptly to take full advantage of their capabilities. While your existing code will continue to work until the deprecation date around Q2 '24, transitioning now ensures you stay ahead of the curve and avoid any future service interruptions. Follow along with this guide!

Major Version Release of the SDK

Here's What's New:

More extensible SDK that can be used with many more LLM providers
Out-of-the-box support for streaming
Completely follows OpenAI's SDK signature reducing your technical debt
Native support for Langchain & Llamaindex within the SDK (Python)
Support for the Portkey Feedback endpoint
Support for Portkey Prompt Templates
Older SDK versions to be deprecated soon

Here's What's Changed:

FROM ➡

import portkey
from portkey import Config, LLMOptions

portkey.config = Config(
    mode="single",
    llms=LLMOptions(provider="openai", api_key="OPENAI_API_KEY")
)

response = portkey.ChatCompletions.create(
    model="gpt-4", 
    messages=[
        {"role": "user","content": "Hello World!"}
    ]
)

TO ⬇

from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",
    Authorization="OPENAI_KEY"
)

response = portkey.chat.completions.create(
    messages=[{'role': 'user', 'content': 'Say this is a test'}],
    model='gpt-3.5-turbo'
)

print(response)

Installing the New SDK,

pip install -U portkey-ai

FROM ➡

import { Portkey } from "portkey-ai";

const portkey = new Portkey({
    mode: "single",
    llms: [{ provider: "openai", virtual_key: "open-ai-xxx" }]
});

async function main() {
    const chatCompletion = await portkey.chat.completions.create({
        messages: [{ role: 'user', content: 'Say this is a test' }],
        model: 'gpt-4'
    });

    console.log(chatCompletion.choices);
};

main();

TO ⬇

import Portkey from 'portkey-ai';

// Initialize the Portkey client
const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY",  // Replace with your Portkey API key
    Authorization: "OPENAI_KEY" 
});

// Generate a chat completion
async function getChatCompletion() {
    const chatCompletion = await portkey.chat.completions.create({
        messages: [{ role: 'user', content: 'Say this is a test' }],
        model: 'gpt-3.5-turbo',
    });

    console.log(chatCompletion);
}
getChatCompletion();

Installing the New SDK:

npm i -U portkey-ai

All-New APIs

Here's What's New:

Introduced 3 new routes /chat/completions, /completions, and /embeddings
Simplified the headers:
1. x-portkey-mode header is deprecated and replaced with x-portkey-provider
  1. Which takes values: openai, anyscale, cohere, palm, azure-openai, and more.
2. New header x-portkey-virtual-key is introduced.
/complete and /chatComplete endpoints to be deprecated soon
Prompts endpoint /prompts/$PROMPT_ID/generate is upgraded to /prompts/$PROMPT_ID/completions and the old route will be deprecated soon
1. We now support updating the model params on-the-fly (i.e. changing temperature etc at the time of making a call)
2. Prompt response object on the /completions route is now fully OpenAI compliant
New /gateway endpoint that lets you make calls to third-party LLM providers easily

Here's What's Changed

FROM ➡

from openai import OpenAI

client = OpenAI(
    api_key="OPENAI_API_KEY", # defaults to os.environ.get("OPENAI_API_KEY")
    base_url="https://api.portkey.ai/v1/proxy",
    default_headers= {
        "x-portkey-api-key": "PORTKEY_API_KEY",
        "x-portkey-mode": "proxy openai",
        "Content-Type": "application/json"
    }
)

chat_complete = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Say this is a test"}],
)

print(chat_complete.choices[0].message.content)

TO ⬇

# pip install -U portkey-ai 

from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders

client = OpenAI(
    api_key="OPENAI_API_KEY", # defaults to os.environ.get("OPENAI_API_KEY")
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        provider="openai",
        api_key="PORTKEY_API_KEY" # defaults to os.environ.get("PORTKEY_API_KEY")
    )
)

chat_complete = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Say this is a test"}],
)

print(chat_complete.choices[0].message.content)

FROM ➡

import OpenAI from 'openai';

const openai = new OpenAI({
  apiKey: 'OPENAI_API_KEY', // defaults to process.env["OPENAI_API_KEY"],
  baseURL: "https://api.portkey.ai/v1/proxy",
  defaultHeaders:{
    "x-portkey-api-key": "PORTKEY_API_KEY",
    "x-portkey-mode": "proxy openai",
    "Content-Type": "application/json"
  }
});

async function main() {
  const chatCompletion = await openai.chat.completions.create({
    messages: [{ role: 'user', content: 'Say this is a test' }],
    model: 'gpt-3.5-turbo',
  });

  console.log(chatCompletion.choices);
}

main();

TO ⬇

// npm i portkey-ai

import OpenAI from 'openai'; // We're using the v4 SDK
import { PORTKEY_GATEWAY_URL, createHeaders } from 'portkey-ai'

const openai = new OpenAI({
  apiKey: 'OPENAI_API_KEY', // defaults to process.env["OPENAI_API_KEY"],
  baseURL: PORTKEY_GATEWAY_URL,
  defaultHeaders: createHeaders({
    provider: "openai",
    apiKey: "PORTKEY_API_KEY" // defaults to process.env["PORTKEY_API_KEY"]
  })
});

async function main() {
  const chatCompletion = await openai.chat.completions.create({
    messages: [{ role: 'user', content: 'Say this is a test' }],
    model: 'gpt-3.5-turbo',
  });

  console.log(chatCompletion.choices);
}

main();

FROM ➡

curl http://api.portkey.ai/v1/proxy/completions \
  -H 'x-portkey-api-key: $PORTKEY_API_KEY' \
  -H 'x-portkey-mode: proxy openai' \
  -H 'Authorization: Bearer <OPENAI_API_KEY>' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-3.5-turbo-instruct",
    "prompt": "Top 20 tallest buildings in the world"
  }'

TO ⬇

curl https://api.portkey.ai/v1/completions \
  -H "x-portkey-api-key: $PORTKEY_API_KEY" \
  -H "x-portkey-provider: openai" \ 
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-3.5-turbo-instruct",
    "prompt": "Top 20 tallest buildings in the world"
  }'

Simlarly, for Prompts

FROM ➡

curl https://api.portkey.ai/v1/prompts/$PROMPT_ID/generate \
  -H 'x-portkey-api-key: $PORTKEY_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{"variables": {"variable_a": "", "variable_b": ""}}'

TO ⬇

curl https://api.portkey.ai/v1/prompts/$PROMPT_ID/completions \
  -H 'x-portkey-api-key: $PORTKEY_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{"variables": {"variable_a": "", "variable_b": ""}}'

# pip install portkey-ai

from portkey-ai import Portkey

client = Portkey(api_key="PORTKEY_API_KEY")

response = client.prompts.completions.create(
    prompt_id="Prompt_ID",
    variables={
       # The variables specified in the prompt
    },
    max_tokens=250,
    presence_penalty=0.2,
    temperature=0.1
)
print(prompt_completion)

Configs 2.0

Here's What's New

New concept of strategy instead of standalone mode. You can now build bespoke gateway strategies and nest them in a single config.
You can also trigger a specific strategy on specific error codes.
New concept of targets that replace options in the previous Config
If you are adding virtual_key to the target array, you no longer need to add provider,Portkey will pick up the Provider directly from the Virtual Key!
For Azure, only now pass the virtual_key - it takes care of all other Azure params like Deployment name, API version etc.

The Configs UI on Portkey app will autocomplete Configs ONLY in the new format now. All your existing Configs are auto migrated.

Here's What's Changed

FROM ➡

{
	"mode": "single",
	"options": [
		{
			"provider": "openai",
			"virtual_key": "open-4110dd",
		}
	]
}

TO ⬇

{
	"strategy": {
		"mode":"single"
	},
	"targets": [
		{
			"virtual_key": "open-4110dd"
		}
	]
}

Support

PreviousCommon Errors & Resolutions

Last updated 1 year ago

Was this helpful?