Google Vertex AI

Portkey provides a robust and secure gateway to facilitate the integration of various Large Language Models (LLMs), and embedding models into your apps, including Google Vertex AI.

With Portkey, you can take advantage of features like fast AI gateway access, observability, prompt management, and more, all while ensuring the secure management of your Vertex auth through a virtual key system.

Provider Slug: vertex-ai

Portkey SDK Integration with Google Vertex AI

Portkey provides a consistent API to interact with models from various providers. To integrate Google Vertex AI with Portkey:

1. Install the Portkey SDK

Add the Portkey SDK to your application to interact with Google Vertex AI API through Portkey's gateway.

NodeJS

npm install --save portkey-ai

Python

pip install portkey-ai

2. Initialize Portkey with the Virtual Key

To integrate Vertex AI with Portkey, you'll need your Vertex Project Id & Vertex Region, with which you can set up the Virtual key.

Here's a guide on how to find your Vertex Project details.

If you are integrating through Service Account File, refer to this guide.

NodeJS SDK

import Portkey from 'portkey-ai'
 
const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY", // defaults to process.env["PORTKEY_API_KEY"]
    virtualKey: "VERTEX_VIRTUAL_KEY", // Your Vertex AI Virtual Key
})

Python SDK

from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",  # Replace with your Portkey API key
    virtual_key="VERTEX_VIRTUAL_KEY"   # Replace with your virtual key for Google
)

OpenAI Node SDK

import OpenAI from "openai";
import { PORTKEY_GATEWAY_URL, createHeaders } from "portkey-ai";

const portkey = new OpenAI({
  baseURL: PORTKEY_GATEWAY_URL,
  defaultHeaders: createHeaders({
    apiKey: "PORTKEY_API_KEY",
    virtualKey: "PORTKEY_VERTEX_VIRTUAL_KEY",
    authorization: "Bearer $GCLOUD AUTH PRINT-ACCESS-TOKEN"
  }),
});

If you do not want to add your Vertex AI details to Portkey vault, you can directly pass them while instantiating the Portkey client. More on that here.

3. Invoke Chat Completions with Vertex AI and Gemini

Use the Portkey instance to send requests to Gemini models hosted on Vertex AI. You can also override the virtual key directly in the API call if needed.

Vertex AI uses OAuth2 to authenticate its requests, so you need to send the access token additionally along with the request.

NodeJS SDK

const chatCompletion = await portkey.chat.completions.create({
    messages: [{ role: 'user', content: 'Say this is a test' }],
    model: 'gemini-pro',
}, {authorization: "vertex ai access token here"});

console.log(chatCompletion.choices);

Python SDK

completion = portkey.with_options(authorization="...").chat.completions.create(
    messages= [{ "role": 'user', "content": 'Say this is a test' }],
    model= 'gemini-pro'
)

print(completion)

OpenAI Node SDK

async function main() {
  const response = await portkey.chat.completions.create({
    messages: [{ role: "user", content: "1729" }],
    model: "gemini-1.5-flash-001",
    max_tokens: 128,
  });
  console.log(response.choices[0].message.content);
}

main();

Document, Video, Audio Processing

Vertex AI supports attaching mp4, pdf, jpg, mp3, wav, etc. file types to your Gemini messages.

Gemini Docs:

• Document Processing

• Video & Image Processing

• Audio Processing

Using Portkey, here's how you can send these media files:

NodeJS SDK

const chatCompletion = await portkey.chat.completions.create({
    messages: [
        { role: 'system', content: 'You are a helpful assistant' },
        { role: 'user', content: [
            {
                type: 'image_url',
                image_url: {
                    url: 'gs://cloud-samples-data/generative-ai/image/scones.jpg'
                }
            },
            {
                type: 'text',
                text: 'Describe the image'
            }
        ]}
    ],
    model: 'gemini-1.5-pro-001',
    max_tokens: 200
});

Python SDK

completion = portkey.chat.completions.create(
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant"
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "gs://cloud-samples-data/generative-ai/image/scones.jpg"
                    }
                },
                {
                    "type": "text",
                    "text": "Describe the image"
                }
            ]
        }
    ],
    model='gemini-1.5-pro-001',
    max_tokens=200
)

print(completion)

Curl

curl --location 'https://api.portkey.ai/v1/chat/completions' \
--header 'x-portkey-provider: vertex-ai' \
--header 'x-portkey-vertex-region: us-central1' \
--header 'Content-Type: application/json' \
--header 'x-portkey-api-key: PORTKEY_API_KEY' \
--header 'Authorization: GEMINI_API_KEY' \
--data '{
    "model": "gemini-1.5-pro-001",
    "max_tokens": 200,
    "stream": false,
    "messages": [
        {
            "role": "system",
            "content": "You are a helpful assistant"
        },
        {
            "role": "user",
            "content": [
                {    
                    "type": "image_url",
                    "image_url": {
                        "url": "gs://cloud-samples-data/generative-ai/image/scones.jpg"
                    }
                },
                {
                    "type": "text",
                    "text": "describe this image"
                }
            ]
        }
    ]
}'

This same message format also works for all other media types — just send your media file in the url field, like "url": "gs://cloud-samples-data/video/animals.mp4" for google cloud urls and "url":"https://download.samplelib.com/mp3/sample-3s.mp3" for public urls

Your URL should have the file extension, this is used for inferring MIME_TYPE which is a required parameter for prompting Gemini models with files

Sending base64 Image

Here, you can send the base64 image data along with the url field too:

"url": "data:image/png;base64,UklGRkacAABXRUJQVlA4IDqcAAC....."

---

Text Embedding Models

You can use any of Vertex AI's English and Multilingual models through Portkey, in the familar OpenAI-schema.

The Gemini-specific parameter task_type is also supported on Portkey.

NodeJS

import Portkey from 'portkey-ai';

const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY",
    virtualKey: "VERTEX_VIRTUAL_KEY"
});

// Generate embeddings
async function getEmbeddings() {
    const embeddings = await portkey.embeddings.create({
        input: "embed this",
        model: "text-multilingual-embedding-002",
        // @ts-ignore (if using typescript)
        task_type: "CLASSIFICATION", // Optional
    }, {authorization: "vertex ai access token here"});

    console.log(embeddings);
}
await getEmbeddings();

Python

from portkey_ai import Portkey

# Initialize the Portkey client
portkey = Portkey(
    api_key="PORTKEY_API_KEY",  # Replace with your Portkey API key
    virtual_key="VERTEX_VIRTUAL_KEY"
)

# Generate embeddings
def get_embeddings():
    embeddings = portkey.with_options(authorization="...").embeddings.create(
        input='The vector representation for this text',
        model='text-embedding-004',
        task_type="CLASSIFICATION" # Optional
    )
    print(embeddings)

get_embeddings()

REST API

curl 'https://api.portkey.ai/v1/embeddings' \
    -H 'Content-Type: application/json' \
    -H 'x-portkey-api-key: PORTKEY_API_KEY' \
    -H 'x-portkey-provider: vertex-ai' \
    -H 'Authorization: Bearer VERTEX_AI_ACCESS_TOKEN' \
    -H 'x-portkey-virtual-key: $VERTEX_VIRTUAL_KEY' \
    --data-raw '{
        "model": "textembedding-004",
        "input": "A HTTP 246 code is used to signify an AI response containing hallucinations or other inaccuracies",
        "task_type": "CLASSIFICATION"
    }'

Function Calling

Portkey supports function calling mode on Google's Gemini Models. Explore this ⬇️ Cookbook for a deep dive and examples:

Function Calling

Managing Vertex AI Prompts

You can manage all prompts to Google Gemini in the Prompt Library. All the models in the model garden are supported and you can easily start testing different prompts.

Once you're ready with your prompt, you can use the portkey.prompts.completions.create interface to use the prompt in your application.

Making Requests Without Virtual Keys

You can also pass your Vertex AI details & secrets directly without using the Virtual Keys in Portkey.

Vertex AI expects a region, a project ID and the access token in the request for a successful completion request. This is how you can specify these fields directly in your requests:

NodeJS SDK

import Portkey from 'portkey-ai'
 
const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY",
    vertexProjectId: "sample-55646",
    vertexRegion: "us-central1",
    provider:"vertex_ai",
    Authorization: "$GCLOUD AUTH PRINT-ACCESS-TOKEN"
})

const chatCompletion = await portkey.chat.completions.create({
    messages: [{ role: 'user', content: 'Say this is a test' }],
    model: 'gemini-pro',
});

console.log(chatCompletion.choices);

Python SDK

from portkey_ai import Portkey

portkey = Portkey(
    api_key="PORTKEY_API_KEY",
    vertex_project_id="sample-55646",
    vertex_region="us-central1",
    provider="vertex_ai",
    Authorization="$GCLOUD AUTH PRINT-ACCESS-TOKEN"
)

completion = portkey.chat.completions.create(
    messages= [{ "role": 'user', "content": 'Say this is a test' }],
    model= 'gemini-pro'
)

print(completion)

OpenAI Node SDK

import OpenAI from "openai";
import { PORTKEY_GATEWAY_URL, createHeaders } from "portkey-ai";

const portkey = new OpenAI({
  baseURL: PORTKEY_GATEWAY_URL,
  defaultHeaders: createHeaders({
    apiKey: "PORTKEY_API_KEY",
    provider: "vertex-ai",
    vertexRegion: "us-central1",
    vertexProjectId: "xxx"
    Authorization: "Bearer $GCLOUD AUTH PRINT-ACCESS-TOKEN",
    // forwardHeaders: ["authorization"] // You can also directly forward the auth token to Google
  }),
});

async function main() {
  const response = await portkey.chat.completions.create({
    messages: [{ role: "user", content: "1729" }],
    model: "gemini-1.5-flash-001",
    max_tokens: 32,
  });
  console.log(response.choices[0].message.content);
}

main();

REST

curl 'https://api.portkey.ai/v1/chat/completions' \
-H 'Content-Type: application/json' \
-H 'x-portkey-api-key: PORTKEY_API_KEY' \
-H 'x-portkey-provider: vertex-ai' \
-H 'Authorization: Bearer VERTEX_AI_ACCESS_TOKEN' \
-H 'x-portkey-vertex-project-id: sample-94994' \
-H 'x-portkey-vertex-region: us-central1' \
--data '{
    "messages": [{"role": "user","content": "Hello"}],
    "max_tokens": 20,
    "model": "gemini-pro"
}'

For further questions on custom Vertex AI deployments or fine-grained access tokens, reach out to us on [email protected]

---

How to Find Your Google Vertex Project Details

To obtain your Vertex Project ID and Region, navigate to Google Vertex Dashboard.

• You can copy the Project ID located at the top left corner of your screen.

• Find the Region dropdown on the same page to get your Vertex Region.

Get Your Vertex Service Account JSON

• Follow this process to get your Service Account JSON.

---

Next Steps

The complete list of features supported in the SDK are available on the link below.

SDK

You'll find more information in the relevant sections:

1. Add metadata to your requests

2. Add gateway configs to your Vertex AI requests

3. Tracing Vertex AI requests

4. Setup a fallback from OpenAI to Vertex AI APIs