Guardrails

Ship to production more confidently with Portkey Guardrails on your requests & responses

This feature is in beta and available to select users. To enable it for your org, ping us on the Portkey Discord.

LLMs are brittle - not just in API uptimes or their inexplicable 400/500 errors, but also in their core behavior. You can get a response with a 200 status code that completely errors out for your app's pipeline due to mismatched output. With Portkey's Guardrails, we now help you enforce LLM behavior in real-time with our Guardrails on the Gateway pattern.

Using Portkey's Guardrail platform, you can now verify your LLM inputs AND outputs to be adhering to your specifed checks; and since Guardrails are built on top of our Gateway, you can orchestrate your request exactly the way you want - with actions ranging from denying the request, logging the guardrail result, creating an evals dataset, falling back to another LLM or prompt, retrying the request, and more.

Portkey Guardrails

Examples of Guardrails Portkey offers:

• Regex match - Check if the request or response text matches a regex pattern

• JSON Schema - Check if the response JSON matches a JSON schema

• Contains Code - Checks if the content contains code of format SQL, Python, TypeScript, etc.

• Custom guardrail - If you are running a custom guardrail currently, you can also integrate it with Portkey

• ...and many more.

Portkey currently offers 20+ deterministic guardrails like the ones described above as well as LLM-based guardrails like Detect Gibberish, Scan for prompt injection, and more. These guardrails serve as protective barriers that help mitigate risks associated with Gen AI, ensuring its responsible and ethical deployment within organizations.

Browse the full list of supported Guardrail checks here.

Portkey also integrates with your favourite Guardrail platforms like Aporia, SydeLabs, Pillar Security and more. Just add their API keys to Portkey and you can enable their guardrails policies on your Portkey calls! More details on Guardrail Partners here.

---

Using Guardrails

Putting Portkey Guardrails in production is just a 4-step process:

1. Create Guardrail Checks

2. Create Guardrail Actions

3. Enable Guardrail through Configs

4. Attach the Config to a Request

This flowchart shows how Portkey processes a Guardrails request:

Let's see in detail below:

---

1. Create a New Guardrail & Add Checks

On the "Guardrails" page, click on Create and add your preferred Guardrail checks from the right sidebar.

On Portkey, you can configure Guardrails to be run on either the INPUT (i.e. PROMPT) or the OUTPUT. Hence, for the Guardrail you create, make sure your Guardrail is only validating ONLY ONE OF the Input or the Output.

Each Guardrail Check has a custom input field based on its usecase — just add the relevant details to the form and save your check.

• A check is an individual function that assesses the input prompt or output response against predefined conditions.

• You can add as many checks as you want to a single Guardrail.

• A check ONLY returns a boolean (Yes/No) verdict.

Here is a list of all the Guardrail checks available on Portkey and what they do.

---

2. Add Guardrail Actions

This is where you will define a basic orchestration logic for your Guardrail.

Guardrail is created to validate ONLY ONE OF the Input or the Output. The Actions set here will also apply only to either the request or the response.

There are 6 Types of Guardrail Actions

Action	State	Description	Impact
Async	TRUE This is the default state	Run the Guardrail checks asynchronously along with the LLM request.	Will add no latency to your request Useful when you only want to log guardrail checks without affecting the request
Async	FALSE	On Request Run the Guardrail check BEFORE sending the request to the LLM On Response Run the Guardrail check BEFORE sending the response to the user	Will add latency to the request Useful when your Guardrail critical and you want more orchestration over your request based on the Guardrail result
Deny	TRUE	On Request & Response If any of the Guardrail checks FAIL, the request will be killed with a 446 status code. If all of the Guardrail checks SUCCEED, the request/response will be sent further with a 200 status code.	This is useful when your Guardrails are critical and upon them failing, you can not run the request We would advice running this action on a subset of your requests to first see the impact
Deny	FALSE This is the default state	On Request & Response If any of the Guardrail checks FAIL, the request will STILL be sent, but with a 246 status code. If all of the Guardrail checks SUCCEED, the request/response will be sent further with a 200 status code.	This is useful when you want to log the Guardrail result but do not want it to affect your result
On Success	Send Feedback	If all of the Guardrail checks PASS, append your custom defined feedback to the request	We recommend setting up this action This will help you build an "Evals dataset" of Guardrail results on your requests over time
On Failure	Send Feedback	If any of the Guardrail checks FAIL, append your custom feedback to the request	We recommend setting up this action This will help you build an "Evals dataset" of Guardrail results on your requests over time

Set the relevant actions you want with your checks, name your Guardrail and save it! When you save the Guardrail, you will get an associated $Guardrail_ID that you can then add to your request.

---

3. "Enable" the Guardrails through Configs

This is where Portkey's magic comes into play. The Guardrail you created above is yet not an Active guardrail because it is not attached to any request.

Configs is one of Portkey's most powerful features and is used to define all kinds of request orchestration - everything from caching, retries, fallbacks, timeouts, to load balancing.

Now, you can use Configs to add Guardrail checks & actions to your request.

Add Guardrail ID before the request OR after the request

Type	Config Key	Value	Description
Before Request Hook	before_request_hooks	[{"id":"$guardrail_id"}]	This key is used to run Guardrail checks & actions on the INPUT.
After Request Hook	after_request_hooks	[{"id":"$guardrail_id"}]	This key is used to run Guardrail checks & actions on the OUTPUT.

Example Config with Guardrails

Simple Config

{
	"retry": {
		"attempts": 3
	},
	"cache": {
		"mode": "simple"
	},
	"virtual_key":"openai-xxx",
	"before_request_hooks": [{
		"id": "input-guardrail-id-xx"
	}],
	"after_request_hooks": [{
		"id": "output-guardrail-id-xx"
	}]
}

Guardrail Behaviour on the Gateway

For asynchronous guardrails (async=TRUE), Portkey returns the standard, default status codes from the LLM providers — this is because the Guardrails verdict is not affecting how you orchestrate your requests. Portkey will only log the Guardrail result for you.

But for synchronous requests (async=FALSE), Portkey can orchestrate your requests based on the Guardrail verdict. The behaviour is dependent on the following:

• Guardrail Check Verdict (PASS or FAIL) AND

• Guardrail Action — DENY Setting (TRUE or FALSE)

Portkey sends different request status codes corresponding to your set Guardrail behaviour.

For requests where async=FALSE

Guardrail Verdict	DENY Setting	Returned Status Code	Description
PASS	FALSE	200	Guardrails have passed, request will be processed regardless
PASS	TRUE	200	Guardrails have passed, request will be processed regardless
FAIL	FALSE	246	Guardrails have failed, but the request should still be processed. Portkey introduces a new Status code to indicate this state.
FAIL	TRUE	446	Guardrails have failed, and the request should not be processed. Portkey introduces a new Status code to indicate this state.

Example Config Using the New 246 & 446 Status Codes

Fallback to Another Model on Guardrail Fail

{
	"strategy": {
		"mode": "fallback",
		"on_status_codes": [246,446]
	},
	"targets": [
		{"virtual_key": "openai-key-xxx"},
		{"virtual_key": "anthropic-key-xxx"}
	],
	"before_request_hooks": [
		{"id": "guardrails-id-xxx"}
	]
}

Retry on Guardrail Fail

{
	"retry": {
		"on_status_codes": [246],
		"attempts": 5
	},
	"after_request_hooks": [
		{"id": "guardrails-id"}
	]
}

You can create these Configs in Portkey UI, save them, and get an associated Config ID you can attach to your requests. More here.

4. Final Step - Attach Config to Request

Now, while instantiating your Portkey client or while sending headers, just pass the Config ID.

NodsJS

const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY",
    config: "pc-***" // Supports a string config id or a config object
});

Python

const portkey = Portkey(
    api_key="PORTKEY_API_KEY",
    config="pc-***" # Supports a string config id or a config object
)

OpenAI NodeJS

const openai = new OpenAI({
  apiKey: 'OPENAI_API_KEY',
  baseURL: PORTKEY_GATEWAY_URL,
  defaultHeaders: createHeaders({
    apiKey: "PORTKEY_API_KEY",
    config: "CONFIG_ID"
  })
});

OpenAI Python

client = OpenAI(
    api_key="OPENAI_API_KEY", # defaults to os.environ.get("OPENAI_API_KEY")
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        provider="openai",
        api_key="PORTKEY_API_KEY", # defaults to os.environ.get("PORTKEY_API_KEY")
        config="CONFIG_ID"
    )
)

REST API

curl https://api.portkey.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "x-portkey-api-key: $PORTKEY_API_KEY" \
  -H "x-portkey-config: $CONFIG_ID" \ 
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [{
        "role": "user",
        "content": "Hello!"
      }]
  }'

For more, refer to the Config documentation here.

---

Viewing Guardrail Results in Portkey Logs

Portkey Logs will show you detailed information about Guardrail results for each request.

On the Feedback & Guardrails tab on the log drawer, you can see

Guardrail Details

• Overview: How many checks passed and how many failed

• Verdict: Guardrail verdict for each of the checks in your Guardrail

• Latency: Round trip time for each check in your Guardrail

Feedback Details

Portkey will also show the feedback object logged for each request

• Value: The numerical feedback value you passed

• Weight: The numerical feedback weight

• Metadata Key & Value: Any custom metadata sent with the feedback

• successfulChecks: Which checks associated with this request passed

• failedChecks: Which checks associated with this request failed

• erroredChecks: If there were any checks that errored out along the way

---

Defining Guardrails Directly in JSON

On Portkey, you can also create the Guardrails in code and add them to your Configs. Read more about this here:

Creating Raw Guardrails (in JSON)

---

Bring Your Own Guardrails

If you already have a custom guardrail pipeline where you send your inputs/outputs for evaluation, you can also integrate it with Portkey using a modular, custom webhook! Read more here:

Bring Your Own Guardrails

---

Examples of When to Deny Requests with Guardrails

1. Prompt Injection Checks: Preventing inputs that could alter the behavior of the AI model or manipulate its responses.

2. Moderation Checks: Ensuring responses do not contain offensive, harmful, or inappropriate content.

3. Compliance Checks: Verifying that inputs and outputs comply with regulatory requirements or organizational policies.

4. Security Checks: Blocking requests that contain potentially harmful content, such as SQL injection attempts or cross-site scripting (XSS) payloads.

By appropriately configuring Guardrail Actions, you can maintain the integrity and reliability of your AI app, ensuring that only safe and compliant requests are processed.

---