Bring Your Own Guardrails

Portkey supports bringing your own Guardrail using a custom webhook:

You can add Webhook as a Guardrail Check and setup any Guardrail Actions along with it. This is useful when you have an existing custom guardrail pipeline in your app and you are sending your LLM inputs or outputs to it for sync or async evaluation. Now, you can bring that onto Portkey's Gateway, make it production-grade, and enforce LLM behavior in real-time, easily.

How Custom Webhook Works

In the Guardrail check, Portkey expects your Webhook URL and any headers (including Authorization) you need to send.

Headers

Headers here should be a JSON object, like below:

{
    "Authorization":"MY_API_KEY",
    "Content-Type":"Application/JSON"
}

Portkey makes a POST request to your webhook URL and expects two objects in the response: verdict and data.

Webhook Response

Portkey expects two objects: verdict and data

Object	Type	Required?
verdict	boolean	Yes
data	any	No

Here's a sample webhook response:

{
  "verdict": true,
  "data": {
    "reason": "User passed all security checks",
    "score": 0.95,
    "additionalInfo": {
      "userStatus": "verified",
      "lastCheckTimestamp": "2024-08-21T14:30:00Z"
    }
  }
}

Check out the Webhook implementation here:

https://github.com/Portkey-AI/gateway/blob/main/plugins/default/webhook.ts

import {
  HookEventType,
  PluginContext,
  PluginHandler,
  PluginParameters,
} from '../types';
import { post, TimeoutError } from '../utils';

function parseHeaders(headers: unknown): Record<string, string> {
  try {
    if (typeof headers === 'object' && headers !== null) {
      return headers as Record<string, string>;
    }
    if (typeof headers === 'string') {
      try {
        const parsed = JSON.parse(headers as string);
        return parsed;
      } catch {
        throw new Error('Invalid headers format');
      }
    }
    return {};
  } catch (error: any) {
    throw error;
  }
}

export const handler: PluginHandler = async (
  context: PluginContext,
  parameters: PluginParameters,
  eventType: HookEventType
) => {
  let error = null;
  let verdict = false;
  let data: any = null;
  const transformedData: Record<string, any> = {
    request: {
      json: null,
      text: null,
    },
    response: {
      json: null,
      text: null,
    },
  };
  let transformed = false;

  try {
    const url = parameters.webhookURL;

    if (!url) {
      throw new Error('Missing webhook URL');
    }

    // Validate URL format
    try {
      new URL(url);
    } catch {
      throw new Error('Invalid webhook URL format');
    }

    let headers: Record<string, string>;
    try {
      headers = parseHeaders(parameters.headers);
    } catch (e: any) {
      throw new Error(`Failed to parse headers: ${e.message}`);
    }

    const requestBody = {
      ...context,
      // Setting headers to undefined to avoid passing sensitive information to the webhook endpoint.
      // This can later be controlled through parameters.
      request: { ...context.request, headers: undefined },
      eventType,
    };

    const response = await post(
      url,
      requestBody,
      { headers },
      parameters.timeout || 3000
    );
    verdict = response.verdict;

    if (
      response.transformedData?.request?.json &&
      eventType === 'beforeRequestHook'
    ) {
      transformedData.request.json = response.transformedData.request.json;
      transformed = true;
    }

    if (
      response.transformedData?.response?.json &&
      eventType === 'afterRequestHook'
    ) {
      transformedData.response.json = response.transformedData.response.json;
      transformed = true;
    }

    data = {
      verdict,
      explanation: verdict
        ? 'Webhook request succeeded'
        : 'Webhook request failed',
      webhookUrl: url,
      responseData: response.data,
      requestContext: {
        headers,
        timeout: parameters.timeout || 3000,
      },
    };
  } catch (e: any) {
    error = e;
    delete error.stack;

    const isTimeoutError = e instanceof TimeoutError;

    const responseData = !isTimeoutError && e.response?.body;
    const responseDataContentType = e.response?.headers?.get('content-type');

    data = {
      explanation: `Webhook error: ${e.message}`,
      webhookUrl: parameters.webhookURL || 'No URL provided',
      requestContext: {
        headers: parameters.headers || {},
        timeout: parameters.timeout || 3000,
      },
      // return response body if it's not a ok response and not a timeout error
      ...(responseData &&
        responseDataContentType === 'application/json' && {
          responseData: JSON.parse(responseData),
        }),
    };
  }

  return { error, verdict, data, transformedData, transformed };
};

Based on the verdict value, the Guardrail Check will PASS or FAIL, and will have subsequent impact on the Guardrail Actions you've set.

The webhook request automatically time out after 3 seconds - this can not be changed. So, if a webhook request times out, the Guardrail verdict will return PASS for that request.

Head over to the Portkey Discord community if you are building out custom webhooks and need any help!