Enforcing JSON Schema with Anyscale & Together

Get the LLM to adhere to your JSON schema using Anyscale & Together AI's newly introduced JSON modes

LLMs excel at generating creative text, but production applications demand structured outputs for seamless integration. Instructing LLMs to only generate the output in a specified syntax can help make their behaviour a bit more predictable. JSON is the format of choice here - it is versatile enough and is widely used as a standard data exchange format.

Several LLM providers offer features that help enforce JSON outputs:

• OpenAI has a feature called JSON mode that ensures that the output is a valid JSON object.

• While this is great, it doesn't guarantee adherence to your custom JSON schemas, but only that the output IS a JSON.

• Anyscale and Together AI go further - they not only enforce that the output is in JSON but also ensure that the output follows any given JSON schema.

Using Portkey, you can easily experiment with models from Anyscale & Together AI and explore the power of their JSON modes:

Node

import Portkey from 'portkey-ai';

const portkey = new Portkey({
    apiKey: "PORTKEY_API_KEY",
    // OR "TOGETHER_VIRTUAL_KEY"
})

async function main(){
  const json_response = await portkey.chat.completions.create({
    messages: [{role: "user",content: `Give me a recipe for making Ramen, in JSON format`}],
    model: "mistralai/Mistral-7B-Instruct-v0.1",
    response_format: {
      type: "json_object",
      schema: {
        type: "object",
        properties: {
          title: { type: "string" },
          description: { type: "string" },
          steps: { type: "array" }
        }
      }
    }
  });
}

console.log(json_response.choices[0].message.content);

main()

Python

from portkey_ai import Portkey
from pydantic import BaseModel, Field

portkey = Portkey(
    api_key="PORTKEY_API_KEY",
    virtual_key="ANYSCALE_VIRTUAL_KEY" # OR #TOGETHER_VIRTUAL_KEY
)

class Recipe(BaseModel):
    title: str
    description: str
    steps: List[str]
    
json_response = portkey.chat.completions.create(
    messages = [{ "role": 'user', "content": 'Give me a recipe for making Ramen, in JSON format' }],
    model = 'mistralai/Mistral-7B-Instruct-v0.1',
    response_format = {
        "type":"json_object",
        "schema": Recipe.schema_json()
    }
)

print(json_response.choices[0].message.content)

Output JSON:

{
  "title": "Fried Rice with Chicken and Vegetables",
  "description": "A delicious recipe for fried rice that includes chicken and a mix of colorful vegetables. Perfect for a healthy and satisfying meal. yum yum yum yum yum",
  "steps": [
    "1. Cook rice and set aside",
    "2. In a large skillet or wok, sauté sliced chicken in oil until browned",
    "3. ..."
  ]
}  

As you can see - it's pretty simple. Just define the JSON schema, and pass it at the time of making your request using the response_format param. The response_format's type is json_object and the schema contains all keys and their expected type.

Supporting Models

Model/Provider	Ensure JSON	Ensure Schema
mistralai/Mistral-7B-Instruct-v0.1 Anyscale	✅	✅
mistralai/Mixtral-8x7B-Instruct-v0.1 Anyscale	✅	✅
mistralai/Mixtral-8x7B-Instruct-v0.1 Together AI	✅	✅
mistralai/Mistral-7B-Instruct-v0.1 Together AI	✅	✅
togethercomputer/CodeLlama-34b-Instruct Together AI	✅	✅
gpt-4 and previous releases OpenAI / Azure OpenAI	✅	❌
gpt-3.5-turbo and previous releases OpenAI / Azure OpenAI	✅	❌
Ollama models	✅	❌

Creating Nested JSON Object Schema

Here's an example showing how you can also create nested JSON schema and get the LLM to enforce it:

class Ingredient(BaseModel):
    name: str
    quantity: str

class Recipe(BaseModel):
    title: str
    description: str
    ingredients: List[Ingredient]
    steps: List[str]
    
json_response = portkey.chat.completions.create(
    messages = [{ "role": 'user', "content": 'Give me a recipe for making Ramen, in JSON format' }],
    model="mistralai/Mistral-7B-Instruct-v0.1",
    response_format={
        "type": "json_object", 
        "schema": Recipe.schema_json()
    }
)

Add your Anyscale or Together AI virtual keys to Portkey vault, and get started!