The Problem with Raw LLM Outputs

Large Language Models (LLMs) like those from OpenAI are incredibly powerful tools for generating text. However, getting them to produce data in a structured, predictable format has always been a significant challenge. Developers often resort to complex prompt engineering, hoping the LLM will adhere to a specific JSON schema or format. Even then, the output is frequently imperfect: missing keys, incorrect data types, or malformed JSON that requires manual parsing and error handling. This manual process is not only time-consuming but also brittle. A slight change in the LLM's response can break your entire parsing logic, leading to runtime errors and unreliable applications.

Consider the typical workflow: you send a prompt to an LLM, asking it to return information about a product, such as its name, price, and available colors, in JSON format. The LLM might return something like:

{
  "product_name": "SuperWidget",
  "price": "$99.99",
  "colors": ["red", "blue", "green"
}

This output, while close, has issues. The price is a string with a currency symbol, not a number. The `colors` array is missing a closing bracket. A naive parser would likely crash or produce incorrect data. Developers would then write extensive code to clean this up, often using `try-except` blocks and string manipulation, which is far from ideal.

Illustrative example of a malformed JSON output from an LLM

Introducing Pydantic for LLM Output Validation

Pydantic is a Python library for data validation and settings management using Python type annotations. It allows you to define data models with type hints, and Pydantic automatically parses, validates, and serializes your data accordingly. The real power of Pydantic emerges when you combine it with LLM APIs. Instead of just asking the LLM to return JSON, you can instruct it to return data that conforms to a Pydantic model. The LLM's response is then passed directly to the Pydantic model for validation.

Here’s how the process typically works:

  1. Define a Pydantic Model: You create a Python class that inherits from pydantic.BaseModel. Inside this class, you define your desired data structure using standard Python type hints (e.g., str, int, float, list[str], datetime).
  2. Construct the Prompt: You craft a prompt for the LLM that includes instructions to return data in a JSON format that matches your Pydantic model's schema. You can even include the JSON schema itself within the prompt.
  3. Call the LLM: Send the prompt to the LLM API (e.g., OpenAI's Chat Completions API).
  4. Validate with Pydantic: Take the LLM's JSON output and pass it directly to your Pydantic model for instantiation. Pydantic will automatically:
    • Parse the JSON string.
    • Validate that all fields are present and match the expected types.
    • Coerce types where possible (e.g., converting a string price like "$99.99" to a float 99.99 if your model specifies a float).
    • Raise a ValidationError if the data does not conform to the model.

This approach shifts the burden of data integrity from brittle, manual parsing code to Pydantic's robust validation system. It's akin to having a strict but fair gatekeeper for your LLM's output, ensuring that only correctly formatted data gets through.

Implementing Pydantic with OpenAI Completions

Let's illustrate with a practical example using OpenAI's API. Suppose you want to extract structured information about a book, including its title, author, publication year, and a list of genres.

First, define your Pydantic model:


from pydantic import BaseModel, Field
from typing import List

class Book(BaseModel):
    title: str
    author: str
    publication_year: int
    genres: List[str] = Field(default_factory=list)

Next, construct a prompt for the LLM. A common technique is to ask the model to output JSON and provide the schema. You can generate the JSON schema from your Pydantic model using model.schema() or model.model_json_schema().


from openai import OpenAI

client = OpenAI(api_key='YOUR_API_KEY')

# Generate JSON schema from Pydantic model
book_schema = Book.model_json_schema()

prompt = f"""
Extract the following information about the book from the text below and return it as a JSON object conforming to the following schema:\n\n{book_schema}\n\nText: 'The Hitchhiker's Guide to the Galaxy' is a comedic science fiction series by Douglas Adams. The first book was published in 1979 and is often categorized as humor and science fiction.
"""

response = client.chat.completions.create(
    model="gpt-4o", # Or gpt-3.5-turbo, etc.
    messages=[
        {"role": "system", "content": "You are a helpful assistant designed to output JSON."},
        {"role": "user", "content": prompt}
    ],
    temperature=0.0,
)

llm_output_json_string = response.choices[0].message.content

Finally, validate the LLM's output using your Pydantic model:


try:
    book_data = Book.parse_raw(llm_output_json_string)
    print("Successfully parsed book data:", book_data)
    print("Title:", book_data.title)
    print("Publication Year:", book_data.publication_year)
except ValidationError as e:
    print(f"Validation Error: {e}")
except Exception as e:
    print(f"An unexpected error occurred: {e}")

If the LLM output is valid according to the schema, Book.parse_raw() will return a Book object. If not, it will raise a ValidationError, providing detailed information about what went wrong. This is significantly cleaner and more robust than trying to parse and validate raw JSON strings manually.

Benefits and Use Cases

The combination of Pydantic and LLMs offers several key advantages:

  • Data Integrity: Ensures that the data you receive from the LLM is structured correctly and adheres to predefined types.
  • Reduced Boilerplate: Eliminates the need for manual JSON parsing and error-checking code, simplifying your application logic.
  • Improved Reliability: Makes your application more resilient to variations in LLM output.
  • Type Safety: Leverages Python's type hinting system for clearer, more maintainable code.
  • Enhanced Developer Experience: Provides clear error messages when validation fails, aiding debugging.

This pattern is invaluable for a wide range of applications:

  • Information Extraction: Extracting entities, facts, or summaries from unstructured text.
  • Content Generation: Generating structured content like product descriptions, user profiles, or API request bodies.
  • Chatbots and Virtual Assistants: Ensuring user intents and extracted parameters are correctly formatted for downstream processing.
  • Data Transformation: Converting free-form text into structured data for databases or analytics.

The Surprising Simplicity

What's genuinely surprising is how straightforward this integration is. Developers often assume that getting reliable structured data from LLMs requires intricate prompt tuning or complex post-processing. By leveraging Pydantic, you essentially delegate the task of data structure enforcement to a battle-tested library. The LLM's job becomes generating content that *looks* like the desired structure, and Pydantic's job is to rigorously verify it. This separation of concerns leads to remarkably clean and effective code. It’s less about coaxing the LLM into perfect behavior and more about defining your desired outcome and letting Pydantic guarantee it.

The future of interacting with LLMs for structured data likely involves this kind of declarative approach. Instead of imperative instructions for formatting, developers will define data models, and the LLM will be tasked with fulfilling those definitions. This paradigm shift promises more stable, scalable, and trustworthy AI-powered applications.