The Problem with Textual Outputs in RAG

Retrieval Augmented Generation (RAG) systems promise to ground Large Language Models (LLMs) in factual data, reducing hallucinations. However, a critical flaw persists: most RAG pipelines return raw text from their retrieved documents to the LLM. This textual output acts as a weak contract. The LLM receives unstructured information and must then interpret it, a process ripe for misinterpretation and, consequently, hallucination. The LLM doesn't know *why* a piece of text was retrieved or what specific role it's intended to play in the final answer. It's like asking a brilliant student to answer a complex question based on a stack of unorganized notes; they might make connections that aren't there or miss crucial context.

This is where the concept of a 'typed answer contract' becomes essential. Instead of returning plain text, RAG systems should return structured data, where each piece of information is explicitly defined by its type and purpose. This structure transforms the LLM's input from a vague suggestion into a precise set of facts and constraints. The schema itself becomes the contract: every field represents a specific question the pipeline needs answered, and every returned value is a checkable piece of data.

Diagram illustrating a RAG pipeline with a typed answer contract enforcing structured LLM inputs.

Engineering Context with Typed Inputs

The practice of meticulously engineering the context provided to LLMs, including RAG, has gained prominence. Tobi Lütke and Andrej Karpathy, in 2025, highlighted the importance of this structured approach. For a single document, each retrieved 'brick' of information should emit typed pieces that converge into a single, well-defined LLM call. This isn't just about retrieving more data; it's about retrieving the *right kind* of data, presented in a way that the LLM can unequivocally understand and utilize.

Consider the components of a well-engineered context for RAG. Beyond the core retrieved document snippets, a robust system would include several typed inputs:

  • Corpus Data: This is the foundational knowledge base. Instead of returning raw paragraphs, it could return structured entities, relationships, or even pre-validated facts extracted from the corpus. For example, instead of just a sentence about a company's founding date, the system could return a structured object like { "entity": "Acme Corp", "attribute": "founded_date", "value": "1998-05-15" }.
  • Conversation History: For conversational agents, returning the entire chat log as plain text is inefficient and error-prone. Typed inputs can represent turns in the conversation, user intents, and system responses in a structured format, allowing the LLM to track context more effectively.
  • Tool Extensions/Function Calls: When RAG systems integrate with external tools or APIs, the parameters and expected outputs of these tools should be typed. This allows the LLM to know precisely what information a tool can provide or what parameters it requires, rather than inferring it from documentation strings.
  • User Query Breakdown: The original user query can also be broken down into typed components, such as identified entities, intents, and constraints. This explicit parsing helps the LLM focus on the core task and avoid misinterpreting nuanced user requests.

By structuring these inputs, we move away from the LLM guessing the meaning and intent behind the data. The 'typed answer contract' ensures that the LLM receives specific, verifiable pieces of information, significantly reducing the cognitive load and the potential for generating incorrect or fabricated content.

The Typed Answer Contract in Practice

Implementing a typed answer contract means that the RAG pipeline, before sending information to the LLM, transforms retrieved text into a predefined schema. This schema acts as a blueprint for the expected output. Each field in the schema is a clear question or data point the pipeline is seeking. For instance, if a user asks, "What is the net profit of Company X in Q3 2023?", a text-based RAG might return a paragraph containing that figure. A typed RAG, however, would aim to return a structured object:

{
  "company_name": "Company X",
  "quarter": "Q3 2023",
  "financial_metric": "net_profit",
  "value": 15000000,
  "currency": "USD",
  "source_document": "Q3_2023_Earnings_Report.pdf"
}

This structured output offers several advantages:

  • Verifiability: Each field is directly checkable. The system knows exactly which piece of data corresponds to which aspect of the user's query.
  • Reduced Hallucination: If a piece of information is missing in the retrieved documents (e.g., the currency wasn't specified), the corresponding field in the output can be explicitly marked as null or absent, rather than the LLM inventing a currency.
  • Downstream Processing: Structured data is far easier for subsequent processes to consume, whether that's another LLM, a database, or a user interface. Calculations, comparisons, and data validation become straightforward.
  • Schema Evolution: The schema itself can evolve. If new types of information need to be extracted, the schema can be updated, providing a clear roadmap for pipeline modifications.

The surprise here is not that structured data is better, but how profoundly it changes the LLM's role. Instead of a text interpreter, the LLM becomes a structured data synthesizer, working with a contract that guarantees the integrity of its inputs.

What Lies Ahead?

The shift from returning raw text to enforcing typed answer contracts is a significant evolution for RAG systems. It moves the reliability of LLM outputs from a probabilistic guess to a deterministic guarantee, at least concerning the grounding in retrieved data. This approach is crucial for enterprise applications where accuracy and trustworthiness are paramount. As these systems become more sophisticated, we can expect to see more advanced schema definitions, richer data types, and tighter integration between retrieval mechanisms and structured output enforcement. The ultimate goal is to build LLM applications that are not only intelligent but also inherently reliable, grounded in verifiable facts, and free from the scourge of hallucination.