RAG on Legal Text Hallucinates Dates Due to Data Structure, Not Model

The Date Hallucination Problem in Legal RAG

Retrieval-Augmented Generation (RAG) systems, when applied to complex legal texts, frequently exhibit a peculiar hallucination: incorrect dates. This isn't a failing of the Large Language Model (LLM) itself, but rather a consequence of how the source data is structured and processed. Artem Sulyma, whose team at Platanor builds embedded security for IoT devices, encountered this issue repeatedly while creating an internal reference for regulations like the EU's CRA (Cyber Resilience Act), RED (Radio Equipment Directive), NIS2, and CSA (Cybersecurity Act).

The problem manifests clearly when asking an LLM about key dates within a regulation. For instance, querying ChatGPT about the CRA's effective dates yielded a confident but incorrect response. The model conflated the regulation's entry-into-force date (2024) with the date its main requirements become applicable (2027), a three-year discrepancy presented as fact. This isn't an isolated incident; Sulyma's team experienced similar date mix-ups when feeding raw regulation PDFs directly to models.

The root cause lies in the inherent nature of legal documents and standard text processing. Dates are often distributed across various articles or sections without explicit cross-referencing. When these documents are chunked for RAG systems, common methods like token-based splitting can sever sentences mid-thought, isolating date references from their contextual anchors. Furthermore, standard chunking provides no inherent mechanism for the LLM to discern the 'freshness' or applicability of a particular date within the document's lifecycle.

Why Standard Chunking Fails Legal Documents

Legal texts are not designed for simple linear processing. They are intricate webs of interconnected clauses, definitions, and effective dates. A single date might refer to the date of enactment, the start of a public consultation period, the deadline for compliance, or the date specific provisions come into effect. Without explicit semantic links or a structured understanding of these temporal relationships, an LLM relying on fragmented text chunks is essentially guessing.

Consider a regulation that states:

• Article 1: This Regulation shall enter into force on the twentieth day following that of its publication in the Official Journal of the European Union.
• Article 5: The provisions laid down in Chapters II, III, and IV shall apply from 1 January 2027.

A standard RAG system might process Article 1 and Article 5 as separate chunks, or worse, split Article 5 mid-sentence if it exceeds a token limit. When asked about when requirements apply, the model might retrieve fragments from both, and without a clear understanding that Article 5 defines the *application* date while Article 1 defines the *entry into force* date, it could easily synthesize a nonsensical or incorrect answer. The model lacks the contextual awareness to prioritize or correctly link these disparate temporal markers.

The problem is exacerbated by the lack of metadata. Raw PDFs often strip away structural information that humans intuitively use. The LLM sees a wall of text; it doesn't inherently understand that a date mentioned in a preliminary clause might be less critical for compliance than a date buried in an annex detailing enforcement timelines.

The File Structure Solution

Sulyma's team discovered that the solution wasn't in complex prompting techniques or fine-tuning the LLM, but in fundamentally restructuring the source data. When they rebuilt their internal reference as a public repository, they focused on organizing the information architecturally, not just textually.

This involved creating a deliberate file structure that mirrors the logical organization of the regulations. Instead of a single monolithic PDF, the content was broken down into smaller, semantically coherent files. For example:

• A file for the regulation's overall entry-into-force date.
• Separate files for different sets of applicable requirements, each clearly linked to its compliance deadline.
• Files dedicated to specific definitions that might include temporal elements.
• Sections detailing enforcement mechanisms with their associated timelines.

This approach treats the document's structure as a form of implicit metadata. When the RAG system retrieves information, it's more likely to fetch related pieces of information together. A query about compliance dates would naturally retrieve the file(s) specifically detailing those dates, alongside the relevant articles that define them. This provides the LLM with a more cohesive and contextually rich set of information, significantly reducing the likelihood of date hallucinations.