The Unreliable Output of Local AI Agents

Running sophisticated AI models locally offers significant advantages: enhanced privacy, reduced latency, and freedom from API costs. However, this autonomy comes with a critical challenge: reliability, particularly when AI agents need to produce structured data like JSON. Unlike large, hosted services such as OpenAI, which handle output validation server-side, local models often falter. Users frequently report instances where an agent asked to output JSON might forget a closing brace, invent a non-existent tool name, or append extraneous conversational text after the intended data structure. This unreliability makes local agents prone to breaking downstream processing pipelines.

The Eris project, a local agent built in Rust that leverages Markdown notes as its memory and operates entirely on the user's machine, faced this exact problem. To address it, Eris's developer implemented a grammar-based approach to constrain the model's output at the token level. This method ensures that the AI can only generate sequences of tokens that conform to a predefined structure, effectively preventing malformed JSON.

Constraining Tokens with GBNF Grammars

The core of this solution lies in the use of Generalized Bounded Natural Language Generation (GBNF) grammars. llama.cpp, a popular framework for running large language models locally, supports GBNF grammars. These grammars act as a set of rules that dictate which tokens a model is allowed to produce at any given point in its generation process. Instead of allowing the model to freely predict any next token, GBNF restricts its choices to a valid subset, ensuring adherence to a specified format.

The implementation involves a compiler that translates the schema definitions for each tool an AI agent might use into these GBNF grammar rules. This means that when the agent needs to output information, the grammar guides the model token by token. For example, if a JSON object requires a specific key-value pair, the grammar will only permit tokens that form valid keys and values according to the schema. This process continues for every token, making it literally impossible for the model to produce syntactically incorrect JSON. The model's output is constrained at every step, guaranteeing well-formed data structures.

Schema definition translated into GBNF grammar rules for AI output constraints

Dynamic Grammar for Contextual Relevance

Beyond ensuring basic JSON validity, the Eris project also employs a strategy to manage the complexity of tool schemas. AI agents often have access to a wide array of tools, each with its own schema. Presenting all available tools and their schemas at once can overwhelm a smaller local model, leading to confusion and degraded performance. To combat this, the Eris system dynamically narrows the grammar used for each turn of interaction.

Instead of exposing all 50 potential tools, the system intelligently identifies the 3 to 5 tools that are most relevant to the current user query or the ongoing conversation context. The GBNF grammar is then updated to only include rules for these relevant tools. This significantly reduces the search space for the model, allowing it to focus its generative capabilities on the most probable and useful actions. This dynamic adjustment not only improves the accuracy of tool selection but also further enhances the reliability of the structured output by limiting the model's exposure to irrelevant options.

The Eris Project: Local Agents in Action

The Eris project serves as a practical demonstration of this grammar-based approach. It's designed to run entirely on a user's machine, making it a prime candidate for exploring the challenges and solutions of local AI agent deployment. By integrating GBNF grammars, Eris aims to provide a robust and dependable experience for users who want to leverage AI for tasks requiring structured data output without relying on external cloud services.

The post detailing this implementation offers a deep dive into the technical specifics, including actual code examples from the Eris project. It showcases how a developer can take a tool's schema, compile it into GBNF rules, and integrate it into a local AI agent framework like llama.cpp. This level of detail is crucial for developers looking to replicate or adapt this solution for their own local AI applications.

Broader Implications for Local AI Development

The success of this grammar-based approach has significant implications for the future of local AI agents. It addresses a fundamental limitation that has hindered their practical adoption for tasks requiring precise, structured outputs. By enforcing output validity at the token level, developers can build more reliable AI assistants, automated workflows, and data processing tools that run entirely on user hardware.

This technique is not limited to JSON. Any structured data format that can be defined by a grammar—like XML, YAML, or even custom domain-specific languages—can benefit from this method. The ability to constrain AI output opens up new possibilities for integrating local AI agents into existing software ecosystems where data integrity and predictable formats are paramount. As local models continue to improve in capability and efficiency, solutions like Eris's grammar-based validation will be essential for unlocking their full potential.