The Generator-Verifier Split in Practice
Creating complex outputs, whether code, creative text, or structured data, often involves a tension between making the generator "smarter" and ensuring the output is actually correct. A Go CLI called verigen, designed to produce cryptarithmetic puzzles (alphametics), offers a compelling case study. Its approach is not to make the generator perfect, but to let it produce a vast quantity of imperfect solutions, with a separate verifier ensuring correctness. This strategy results in a staggering 99.7% rejection rate for generated puzzles, with only 0.26% passing verification. This isn't a sign of failure; it's the system working precisely as designed.
The core idea behind verigen is to decouple the generation process from the validation process. Instead of striving for a generator that only outputs valid puzzles, the system embraces a high failure rate. The goal is to generate five valid puzzles, a task that requires approximately 1,947 attempts and takes only 84 milliseconds. This speed is achieved precisely because the generator isn't burdened with the task of ensuring correctness. It simply produces candidates, and a dedicated verifier sifts through them.
This model has direct parallels to how developers interact with large language models (LLMs). When using an LLM for tasks like code generation or content creation, the temptation is to "make the LLM smarter" – to prompt it more effectively, fine-tune it, or use more complex internal reasoning. However, just like with verigen, the most efficient approach might be to accept that the LLM will produce many incorrect or suboptimal outputs. The real power then lies in a robust, independent verifier or validator that can quickly and reliably assess the LLM's output.
How Verigen Works: The Cryptarithmetic Case
verigen focuses on cryptarithmetic puzzles, a genre where each letter represents a distinct digit, and a given equation must hold true. The classic example, SEND + MORE = MONEY, famously resolves to 9567 + 1085 = 10652. For verigen to generate such puzzles, it doesn't attempt to solve them internally as it generates. Instead, it follows a process that prioritizes speed and volume of generation.
The generator's task is to produce potential puzzle structures and assignments of letters to digits. The verifier's task is to take these generated candidates and check if they meet the criteria of a valid cryptarithmetic puzzle: each letter maps to a unique digit, and the arithmetic equation is mathematically sound. The high rejection rate means that for every 1,000 puzzle candidates generated, only about 3 are deemed valid by the verifier.
This separation of concerns is crucial. If the generator were tasked with ensuring validity, it would need to perform complex constraint satisfaction checks for every single output. This would dramatically slow down the generation process. By offloading the correctness guarantee entirely to the verifier, the generator can operate at maximum speed, churning out possibilities. The 84 milliseconds to produce five valid puzzles is a testament to this efficient division of labor.
Lessons for LLMs and Beyond
The philosophy behind verigen offers a powerful paradigm shift for anyone working with generative AI, particularly LLMs. Developers often find themselves wrestling with LLM outputs, trying to coax them into producing perfect results on the first try. This can involve intricate prompt engineering, few-shot examples, and chain-of-thought prompting, all aimed at making the LLM itself "smarter" or more accurate.
However, the verigen model suggests an alternative. Instead of focusing solely on improving the generator (the LLM), invest heavily in building a robust, efficient, and accurate verifier. This verifier could be a set of unit tests for code generation, a grammar and style checker for text, a schema validator for structured data, or a specific rules engine for creative content.
Consider code generation: an LLM might produce a function that looks plausible but contains subtle bugs or doesn't adhere to specific API conventions. The "generator-smarter" approach would involve trying to get the LLM to write perfect code. The "verifier-first" approach would involve letting the LLM generate code rapidly and then using automated tests, linters, and static analysis tools to catch and correct errors. The 99.7% rejection rate from verigen is analogous to an LLM producing many code snippets that fail compilation or unit tests, but doing so so quickly that the overall process of getting to working code is accelerated.
What remains an open question is the optimal balance between generator sophistication and verifier complexity. For highly complex tasks where the LLM's internal reasoning is critical, a purely brute-force generation approach might be insufficient. Yet, for many applications, a simpler, faster generator paired with a powerful verifier could unlock significant performance gains and reduce development cycles. The verigen example highlights that "making the generator smarter" is not always the most effective optimization path; sometimes, it's about building a better judge.
The efficiency gained by verigen is substantial. If each of the 1,947 attempts took even a fraction of a second longer, the total time to produce five puzzles would increase dramatically. This speed allows for a much larger search space to be explored in a short amount of time, increasing the probability of finding valid solutions without requiring the generator itself to be an oracle of correctness. This principle is broadly applicable: embrace imperfection in generation and perfect the art of verification.
This approach is particularly relevant in scenarios where the cost of a single incorrect output is low, but the cost of producing correct outputs is high. By accepting a high rate of waste, the system prioritizes throughput and speed, making it practical for tasks where generating many options and filtering them is more efficient than trying to generate perfect options from the outset. The 0.26% acceptance rate is not a bug; it's a feature of a system designed for speed and scalability through ruthless verification.