The Probabilistic Problem with AI Outputs
Large language models (LLMs) are inherently probabilistic. This means that even with the most meticulously crafted prompts, edge cases will inevitably arise. These can manifest as hallucinations, omissions, deviations in output format, or confidently stated rationalizations that ultimately prove incorrect. This unreliability is a significant hurdle for developers integrating AI into critical workflows.
The author experienced this firsthand while using Claude Code daily. The AI would report tasks as completed, such as writing a file or updating logs, only for the developer to find that the file was never created or the log timestamps were outdated. The AI wasn't intentionally misleading; its output was a direct consequence of its probabilistic nature, leading to inherent instability.
Relying solely on prompt engineering to overcome this is akin to fighting probability with more probability. For true reliability, a deterministic, mechanical approach is necessary. This means moving beyond what the AI *says* it did and verifying what it *actually* did through an independent, verifiable process.
Building a Deterministic Feedback Loop
To address this, the author developed an agent configuration system featuring a closed-loop feedback mechanism. The core idea is to move away from trusting the AI's self-reporting and instead implement a system that verifies AI actions through objective, measurable criteria. The system is designed such that the vast majority of its components are simple scripts, with AI involvement limited to a single, focused task within the loop.
The process begins with the AI's current self-cognition, stored in a `self-model.md` file. This is fed into the session execution phase. After the AI performs an action, such as writing a file, the system then executes a series of verification scripts. These scripts are the deterministic backbone of the solution. They check for concrete, objective outcomes rather than relying on the AI's textual assertions.
A key element of this system is the use of file timestamps. When an AI is tasked with creating or modifying a file, the system doesn't just trust the AI's confirmation. Instead, it checks the actual file's last modified timestamp. If the AI claims it updated a log file, the script verifies the timestamp of that log file. If the AI states it created a new configuration file, the script checks for the existence and modification time of that file.
This approach transforms the AI's output from a probabilistic claim into a verifiable event. The feedback loop works by comparing the AI's intended action (derived from its prompts and self-cognition) with the actual, measurable outcome. If the outcome does not match the expectation—for instance, if a file timestamp indicates no modification when one was expected—this discrepancy is fed back into the system. This feedback allows for corrective actions, such as re-prompting the AI, adjusting parameters, or flagging the output for human review.
The Four Scripts and One AI
The elegance of this solution lies in its reliance on automation. Out of the five core steps in the process, four are handled by deterministic scripts. This significantly reduces the potential for errors introduced by the AI itself during the verification phase.
The scripted steps typically involve:
- Action Execution: The AI performs its task (e.g., writing code, generating text, modifying a configuration).
- File System Check: A script verifies the existence and, critically, the last modified timestamp of the relevant file(s). This is the primary deterministic check.
- Timestamp Comparison: The script compares the actual file timestamp against an expected timestamp or a threshold. For instance, if a file should have been created or modified within the last minute, the script checks if the timestamp falls within that window.
- Result Logging: The outcome of the verification (success or failure) is logged systematically. This log itself can be a timestamped artifact, creating a chain of verifiable events.
The single step that requires AI involvement is the initial task generation or refinement. The AI is used for its strengths—understanding natural language, generating creative content, or performing complex reasoning—but its output is then rigorously checked by the automated scripts. This separation of concerns is crucial: AI for generation, scripts for verification.
Handling Discrepancies and Iteration
When a script detects a discrepancy—for example, if a file timestamp is older than expected, or a file that should exist does not—this information is fed back into the AI. This isn't just a simple error message; it's contextual feedback. The AI receives data indicating *what* went wrong and *why* it's considered wrong, based on the deterministic check.
This feedback can trigger several responses:
- Re-prompting: The original prompt can be augmented with the failure information, asking the AI to retry the task with corrected understanding.
- Parameter Adjustment: If the issue is related to output format or detail level, the system might adjust parameters for the next attempt.
- Human Review Flagging: For critical failures or persistent issues, the system can automatically flag the output and the context for a human operator to review.
This iterative process, driven by objective data, allows the system to converge on the desired output quality. It transforms the AI from an unreliable oracle into a component within a more robust, verifiable system. The file timestamp acts as an objective arbiter, a silent witness to the AI's actions, ensuring that claims of completion or modification are backed by tangible evidence.
Broader Implications for AI Development
The technique of using deterministic checks, particularly file timestamps, to validate AI outputs has significant implications. It moves beyond the current paradigm of prompt engineering and fine-tuning, offering a complementary layer of control. This approach is particularly valuable for AI agents designed to perform autonomous tasks in file systems, code repositories, or any environment where concrete, state-changing actions are expected.
Consider the development of AI agents that manage cloud infrastructure, deploy code, or process data pipelines. In such scenarios, trusting the AI's word that a deployment succeeded or a file was updated is a recipe for disaster. A system like this provides a crucial safety net. It's the equivalent of a human developer not just running a script but also checking the output directory and verifying file modification dates before declaring a task complete.
The surprising detail here is not the complexity of the AI's task, but the simplicity of the verification mechanism. By focusing on readily available metadata like file timestamps, a robust feedback loop can be constructed with minimal AI overhead. This makes the solution scalable and applicable to a wide range of AI-driven automation scenarios where reliability is paramount.
What remains to be seen is how widely this pattern will be adopted. As AI agents become more autonomous, the need for such deterministic validation will only grow. Developers building complex AI systems must consider this fundamental challenge: how to bridge the gap between probabilistic generation and deterministic execution.