The Problem: Untrustworthy Documentation
Software documentation often relies on prose claims that are difficult to verify. Developers might write, "The reader handles the TTL cache-write breakdown correctly," but without a direct, verifiable link to the code that proves this, such statements are little more than assertions. This lack of direct evidence makes documentation brittle and untrustworthy. As projects grow, these claims can silently go stale, leading to bugs that are hard to track down.
This was the exact problem faced by the creator of agent-cost, a small open-source CLI tool designed to read local Claude Code / Codex CLI usage logs and estimate token costs. The developer aimed for documentation that was more robust than a standard README, specifically seeking a way to link every claim to an exact test or source line, validated against git history. This ensures that any pointer to supporting evidence cannot silently break over time.
To achieve this, a new tool, evidence-docs, was employed. This tool is designed to build documentation where every statement is backed by concrete, verifiable evidence directly from the codebase. The core idea is to move beyond narrative and towards a system where documentation is, in essence, a series of executable assertions about the software's behavior.
Enter Evidence-Docs: Linking Claims to Code
evidence-docs operates on a simple yet powerful principle: claims made in documentation must be directly tethered to specific, verifiable pieces of code. This isn't just about linking to a file; it's about linking to a specific line, a test case, or a commit that substantiates the claim. The tool parses documentation, identifies claims, and then seeks out corresponding evidence within the project's repository. This process ensures that documentation remains accurate and trustworthy, as any change to the underlying code that invalidates a claim will be immediately apparent.
The developer used evidence-docs to generate documentation for agent-cost. The process involved writing claims about how the CLI tool handles specific scenarios, such as cache management and log parsing. Each of these claims was then explicitly linked to the relevant code sections or unit tests responsible for implementing that functionality. This creates a tight feedback loop: if the code changes, and the documentation's evidence no longer matches, the documentation generation process will flag an error.
The Unexpected Discovery: Two Missing Regression Tests
The true power of this evidence-linking approach became apparent when evidence-docs, while attempting to verify the claims against the agent-cost codebase, uncovered a critical flaw. Despite the developer's belief that the cache-write breakdown was handled correctly, the tool's verification process failed. This failure wasn't due to a bug in evidence-docs itself, but rather a direct consequence of its meticulous cross-referencing.
Specifically, the tool identified two distinct scenarios where the expected behavior, as documented and assumed by the developer, did not align with the actual execution path of the code. These discrepancies pointed to the absence of crucial regression tests. The claims made in the documentation were, in effect, assertions that should have been covered by tests. When evidence-docs could not find supporting test code or code paths that validated these documented assertions, it signaled a gap in the project's test suite.
The first missing test related to the TTL cache-write breakdown. The documentation asserted correct handling, implying that writes to the cache with a Time-To-Live (TTL) were managed according to specification. However, the verification process revealed that there was no specific test case designed to catch potential edge cases or failure modes in this exact operation. The second missing test likely pertained to another critical aspect of log parsing or cost estimation, where a documented behavior was not rigorously validated by an automated test.
Implications for Software Development and Verification
This incident highlights a significant gap in traditional software development practices. While unit tests and integration tests are standard, they often focus on verifying functionality without explicitly linking back to documented claims. Documentation, on the other hand, is frequently treated as a separate artifact, prone to becoming outdated or inaccurate. evidence-docs bridges this gap by treating documentation not as a static description, but as a living, verifiable contract with the codebase.
The implications are far-reaching:
- Enhanced Trustworthiness: Documentation becomes a reliable source of truth, directly reflecting the code's current state.
- Proactive Bug Detection: Gaps in testing are exposed early in the development cycle, before they manifest as production bugs.
- Improved Maintainability: As code evolves, evidence-linked documentation acts as an immediate indicator of what needs updating or re-verification.
- Shift in Verification Paradigm: This approach encourages a more holistic view of software quality, integrating documentation, testing, and code into a single, verifiable system.
The developer's experience with agent-cost serves as a compelling case study. By using evidence-docs, they didn't just improve their documentation; they inadvertently performed a valuable audit of their test suite. This demonstrates that well-structured, evidence-linked documentation can function as a powerful meta-testing tool, revealing vulnerabilities that might otherwise remain hidden.
The Unanswered Question: Scalability and Adoption
What remains to be seen is how readily development teams will adopt this evidence-linking methodology. The initial setup and the discipline required to maintain these links might present a barrier, especially for larger, more complex projects or teams accustomed to traditional documentation workflows. The success of evidence-docs and similar tools will likely depend on their ability to integrate seamlessly into existing development pipelines and demonstrate clear, quantifiable benefits in terms of reduced bugs and increased confidence in software quality.
Furthermore, the tooling needs to mature. While evidence-docs has proven its concept, its ability to handle diverse programming languages, complex code structures, and various testing frameworks will be critical for widespread adoption. The effort involved in creating and maintaining these links must be demonstrably lower than the cost of the bugs they prevent.
In essence, the challenge is to shift the industry's perspective from documentation as an afterthought to documentation as an integral, verifiable component of the software development lifecycle. The discovery of two missing regression tests by evidence-docs is a strong signal that this shift is not only possible but potentially necessary for building more robust and reliable software.