The Unseen API: Error Taxonomies as the Real Product Surface
Developers often treat APIs as collections of functions that return data. The reality is more complex: an API's error taxonomy is its true contract. This taxonomy, the set of specific errors and their meanings an API can return, is the actual product surface that downstream systems rely on. When this contract is not explicitly defined and rigorously tested, it becomes brittle. AI coding assistants and even human developers, in their pursuit of cleaner code or preferred error-handling styles (like favoring exceptions over returning None), frequently rewrite these error contracts without realizing the downstream impact.
Consider a typical dispatcher function in a complex system. It might handle various inputs and scenarios. For a bad payload, it might raise a ValueError. On a send failure, it might return None. Or, it could return a structured error like {"ok": False, "status": 429}. Downstream code, often written by other teams or even by developers years ago, will have branched logic to handle each of these distinct outcomes. It's not just checking if a value is None; it's checking for the presence of specific keys in a dictionary, the type of the returned object, or the specific exception raised.
When a model rewrite or a refactor collapses these distinct shapes—perhaps by deciding to always raise an exception instead of returning None, or by changing the structure of a returned error dictionary—production systems break. The defect isn't a missing type annotation or a simple logical flaw. It's a failure to recognize and preserve the unrecorded, implicit error taxonomy that the system has evolved to depend on.
Why Model Rewrites Miss the Contract
The current trend in AI-assisted coding often involves offering full-file cleanups or suggesting code simplifications. These tools, and many developers following similar patterns, tend to gravitate towards a single error-handling style. The most common preference is to raise exceptions rather than return None or structured error objects. This is often presented as a stylistic improvement, promoting clarity by centralizing error handling in try...except blocks.
However, this preference is not evidence-based when applied universally. It overlooks the crucial point that the system's *current* behavior, however messy, is the contract. Callers have already adapted. They have written code that branches on None returns, specific dictionary key absences, or the exact types of returned values. If your API returns None on a specific type of failure, and a downstream service checks for that None, changing the API to raise an exception instead will cause that downstream service to crash.
Happy-path unit tests, which only verify the correct, expected execution paths, will not catch these refactoring errors. They don't test the edge cases or the implicit error conditions that callers rely on. The real problem isn't a lack of types; it's an unrecorded, uncharacterized taxonomy of error conditions that the system has implicitly agreed upon through its observable behavior.
Pinning the Taxonomy with Characterization Tests
The solution is to treat the error taxonomy as the primary product surface. Before undertaking any significant model rewrite or refactor that might touch error handling, the first step must be to characterize the existing behavior. This involves writing specific tests—often called characterization tests or golden master tests—that capture the current output for a wide range of inputs, including all known error conditions.
These tests should not aim to validate correctness in a business logic sense, but to record *exactly* what the system currently does. For each input, what is returned? Is it None? A dictionary with "ok": False? A specific exception type with a particular message? These tests act as a snapshot of the current contract. They are the