Agent Patches Need 3 Local Lane Approvals Before Shared Host Deployment

The Three-Lane Verdict System for Agent Patches

Deploying agent patches to shared CI/CD environments requires a robust validation process. Currently, an agent patch should remain off a shared evaluation host until three independent local lanes reach consensus. This ensures that the patch is stable and doesn't introduce regressions or consume valuable compute resources unnecessarily. The core principle is to shift rejection of problematic patches as early as possible in the development lifecycle, ideally on a developer's local machine, before they ever touch shared infrastructure.

This proposal introduces a structured, three-lane approach to agent patch validation. Each lane is responsible for a distinct set of checks. The objective is to create a gatekeeping mechanism that is both thorough and efficient, preventing the waste of shared host time on diffs that could have been identified and rejected locally. The steps outlined are a proposal, with commands and snippets serving as illustrative examples for adaptation within a development branch. It is important to note that this document does not present measured pass rates, quotas, or claims about the availability of hosted options.

The fundamental idea is that a patch is only cleared for shared evaluation once it has passed these three independent local checks. This mirrors a quality control process where different inspectors look for different types of flaws. If any single lane identifies a problem, the patch is rejected or flagged for further review, preventing it from proceeding to more expensive testing stages.

Lane Responsibilities and Decision Authority

Each lane within this system has a specific mandate:

Lane One: Behavior Contract Verification

The first lane's primary responsibility is to determine if a behavior contract has been broken. Behavior contracts define the expected interactions and interfaces of the agent. If a patch modifies the agent in a way that violates these predefined contracts, lane one will flag it. This check is crucial for maintaining API stability and ensuring that the agent continues to function as expected by other components and services that rely on it.

Lane Two: Fixture Byte Integrity

The second lane focuses on the integrity of fixture bytes. Fixtures are essential for testing, providing consistent and reproducible data sets. Lane two verifies whether the fixture bytes have been altered without proper review. This guards against accidental corruption or unauthorized modifications to test data, which could lead to false positives or negatives in subsequent testing phases. A change in fixture bytes without a corresponding review indicates a potential issue that needs immediate attention.

Lane Three: Flaky Assertion Management

The third lane addresses the issue of flaky assertions. Flaky assertions are tests that intermittently pass or fail without a clear reason, often due to race conditions or environmental factors. Lane three's role is to determine if a red result (a test failure) is due to a known unstable assertion that is currently under a freeze, or if it represents a genuine new failure. An unexpired freeze means the assertion is known to be flaky and is temporarily ignored for the purpose of this evaluation, provided it doesn't impact other lanes. If the failure is not covered by an unexpired freeze, it is treated as a new, critical failure.

Interdependencies and Rejection Logic

These lanes are not independent substitutes for one another. A green test suite, for instance, can mask a broken behavior contract if the fixtures were rewritten to accommodate the change. Similarly, a freeze on a flaky assertion might allow a patch to pass lane three, but if that patch also broke a fundamental behavior contract, the entire process is undermined. The order of operations is critical: lane one's contract verification, lane two's fixture integrity check, and lane three's flaky assertion management must all pass. Reversing this order, or allowing a patch to proceed without all three lanes giving a 'green light,' can lead to wasted host time on diffs that a developer's laptop could have already rejected.

The property oracle, a mechanism for verifying expected properties, is also a key component. It must hold true, meaning the agent's behavior aligns with its declared properties. Coupled with the fixture digest matching a locked set of known-good fixture data, these checks form a comprehensive local validation. Any flaky assertion must be behind an unexpired freeze. Crucially, this freeze must not inadvertently switch off or mask failures in the other two lanes. If a patch fails any of these conditions, it should be rejected locally, preventing it from consuming resources on shared evaluation hosts.

The Proposal's Impact on Development Workflow

Implementing this three-lane verdict system aims to refine the agent patch deployment pipeline. By enforcing rigorous local checks, the system seeks to reduce the burden on shared CI/CD infrastructure. This translates to faster feedback loops for developers, as problematic patches are identified and addressed earlier. It also conserves compute resources, as fewer patches will reach the more resource-intensive shared evaluation stages only to be rejected.

The proposal emphasizes that this system is about optimizing resource utilization and improving the signal-to-noise ratio in testing. It's not about setting arbitrary quotas but about establishing a clear, deterministic process for patch validation. The success of this system relies on developers integrating these local checks into their workflow, ensuring that patches are thoroughly vetted before being submitted for broader evaluation. The ultimate goal is a more efficient and reliable development process for agent patches.