The Silent Rot of Agent Evaluation Sets
An agent evaluation set begins healthy. It comprises obvious intents, a few anticipated tool failures, and the happy paths crucial for initial development. However, six months down the line, production environments rarely mirror this static snapshot. New tools emerge, user phrasing evolves, and previously minor fallbacks might now handle a significant portion of traffic. Yet, continuous integration (CI) systems often continue to pass the same limited set of twelve examples, reporting a false sense of security.
The core issue isn't solely stale prompts; the evaluation set itself is rotting. Without a mechanism to integrate observed production behavior back into the testing framework, these sets become increasingly detached from reality. A truly durable evaluation program requires a controlled, systematic path from observed failures in production back into the testing suite.
This process can be visualized as a loop: observed run -> candidate -> human review -> sanitization -> eval case -> CI -> release comparison. Each step transforms raw production data into a reliable test case.
Defining a Failure Candidate
The first critical step is defining what constitutes a 'failure candidate.' It is imperative to avoid directly copying complete production conversations into a test folder. This approach is brittle and often includes sensitive user data or irrelevant conversational context. Instead, create a bounded, intermediate record for each potential failure.
A failure candidate should capture the essential elements needed to reproduce and understand the failure without carrying the entire production baggage. This includes:
- User Utterance: The exact phrasing the user employed.
- Agent Response: The output generated by the agent, including any tool calls or final replies.
- Tool Execution Details: If a tool was invoked, record the tool name, its parameters, and the result returned by the tool. This is vital for diagnosing tool-specific failures.
- Observed Outcome: A brief description of why this interaction is considered a failure. Was it an incorrect answer, an inappropriate tool use, a user frustration, or a missing capability?
This structured intermediate record acts as a clean slate, ready for the next stage of review and sanitization. It isolates the problematic interaction from its surrounding noise.
Human Review and Sanitization
Raw production data is often messy and may contain personally identifiable information (PII) or sensitive details. Therefore, human review and sanitization are non-negotiable steps. A dedicated reviewer, or a team of them, examines each failure candidate.
The reviewer's tasks include:
- Validation: Confirming that the interaction is indeed a failure and warrants inclusion in the evaluation set. Not all production issues might be relevant for CI testing.
- Contextualization: Adding any necessary context that might not be immediately obvious from the raw interaction.
- Anonymization: Scrubbing all PII and sensitive data. This might involve replacing names with generic placeholders (e.g., `[USER_NAME]`), masking credit card numbers, or removing any other confidential information. The goal is to make the test case safe for inclusion in version control and accessible to the entire development team.
- Generalization: Modifying specific details to make the test case more broadly applicable. For instance, changing a specific product name to a generic category if the failure is related to product handling in general, not a bug in a single product.
This stage is crucial for transforming a real-world, potentially problematic interaction into a safe, representative, and actionable evaluation case.
Creating Eval Cases and CI Integration
Once a failure candidate has been reviewed and sanitized, it is transformed into a formal evaluation case. This case is then integrated into the CI pipeline. The goal is to have these new, production-derived test cases run automatically with every code change.
An evaluation case typically includes:
- The Sanitized Utterance: The user input, cleaned of PII and generalized.
- Expected Output/Behavior: The desired outcome from the agent. This could be a specific text response, a particular tool call with expected parameters, or a refusal to answer if the query is out of scope.
- Assertions: Automated checks to verify if the agent's actual output matches the expected output.
Integrating these cases into CI ensures that any regressions related to these failure modes are caught immediately. If a new code change causes an agent to fail on a previously handled scenario, the CI pipeline will flag it. This provides a safety net against the 'rotting' effect of static eval sets.
Release Comparison and Continuous Improvement
The final piece of the loop is release comparison. After new code is deployed to production, monitoring its performance is key. The system should track how the agent performs on the new set of evaluation cases in a live environment. This involves comparing the agent's actual production behavior against the expected behavior defined in the evaluation cases.
This comparison serves several purposes:
- Validating the Loop: It confirms that the newly added evaluation cases are correctly identifying failures. If the agent passes the new test case in CI but fails in production, it indicates an issue with the test case itself or the sanitization process.
- Identifying New Failures: By analyzing production logs and metrics, teams can identify entirely new types of failures that were not captured by the existing eval set. These new failures can then feed back into the 'failure candidate' stage, restarting the loop.
- Measuring Improvement: Over time, teams can track the reduction in specific failure categories as they are addressed and new test cases are added.
This continuous feedback mechanism ensures that the agent's evaluation set evolves alongside the production environment, maintaining its relevance and effectiveness. Without this loop, development teams risk deploying agents that are increasingly out of sync with real-world user interactions and system capabilities.