The Challenge: High Volume, One QA
Imagine an AI-first development team shipping an average of 17 pull requests daily, peaking at 33 on busy days. Now, picture that entire output being overseen by a single QA engineer. This was the reality at pdf.net, where scaling manual testing alongside headcount was not an option. The team's strategy: a heavy bet on automation, specifically end-to-end (E2E) tests on staging, coupled with an automated system to manage their failures.
The core problem wasn't the E2E tests themselves, but the sheer volume of failures that would overwhelm a single human. Without an automated triage system, every red run would demand immediate investigation, grinding development to a halt. The solution needed to mimic, and then surpass, the diagnostic capabilities of a human QA, but at machine speed and scale. This meant not just identifying a failure, but understanding its root cause and assigning responsibility automatically.
Architecture: GitHub Actions, Claude, and Custom Logic
The automated triage system at pdf.net is built on a combination of GitHub Actions, Anthropic's Claude AI, and custom scripting. The goal was to create a bot that could autonomously analyze E2E test failures, determine the likely cause, and assign ownership or create necessary tickets, all without human intervention for the initial triage phase.
The process begins when a pull request is merged. GitHub Actions orchestrates the deployment to a staging environment and triggers the E2E test suite. If any tests fail, this is where the custom bot takes over. The bot's primary responsibility is to parse the test failure logs. This is a complex task, as E2E test failures can stem from various sources: application bugs, infrastructure issues, flaky tests, or even environmental problems unrelated to the specific code change.
Instead of simply flagging a failure, the bot uses Claude to interpret the error messages and stack traces. This AI-powered analysis allows the bot to go beyond pattern matching. It can understand the context of the failure, correlating error messages with the code changes introduced in the pull request. For instance, if a test fails due to a database connection error, the bot can infer whether the PR modified database-related code or if it's a broader infrastructure problem.
The Bot's Brain: Claude for Contextual Analysis
The integration of Claude is the secret sauce. The bot sends detailed information about the failing test, including the specific error, the relevant code diffs from the PR, and historical test results, to Claude. Claude's large language model capabilities are leveraged to:
- Identify Root Cause: Analyze the provided context to pinpoint the most probable cause of the failure. Is it a new bug introduced by the PR, or a pre-existing issue exacerbated by the changes?
- Assign Ownership: Based on the identified root cause, determine which developer or team is most likely responsible for fixing the issue. This is crucial for efficient bug resolution.
- Determine Action: Decide on the appropriate next step: create a GitHub issue, comment on the PR with findings, assign the PR to a developer, or escalate if the cause is unclear.
This AI-driven interpretation is far more sophisticated than traditional rule-based systems. It allows the bot to adapt to new types of errors and learn from its own triage decisions over time, continuously improving its accuracy.
Navigating the Rakes: Challenges and Lessons Learned
Automating E2E test triage is not without its pitfalls. The pdf.net team encountered several challenges:
- The Bot Blaming Innocent People: Early iterations of the bot incorrectly attributed failures to developers whose code was not the actual cause. This often happened when the bot failed to correctly correlate the test error with the specific code changes in the PR, or when unrelated infrastructure issues masked the true problem. Refining the prompt engineering for Claude and improving the context provided by GitHub Actions was key to mitigating this.
- Losing Tickets: Ensuring that every critical failure resulted in a trackable ticket was essential. The bot needed robust error handling and retry mechanisms to guarantee that if a ticket creation failed, the issue wouldn't be lost. Integrations with issue tracking systems had to be flawless.
- Silence When Silence Was Worst: The bot sometimes remained silent when a failure occurred, failing to report it or take action. This could be due to logic errors in the bot, network issues, or Claude returning an uninterpretable response. The system was enhanced with health checks and explicit failure-to-report alerts to address this.
- Flaky Tests: E2E tests are notorious for flakiness. The bot needed to be able to distinguish between genuine regressions and transient test failures. Strategies like running a test multiple times or analyzing historical data were incorporated to identify and quarantine flaky tests, preventing them from triggering unnecessary alerts.
The team learned that building such a system requires constant iteration. Prompt engineering for Claude, refining the data passed to the AI, and improving the logic for interpreting Claude's responses were ongoing tasks. The key was to treat the bot as an evolving tool, not a static solution.
The Cost-Benefit Analysis: Engineering Time vs. Bot Cost
The most compelling aspect of this automated triage system is its cost-effectiveness. The article states the entire setup costs less per month than one hour of engineering time. This figure starkly contrasts the potential cost of developer hours spent debugging failed E2E tests.
Consider the alternative: if a single QA engineer is triaging 17 PRs a day, and even a fraction of those result in complex E2E failures, their time is quickly consumed. Each failure might take anywhere from 30 minutes to several hours of investigation, depending on its complexity. If multiple developers are pulled into these investigations, the cost balloons rapidly. By automating this process, the team frees up engineers to focus on developing new features and addressing actual bugs, rather than spending time on diagnostic overhead.
The ROI is clear: a modest monthly spend on AI services and infrastructure yields significant savings in engineering productivity. This allows the company to scale its development velocity without proportionally increasing its QA headcount or suffering from increased bug leakage.
The Future of E2E Triage
This approach demonstrates a viable path for teams struggling with the scalability of E2E testing in high-velocity development environments. By leveraging AI for intelligent analysis and GitHub Actions for orchestration, it's possible to maintain high code quality standards without becoming a bottleneck. The success at pdf.net suggests that similar automation strategies could be widely adopted, fundamentally changing how development teams manage test failures and ensure software stability.