The Illusion of Uptime

An agent endpoint returning an HTTP 200 status code signals that the service is technically "up." This is the traditional measure of availability. However, for sophisticated agent systems, this metric is woefully insufficient. The system might respond successfully, but the actual task it was meant to perform could fail in numerous ways. Successful responses might skip crucial retrieval steps, tool calls could exceed their allocated budgets, or critical write actions might proceed without the necessary approvals. The system is available, but the product's functionality is not.

Traditional service-level indicators (SLIs) like uptime remain important, but agent systems introduce a new, critical layer of complexity. The request handler might not fail, but the path and outcome of the agent's execution can still lead to failure. This disconnect between perceived availability and actual functional success necessitates a shift in how we define and measure service level objectives (SLOs) for these systems.

Starting From the User-Facing Promise

Google's Site Reliability Engineering (SRE) guidance defines an SLO as a target value or range for a measured service level. Crucially, this target should reflect what users actually need and expect from the service, not just the metric that is easiest to collect. For an agent system, a vague definition of "successful request" is inadequate. The promise to the user must be broken down into distinct, measurable indicator families that capture the true success of the agent's operation.

Outcome: The True Measure of Agent Success

The most critical indicator for an agent system is whether it produced the intended, externally observable result. This is the core of the user's expectation. Did the agent complete the task it was designed for, and did it achieve the desired end state? This goes beyond a simple HTTP 200. It requires deep inspection of the agent's actions and their consequences.

Defining Measurable Outcome Indicators

To effectively measure outcome, we must define specific, quantifiable metrics. For example, if an agent's purpose is to summarize a document, the outcome SLO should not just be about the API call succeeding. It should measure the quality and completeness of the summary, or whether the summary was delivered within a specific timeframe and format. If an agent is tasked with booking a flight, the outcome SLO would be whether a valid booking confirmation was received, not just that the booking API was called.

Consider an agent designed to automate customer support ticket resolution. An HTTP 200 might mean the agent processed the ticket, but if it failed to categorize it correctly, assign it to the right team, or provide a helpful response to the customer, the outcome is a failure. The user-facing promise of efficient and accurate support was not met, despite the system being technically "available."

Beyond Availability: Path and Execution Metrics

Agent systems involve complex workflows. The path an agent takes to achieve an outcome can be as important as the outcome itself. Failures can occur at various stages within this path:

1. Retrieval Failures

An agent might need to retrieve data from multiple sources before executing its primary task. If these retrieval steps fail, even if the final API call to, say, update a database, returns a 200, the overall operation has failed. The agent could not access the necessary information to perform its job correctly.

2. Tool Call Failures

Many agents rely on external tools or APIs to perform sub-tasks. These tool calls can fail for various reasons: exceeding rate limits, invalid parameters, or the tool itself being unavailable. If an agent proceeds with its workflow despite a critical tool call failing, the outcome will be compromised.

3. Budget Exceedances

In contexts where agent actions are metered (e.g., API call costs, token limits for LLMs), exceeding a budget can be a form of failure. An agent might successfully complete a task but at an exorbitant cost, or it might truncate its work prematurely because it hit a financial or computational limit. This is a functional failure, even if the endpoint responded with a 200.

4. Approval Gate Failures

For agents performing sensitive actions (e.g., financial transactions, data modifications), approval workflows are critical. If an agent initiates an action that requires human or system approval, and that approval is denied or never obtained, the agent's execution path has failed to reach a successful conclusion. A successful API call to initiate the action does not guarantee the desired end state.

Rethinking Agent SLOs

To address these complexities, agent SLOs must evolve. Instead of focusing solely on availability, we need to incorporate metrics that capture the end-to-end success of the agent's intended function. This requires a multi-dimensional approach:

  • Outcome-Oriented SLOs: Directly measure the achievement of the user's goal. What is the success rate of the agent producing the expected output or state?
  • Functional Path SLOs: Monitor the success of critical internal steps within the agent's workflow. What is the success rate of essential data retrieval or tool calls?
  • Resource Adherence SLOs: Track adherence to defined budgets and limits. What percentage of agent tasks complete within allocated costs or computational bounds?
  • Workflow Integrity SLOs: For agents with approval gates, measure the success rate of actions that require and receive proper authorization.

This shift requires a deeper understanding of the agent's purpose and its internal mechanics. It means instrumenting the system not just for endpoint health, but for the success of each critical step in its execution. The goal is to ensure that when an agent responds, it has not only responded successfully but has also accomplished what the user intended.