LLM APIs Fail with Personality

Every API encounters failures. Large Language Model (LLM) APIs, however, exhibit distinct failure modes that demand specific handling strategies. Ignoring these nuances can lead to cascading failures, production downtime, and a degraded user experience. The most common and often misunderstood failure is the 429 Too Many Requests error. This is not a bug in your application but a signal from the provider to reduce your request rate. Treating it as a fatal error, as some teams have, can result in significant outages. Beyond rate limits, LLM APIs commonly fail with 5xx server errors, network timeouts, and sometimes, custom error codes indicating issues with the request payload or internal provider states. Each failure type requires a tailored response to ensure resilience.

The core strategy for handling these failures revolves around three key patterns: Retry, Backoff, and Circuit Breakers. These are not novel concepts, but their application to LLM APIs requires careful consideration of the specific failure modes and the often-unpredictable nature of these services.

Implementing Robust Retries

When an LLM API call fails, the first instinct might be to retry. However, blind retries are dangerous, especially for non-idempotent operations. For LLM APIs, retries are appropriate for transient errors like rate limits (429), server errors (5xx), and network timeouts. Validation errors (e.g., 400 Bad Request) should never be retried blindly, as they indicate a fundamental issue with the request itself that will persist.

The most effective retry strategy involves several components:

  • Conditional Retries: Only retry specific error codes (429, 5xx, timeouts). Never retry 4xx errors unless explicitly stated by the API provider for specific cases.
  • Respecting Retry-After Header: When a provider sends a 429 error, it often includes a Retry-After header specifying a duration to wait before retrying. Always honor this header. It's the provider's explicit instruction on how to manage rate limits.
  • Exponential Backoff with Jitter: Instead of retrying immediately or with a fixed delay, use exponential backoff. This means the wait time between retries increases exponentially (e.g., 1s, 2s, 4s, 8s). Adding jitter—a small, random delay—to each backoff period prevents multiple clients from retrying simultaneously after an outage, which can re-overwhelm the server. Think of it like a group of people trying to get through a narrow door; if everyone tries at the exact same moment, no one gets through. Spreading out the attempts makes it more likely for everyone to pass eventually.
  • Deadline/Timeout for Retries: Impose an overall deadline for all retry attempts. This prevents a failing API call from consuming excessive resources or blocking your system indefinitely. If the total time spent retrying exceeds this deadline, the operation should be considered a failure.

Consider a scenario where your application sends a batch of requests to an LLM API. Without proper retries, a single 429 error could halt the entire batch processing. With exponential backoff and jitter, your application gracefully slows down, respects the provider's limits, and continues processing once the rate limit window has passed or the specified Retry-After duration has elapsed.

Diagram illustrating exponential backoff with jitter for API retries

The Circuit Breaker Pattern

While retries handle individual transient failures, they don't protect your system from a persistently degraded or unavailable service. This is where the circuit breaker pattern becomes critical. A circuit breaker acts like an electrical circuit breaker: if too much current flows (too many failures), it trips, stopping the flow of electricity (requests) to prevent damage. In software, it prevents your application from repeatedly hammering a failing external service.

A typical circuit breaker operates in three states:

  • Closed: Requests are allowed to pass through to the service. If a request fails, a failure counter is incremented. If the failure rate exceeds a predefined threshold within a certain time window, the breaker trips and moves to the Open state.
  • Open: All requests are immediately rejected without attempting to contact the service. This state is maintained for a configured timeout period. This allows the failing service time to recover.
  • Half-Open: After the timeout in the Open state, the breaker moves to Half-Open. A single test request is allowed to pass through. If this request succeeds, the breaker resets to Closed. If it fails, it immediately returns to Open, and the timeout period restarts.

Implementing a circuit breaker around your LLM API calls ensures that if the API becomes unstable, your application doesn't contribute to its downfall or get bogged down in endless retries. Instead, it quickly fails fast, returning an error to the user or initiating a fallback mechanism. This is crucial for maintaining the stability and responsiveness of your own system. Imagine your LLM-powered chatbot suddenly receives a flood of requests, and the LLM provider's service experiences an outage. Without a circuit breaker, your chatbot would keep trying to connect, consuming resources and potentially freezing. With a circuit breaker, it would immediately stop sending requests to the LLM, perhaps responding with a message like "I'm having trouble connecting right now, please try again later," while internally logging the outage and waiting for the LLM service to recover.

Code Shape and Considerations

The implementation of these patterns typically involves a wrapper function or a dedicated client library. For example, you might create a function like callLLMWithRetryAndCircuitBreaker(prompt, model, options).

Key considerations during implementation:

  • Idempotency: Reiterate that non-idempotent actions must never be retried blindly. If an LLM call triggers a side effect (e.g., creating a resource that cannot be duplicated), retrying it on failure could lead to unintended consequences. Ensure your LLM use cases are designed with idempotency in mind or use unique request IDs to detect and prevent duplicate operations.
  • Configuration: Make retry counts, backoff factors, jitter ranges, overall deadlines, and circuit breaker thresholds configurable. These parameters may need tuning based on the specific LLM provider's SLAs, your application's criticality, and observed failure patterns.
  • Observability: Log all retries, backoffs, tripped circuit breakers, and final failures. This data is invaluable for debugging, performance tuning, and understanding the reliability of the LLM service you depend on.
  • Fallback Strategies: Consider what happens when retries are exhausted or the circuit breaker remains open. Can you serve a cached response? Can you use a less capable but more reliable fallback model? Can you gracefully degrade functionality?

By integrating retry logic with exponential backoff and jitter, and wrapping these calls within a circuit breaker, developers can build more resilient applications that interact reliably with LLM APIs. This layered approach prevents common failure modes from cascading and ensures that your system remains available even when external dependencies falter.