Tiny LLM Request Recorder Aids Production Failure Reproduction

The Elusive LLM Failure

Large Language Model (LLM) failures are notoriously difficult to pin down. A user might report an empty response, a vanished tool call, or a request that inexplicably fails on one service but not another. When developers dive into the logs, they often find only generic error messages like 'LLM request failed: 400 Bad Request'. This technical confirmation is operationally useless because it lacks the crucial context needed for reproduction and debugging.

The missing piece of the puzzle is almost always the exact shape of the failed request. This includes the specific model queried, the parameters used, the structure of message roles, any tool definitions involved, the timeout behavior, and the raw, unprocessed response from the provider. Without this granular detail, replicating the failure becomes a frustrating guessing game.

Observability platforms exist, but they can be overkill for this specific problem. The need is for a lightweight solution that captures just enough information to understand and replay a failed call, without the overhead or security risks of logging sensitive API keys. To address this, a custom request recorder was built, leveraging the native fetch API.

What the Recorder Captures

The recorder is designed to capture essential details for each LLM request. The primary goal is to store enough information to allow for inspection and potential replaying of a failed interaction. The key pieces of data logged are:

• A unique identifier for each request, facilitating easy referencing and tracking.
• The full request URL, including any query parameters.
• The HTTP method used (e.g., POST, GET).
• The request headers, excluding sensitive information like API keys.
• The request body, which contains the LLM prompt, parameters, and any other payload.
• The response status code, indicating the success or failure of the request.
• The response headers, providing metadata about the server's reply.
• The raw response body, which is critical for understanding what the LLM or API actually returned.
• An optional timeout duration, capturing how long the request was allowed to run before being aborted.

This curated set of data provides a comprehensive snapshot of the interaction. It’s enough to understand what was sent and what was received, enabling developers to recreate the exact conditions that led to a failure.

Implementation Details

The recorder functions as a wrapper around the standard fetch API. When a request is made using this wrapper, it intercepts the call. Before passing the request to the actual fetch implementation, it serializes and stores the request details. After the fetch call completes, whether successfully or with an error, it captures the response details.

The storage mechanism can be as simple as writing to a file or sending the data to a dedicated logging service. The critical constraint is that no sensitive information, particularly API keys, should ever be persisted. This is typically achieved by filtering headers and potentially redacting specific fields within the request or response bodies if they are known to contain secrets.

The choice of fetch as the interception point is strategic. It is the modern standard for making HTTP requests in browsers and Node.js environments. By wrapping this function, the recorder can capture a wide range of LLM interactions, regardless of the specific LLM provider or library being used, as long as they ultimately rely on fetch.

One of the core advantages of this approach is its minimal overhead. It doesn't require a complex agent or a full-fledged observability stack. For developers dealing with intermittent or hard-to-reproduce LLM issues, this lightweight solution offers a significant improvement in debugging efficiency.

Reproducing Failures

With the logged request data, reproducing a failure becomes a straightforward process. A developer can retrieve the relevant log entry, extract the stored request details, and then replay the request using a tool like curl or a simple script. This direct replay bypasses the application logic that might have introduced subtle timing or state-dependent issues, focusing solely on the interaction with the LLM API.

For example, if a user reported that a specific prompt resulted in an empty array of tool calls, the developer would find the corresponding log entry. They could then reconstruct the exact JSON payload, including the message history, the tool definitions provided to the model, and any other parameters. This reconstructed request could then be sent to the LLM API endpoint. Any discrepancy between the original logged response and the response from the replayed request would immediately highlight changes in the LLM's behavior or issues with the API itself.

This method is invaluable for debugging issues that occur only in production. Production environments often have unique configurations, network conditions, or data states that are difficult to replicate in development or staging. By capturing the exact request that failed in production, developers can isolate the LLM interaction from these environmental variables.

The surprising detail here is not the simplicity of the recorder, but how much debugging power it unlocks from such minimal data capture. It turns vague error reports into actionable, reproducible test cases.

Beyond Basic Logging

While the primary use case is debugging, the logged data can also inform future development. Analyzing patterns in failed requests can reveal common pitfalls in prompt engineering, tool design, or parameter tuning. For instance, if many failures involve complex tool definitions, it might indicate a need to simplify the tool schema or improve the LLM's ability to interpret them.

The recorder can also be extended. One could add capabilities to automatically replay requests against different LLM versions or providers to benchmark performance or check for regressions. Another extension could be to automatically retry failed requests with slightly modified parameters based on common error patterns, effectively creating an automated debugging loop.

The challenge remains in identifying the right information to log without creating excessive data storage or performance overhead. Striking that balance is key to a practical and effective LLM debugging tool.