The Problem: Duplicate Side Effects from Tool Retries

A seemingly successful process can hide critical flaws. The author encountered a situation where a tool, intended to create a single incident record, erroneously generated two identical tickets minutes apart. This wasn't the result of a sophisticated attack, but a subtle failure in a common retry mechanism. The planner requested one record, but the system delivered two.

The root cause was a race condition. The worker successfully posted the ticket but then vanished before its result could be confirmed by the planner. The planner interpreted this silence as a failure, triggering the same step again. A second worker then committed the same action, resulting in a duplicate entry. This highlights a dangerous assumption: trusting a tool-failed retry line without verifying if the intended side effect already occurred.

The core invariant that the common retry loop drops is simple: a single planner step should produce at most one committed side effect, unless a named compensation process is explicitly invoked. When a tool fails to acknowledge its action, it's ambiguous. Does the tool truly fail, or did the action succeed, but the acknowledgment mechanism itself failed?

Diagram illustrating the race condition between tool execution and planner acknowledgment

The Invariant and Its Breakdown

In an ideal system, idempotency is key. A single request, regardless of how many times it's retried, should result in the same state. However, in complex distributed systems, achieving true idempotency, especially with external tools, is challenging. The scenario described involves a planner orchestrating a sequence of tool executions. When a tool executes a step, it's meant to produce a side effect – in this case, creating an incident ticket.

The problem arises when the communication between the tool and the planner breaks down. If the tool executes the action but fails to send an acknowledgment, the planner might assume the action failed. This triggers a retry. If the original action *did* succeed, but the acknowledgment was lost, the retry will lead to a duplicate side effect. This is precisely what happened: the first ticket was created, but the acknowledgment was lost. The planner retried, and a second identical ticket was created.

This violates a fundamental principle of reliable distributed systems: the "at most once" delivery guarantee for side effects, unless explicitly designed for "exactly once" with compensation. The common retry loop often defaults to "at least once" for reliability, but without a mechanism to detect and deduplicate already-processed operations, it can devolve into an uncontrolled "more than once" scenario.

Introducing the Deduped Outbox Pattern

To address this, the concept of treating tool intent as a "deduped outbox" before planner retries is crucial. An outbox pattern is typically used to ensure that an action within a transaction is reliably published to an external system. In this context, we can adapt it to manage tool intents and prevent duplicate commits.

The core idea is to create an intermediate, reliable store (the outbox) for the planner's intents *before* they are sent to the tool. This outbox acts as a single source of truth for what the planner *intends* to happen. When the planner decides to execute a tool step, it first records this intent in the deduped outbox. This record would include all necessary information for the tool to execute the action, along with a unique identifier for the operation.

The worker responsible for executing the tool action would then poll this outbox. Upon picking up an intent, it would execute the corresponding tool command. Crucially, after successful execution, the worker would mark the intent as processed in the outbox. If the worker crashes *after* executing the tool but *before* marking the outbox, the next worker to poll the outbox would see the *same* intent still marked as unprocessed. This is where the deduplication comes in.

The outbox itself must be designed to prevent duplicate entries for the same logical operation. This can be achieved by ensuring that when an intent is recorded, it includes a unique transaction ID or operation ID. If the planner attempts to record the same intent twice (due to a retry loop), the outbox mechanism should detect this duplicate and either ignore the second request or update the existing entry, rather than creating a new one.

How Deduping Prevents Double-Commits

Consider the problematic scenario again: the planner requests a ticket, the worker posts it, but the acknowledgment is lost. Without a deduped outbox, the planner retries, leading to a second ticket. With the deduped outbox:

  1. The planner decides to create an incident ticket. It generates a unique operation ID (e.g., `op-12345`) and records this intent in the outbox: `[op-12345, create_ticket, incident_details]`.
  2. A worker picks up `op-12345` from the outbox.
  3. The worker successfully calls the tool API to create the incident ticket.
  4. The worker marks `op-12345` as processed in the outbox.
  5. If the worker crashes *before* marking it processed, the next worker picks up `op-12345`. It sees it's already marked processed and moves on. If it crashes *after* marking it processed, the next worker sees it marked processed.
  6. Now, imagine the planner *thinks* it failed and tries to retry. It attempts to record `[op-12345, create_ticket, incident_details]` again. The outbox, being deduped, recognizes `op-12345` and either rejects the duplicate or updates the existing entry. No new intent is created.
  7. Consequently, no second worker is dispatched for `op-12345`, and no duplicate ticket is created.

This pattern effectively transforms the planner's intent into a reliable, deduped queue. The outbox acts as a buffer and a state tracker. It ensures that each logical operation requested by the planner is attempted exactly once by a worker, even in the face of worker failures or planner retries. The state of the outbox entry (unprocessed, processing, processed) provides the necessary signal to prevent redundant work.

Broader Implications and Future Considerations

The failure described is not isolated to ticket creation. Any system relying on external tools or services that have side effects is vulnerable to similar issues. If a system cannot reliably determine if an action has already been performed, retries can lead to data corruption, inconsistent states, and operational nightmares. This is particularly relevant in event-driven architectures, microservices, and complex orchestration workflows.

The deduped outbox pattern, while adding a layer of complexity, provides a robust solution for ensuring exactly-once processing semantics for side effects. It requires a reliable, transactional mechanism for the outbox itself, often implemented using a database or a message queue that supports transactions. The cost of this complexity is often far outweighed by the cost of debugging and rectifying duplicate commits in production.

What nobody has addressed yet is the exact performance overhead of maintaining a deduped outbox for extremely high-throughput systems. While conceptually sound, the latency introduced by an extra write and read to the outbox, and the complexity of managing unique operation IDs across distributed components, needs careful benchmarking. However, for critical operations where duplicate commits are unacceptable, the trade-off is usually favorable.