Agent Queue Full: How to Track and Prevent Lost Work

The Perils of Agent State Ambiguity

Long-running agents, critical components in many automation workflows, rarely fail due to simple errors like invalid JSON responses. Their failure modes are more insidious, often occurring at the murky boundaries between distinct states: queued, leased, running, and completed work. This ambiguity is a silent killer of reliable automation. Imagine a worker process restarting while it still holds a lease on a task. Or a queue accepting more work than the system can actually handle. Perhaps a retry mechanism kicks in before the initial attempt has definitively stopped. Without explicit state management, these scenarios can lead to the same side effect being executed multiple times, or worse, work being lost entirely without any clear indication of failure.

This isn't a hypothetical problem. Many automation systems, from CI/CD pipelines to data processing jobs, grapple with this. The core issue lies in representing work not just as pending or done, but with a nuanced understanding of its lifecycle. Relying solely on a binary pending/done state is akin to navigating a complex city with only a map of your starting point and your destination, ignoring all the intersections, traffic lights, and one-way streets in between. The result is often chaos, lost time, and duplicated effort.

The challenge intensifies when agents run on 'always-on' hosts. The temptation is to let them run continuously, assuming they'll manage themselves. However, without a visible, explicit control plane, pressure and ambiguity build up unseen until a critical failure occurs. This article proposes a small, implementable control-plane model using a relational database and a worker loop to make these states visible and manageable, preventing lost work before it becomes a crisis.

Defining Clear Agent States

To combat this ambiguity, we must move beyond a simplistic two-state representation of work. A minimum of four distinct states is necessary to accurately track an agent's progress and prevent lost work:

• Queued: The work has been accepted by the system but has not yet been assigned to a specific worker. It's waiting in line.
• Leased: The work has been assigned to a specific worker. This assignment has a defined deadline, acting as a safety net. If the worker fails to acknowledge completion or extend the lease before the deadline, the work can be reassigned.
• Running: The worker has actively started executing the task. This state signifies that the work is in progress, not just assigned.
• Succeeded: The work has been completed successfully by the worker.
• Failed: The work could not be completed due to an error during execution.
• Cancelled: The work was intentionally stopped before completion.

By explicitly defining and tracking these states, we gain visibility into the entire lifecycle of a task. This model provides the necessary granularity to identify where work might be getting stuck or lost. For instance, a high number of tasks in the 'leased' state without transitioning to 'running' could indicate worker saturation or a failure in the worker's heartbeat mechanism. Similarly, a sudden drop in tasks moving from 'running' to 'succeeded' points to execution-level problems.

Implementing a Robust Control Plane

Building this visibility requires a dedicated control plane. A relational database serves as an excellent foundation for this, offering structured storage, querying capabilities, and transaction integrity. Each task in the system can be represented as a row in a 'tasks' table, containing fields for its unique ID, status, assigned worker ID, lease expiration time, and execution timestamps.

The worker loop, running on each agent, continuously polls the database for tasks. When a worker is ready for work, it queries for tasks in the 'queued' state. It then attempts to 'lease' a task by updating its status to 'leased' and setting a lease expiration timestamp. This lease acts as a timeout; if the worker crashes or becomes unresponsive, the lease will eventually expire, allowing another worker to pick up the task. This is crucial for fault tolerance.

Once a worker has leased a task, it transitions the task's status to 'running' and begins execution. Upon completion, it updates the status to 'succeeded' or 'failed'. If the lease expires before the worker can update the status, a separate monitoring process, or indeed another worker in its next polling cycle, can detect this and re-queue the task, potentially marking the original worker as unhealthy.