Launchd Job Resumption: Solving the Post-Quota Exhaustion Puzzle

The Problem: The Circuit Closes, But Skipped Jobs Don't Come Back

Building robust systems involves anticipating failure. For jobs that rely on external APIs with rate limits, quota exhaustion is a predictable failure mode. My previous work on claude-quota-guard.py focused on the first half of this problem: detecting when a quota is met and halting all subsequent jobs. This script acted as a circuit breaker, preventing further API calls once the limit was reached. The immediate goal was to stop incurring costs or hitting hard rate limits that could lead to temporary bans. The script successfully returned an exit code of 0 upon detecting quota exhaustion, signalling a clean shutdown of the job execution pipeline.

However, this only addresses the symptom, not the long-term consequence. Once the circuit breaker trips, a queue of jobs that were *intended* to run but were skipped due to quota limits remains in limbo. The simpler approach might be to just let them be, assuming they will be retried by a higher-level scheduler or manually re-queued. But in many operational scenarios, especially those involving time-sensitive data synchronization or processing, simply abandoning these jobs is not an option. The harder question, the one that emerges after the immediate crisis of quota exhaustion is averted, is this: once the API quota resets or capacity becomes available again, what exactly should be re-run?

The naive approach of re-running *everything* that was skipped can be problematic. It risks immediately tripping the circuit breaker again if the underlying conditions haven't changed significantly. It also doesn't account for the fact that some skipped jobs might be less critical than others, or that re-running them might have unintended side effects. The core challenge is to intelligently decide which jobs are candidates for resumption, and to do so efficiently without overwhelming the system or the API.

Consider a scenario where you have 15 distinct jobs scheduled to run periodically. Each job performs a task that consumes a certain amount of quota from a shared pool. If the first few jobs consume the entire daily quota, the remaining 10 or more jobs are immediately halted. When the quota resets, perhaps at midnight, simply re-executing all 15 jobs from scratch is inefficient and potentially harmful. Some jobs might be idempotent, meaning running them multiple times has no adverse effect beyond wasted resources. Others might not be. Some might be time-sensitive, and running them late might render their output useless. The system needs a way to differentiate and prioritize.

The operational experience with claude-quota-guard.py highlighted this. Initial attempts to manage timeouts and load averages during periods of intense retries after a circuit breaker event proved challenging. A 1200-second timeout, which seemed generous, needed to be extended to 2700 seconds (45 minutes) to accommodate the backoff and retry logic. Furthermore, during periods of misconfiguration or aggressive retries, the system's load average spiked beyond 40, indicating a system under severe strain. These operational hiccups underscored the need for a more sophisticated strategy than simply stopping and starting jobs.

Intelligent Resumption Strategies

The post-quota exhaustion problem requires a strategy that goes beyond a simple on/off switch. It involves understanding the nature of the jobs themselves and the state of the system. Several approaches can be employed:

Prioritization Based on Job Criticality

Not all jobs are created equal. Some jobs might be responsible for critical data ingestion, security checks, or user-facing updates. Others might be for background analytics or less time-sensitive reporting. When the circuit closes, the system should first attempt to resume the highest-priority jobs. This requires a mechanism to tag or categorize jobs by their criticality level. A simple integer or string tag associated with each job definition could suffice. When the quota guard logic re-enables job execution, it queries this prioritization metadata and starts with the most critical tasks.

Statefulness and Idempotency Checks

A key consideration for resuming jobs is their idempotency. If a job is idempotent, running it multiple times produces the same result as running it once. For such jobs, a simple re-queue after quota reset is feasible, though still potentially inefficient. For non-idempotent jobs, or jobs where partial execution is problematic, a more careful approach is needed. This might involve tracking the progress of a job. For example, if a job involves processing a batch of 100 items and was stopped after item 50, the resumption logic should ideally pick up from item 51. This requires the jobs themselves to be designed with checkpointing or state-tracking capabilities.

A practical implementation could involve a metadata store where each job logs its last successful state or processed item ID. When the circuit breaker is disarmed, the resumption logic consults this store. If a job has a recorded last state, it can be resumed from that point. If no state is recorded, or if the job is known to be fully idempotent, it can be re-run from the beginning.