The Problem: Jobs Die Silently

Many background tasks on macOS, managed by launchd, are designed with an implicit assumption: they run to completion without interruption. However, real-world scenarios frequently break this assumption. A laptop lid closing, a network connection dropping upon wake, or even a process timing out can leave a launchd job in a partially completed state. This isn't just an inconvenience; it's a silent failure that can accumulate, leaving critical tasks unfinished by the time they are needed.

The author, Lily, encountered this precisely with a task to update their Obsidian vault using Claude Code. The process repeatedly died mid-run, often overnight due to macOS sleep or network issues. Instead of fighting perfectly to prevent every possible failure, Lily adopted a more pragmatic approach: the job should be guaranteed to finish by morning, even if it fails and restarts multiple times.

This led to the development of a new pattern for launchd jobs. It's the mirror image of jobs designed to run exactly once and then unload. This pattern assumes the job will die mid-run. It's designed to fire multiple slots a day, with a clear directive: "retry until it succeeds, then quit immediately on success." This approach shifts the focus from preventing failure to ensuring eventual success.

Implementing the Self-Healing Pattern

The core of this self-healing pattern lies in how the job is structured and how its state is managed. Traditional launchd jobs might simply exit on error, leaving the system in an unknown state. This new pattern, however, incorporates mechanisms to detect partial completion and ensure the task is eventually finished.

There are three primary ways a launchd job can fail to run to completion:

  1. Lid-close/Sleep: The system enters sleep mode, suspending all running processes. Upon wake, the job might not resume correctly or might find its environment changed (e.g., no network).
  2. Network Issues: Tasks requiring network connectivity can fail if the connection is lost or unavailable, especially after waking from sleep.
  3. Process Timeouts/Crashes: The job itself might encounter an internal error, hit a resource limit, or be terminated by the system for other reasons.

The new pattern addresses these by not treating a premature exit as a final failure. Instead, it designs the job such that it can be safely interrupted and restarted. This requires careful state management.

The 'Done-Marker' and Multi-Slot Firing

To manage this, two key components are introduced: multi-slot firing and a 'done-marker'.

Multi-Slot Firing

Instead of scheduling a job to run once a day at a specific time, the job is configured to fire multiple times. For instance, it might be scheduled to run every 6 hours. This increases the probability that at least one execution slot will occur when the system is awake, the network is available, and the job can complete successfully. If the job dies at 2 AM, the next scheduled slot at 6 AM will pick up where it left off, or at least attempt the task again.

Diagram illustrating multiple scheduled job execution slots throughout a day

The 'Done-Marker'

Crucially, the job needs a way to signal that it has successfully completed its task. This is achieved through a 'done-marker'. This is typically a file or a specific state within a database that the job creates or updates upon successful completion. Before starting its work, the job checks for this done-marker. If the marker exists, it means the job has already succeeded in a previous run, and the current execution can exit immediately. This prevents redundant work and ensures that once the task is done, it stays done.

The logic flow becomes:

  1. Check for the 'done-marker'.
  2. If the marker exists, exit successfully.
  3. If the marker does not exist, attempt the task.
  4. If the task succeeds, create the 'done-marker' and exit.
  5. If the task fails, exit with an error code. The next scheduled slot will then attempt the task again.

Designing for Interruption

This pattern fundamentally changes how developers should think about background tasks. Instead of aiming for a single, robust execution, the goal is to build a job that is idempotent and resilient to interruption.

Idempotency means that running the job multiple times has the same effect as running it once. If the job creates a file, running it again should not create a duplicate file or cause an error. This is often achieved by checking for the existence of the final output or state before proceeding. The 'done-marker' is a specific implementation of this for the entire job lifecycle.

Resilience to interruption means that if the job is killed mid-process, it can be restarted without causing data corruption or leaving the system in an inconsistent state. This often involves using atomic operations, transaction logs, or checkpoints within the job's execution. For the Obsidian vault update, this might mean ensuring that the sync process only commits changes after all files have been successfully processed and verified.

Practical Implications for macOS Users

For users who rely on automated tasks, this pattern offers a significant improvement in reliability. Tasks that previously might have failed silently and required manual intervention now have a built-in mechanism to self-correct. This is particularly valuable for long-running tasks or those dependent on unstable conditions like network availability.

The author's experience with the Obsidian vault update illustrates this perfectly. By embracing the possibility of failure and designing for retries, the task is now reliably completed by morning, regardless of overnight system events. This moves from a model of