The Silent Failure Problem

Your daily report stopped arriving two days ago. Nobody noticed, because a message that does not arrive looks exactly like a quiet day. This is the essence of a silent failure: a system designed to alert you to problems has itself failed, without generating any visible error.

Consider the scheduled jobs in your repository. How many of them end by sending something? A report, a digest, a backup confirmation, an alert. Now, try to say, without opening any files, which of those actually sent something yesterday. For most teams, this is an impossible question. The cost of this gap remains zero until the single day it becomes critical.

The problem is not new. Developers often implement checks, but these checks themselves can become the source of silent failures. A common pattern is to have a script that runs daily, and if it fails to send its expected output, it should ideally trigger another alert. However, if the alerting mechanism itself is broken, or if the script's output is misinterpreted, the failure propagates without a trace.

Why Source Control Guards Aren't Enough

A reader, Mads Hansen, pointed out a critical flaw in a proposed fix: simply checking that a scheduled job is declared in a workflow file does not guarantee it is actually running or succeeding. This guard proves only that a declaration exists in source control. It's like checking if a fire alarm is installed in a building, but not checking if it's connected to power or if the siren works.

The real test is a deployed canary: a small, live instance or a specific check that runs on the actual production path. This canary must verify that the job is not only scheduled but also executing as expected, using the same identity, secrets, and resources as the main job. This ensures the *deployed* system behaves as intended, not just the code in version control.

This distinction is crucial. Source control tells you what you *intended* to deploy. The running system tells you what is *actually* deployed and operating. When dealing with critical alerting systems, the deployed reality is the only one that matters.

The Two-Grep Solution

To address this, a practical approach involves two specific `grep` commands. These commands are designed to check the *actual execution logs* and *system configurations* rather than just static code. They act as a form of deployed canary, verifying the operational status of your critical jobs.

Step 1: Check for Expected Output in Logs

The first `grep` command aims to find evidence of the job having successfully run and produced its expected output. This typically involves searching through log files or output streams that capture the execution of your scheduled tasks.

Let's assume your critical job is a daily report generator, and its successful execution results in a log entry containing a specific string, like 'Daily report generated successfully'. The command would look something like this:

grep "Daily report generated successfully" /var/log/my_pipeline.log

Or, if your jobs are logged in a more structured way, perhaps within a CI/CD system's logs:

grep "job_name: report_generator, status: success" /var/log/ci_cd_runner.log

The key is to identify a unique, unambiguous string that *only* appears when the job has completed its intended task. If this `grep` command returns no results for a given day, it's a strong indicator that the job did not run, or at least did not produce its expected outcome. This is the first alarm bell.

Example terminal output showing grep command searching for a success string in a log file

Step 2: Verify the Alerting Mechanism Itself

The second `grep` command addresses the possibility that the job ran, but the *alerting mechanism failed*. This means the output was generated, but the notification system (email, Slack, PagerDuty, etc.) did not receive or process it. This is the truly silent failure.

To check this, you need to grep for evidence that the *alerting script or service* was invoked. This might involve looking for log entries from your email service, your messaging queue, or your incident management tool.

For example, if your alerting script is named `send_alert.sh` and it's supposed to be called upon failure:

grep "send_alert.sh invoked with alert_type=failure" /var/log/alerting_system.log

Or, if you use a specific API endpoint for your alerting service:

grep "POST /api/v1/alerts" /var/log/network_traffic.log

This second check is vital. It confirms that even if the primary job failed (and thus should have triggered an alert), the alert itself was not sent. If the first `grep` finds no successful job output, and the second `grep` finds no evidence of alert invocation related to that job's failure, you have a double failure – the job is down, and the system that should tell you about it is also down, or misconfigured.

Implementing the Guard

These `grep` commands should not be run manually. They need to be automated. The most effective way to implement this is to create a separate, simple monitoring job. This job runs *after* your critical jobs are expected to have completed and *after* any potential failure alerts should have been sent.

This monitoring job would execute the two `grep` commands. If either command fails to find the expected output (meaning no successful job log, or no alert log), the monitoring job itself should trigger a high-priority alert. This meta-alert is your true safeguard against silent failures.

The identity, secrets, and resources used by this monitoring job must be independent of the jobs it is monitoring. This prevents a failure in the monitored job's environment from also taking down the monitoring itself. Think of it as having a separate, independent watchman whose sole job is to check if the main guards are doing their job.

Broader Implications

The concept extends beyond simple scheduled reports. Any critical automated process—a data pipeline, a backup job, a security scan—is susceptible to silent failure. The principle remains the same: verify not only that the process ran, but that its intended outcome (success or failure notification) was achieved.

For founders and engineering leaders, this highlights the need for robust observability. It's not just about logging; it's about having systems that actively verify the health of other systems, especially those responsible for alerting. Relying solely on the absence of error messages is a dangerous gamble.

For developers, it's a reminder that the simplest tools can be powerful when applied correctly. `grep` on logs is a fundamental technique, but its application here is elevated to a critical monitoring function. It requires a deep understanding of how jobs log their activity and how alerting systems communicate.

The cost of a silent failure can be immense, ranging from lost revenue to critical data breaches. Implementing these simple, automated checks can prevent those scenarios, ensuring your automated systems are truly working for you, not against you.