Log Order Inference Broke: How a Single Marker Fixed Live Site Processing Status

The Problem: Visualizing Sequential Site Maintenance

In systems that manage tasks across multiple distinct sites sequentially, providing immediate user feedback is crucial. Imagine a dashboard where each site processed by a backend maintenance script is visually represented. A common UI pattern involves a blue border for the site currently undergoing processing, and a green border for sites that have successfully completed. This provides real-time status without requiring manual refreshes or complex polling.

The straightforward approach to implementing this live status update is to monitor the streaming logs emitted by the backend. The assumption is that log lines, when ordered chronologically, directly correspond to the sequence of operations. By parsing these logs, one can infer which site is currently active, which has just finished, and which is next in line.

However, as one development team discovered, this seemingly simple inference mechanism proved surprisingly difficult to stabilize. It took three distinct iterations of fixes before the system reliably reflected the live status of site maintenance. This post details the journey through these challenges, serving as a design log of the trial-and-error process.

First Attempt: Simple State Transition on Log Arrival

The initial implementation was based on a direct correlation between log events and UI state changes. The backend emits log lines like [Site name] Starting maintenance or [Site name] Maintenance completed. The first design strategy was:

• When a Starting maintenance log arrives for a specific site, mark that site as 'running' (e.g., blue border).
• When a Maintenance completed log arrives for a specific site, mark that site as 'done' (e.g., green border).

This logic was implemented with the expectation that logs would arrive in a perfectly ordered, per-site sequence. The code would maintain a state for the currently running site. When a new 'start' log appeared, the previous 'running' site would be marked as 'completed' (if it hadn't already explicitly logged completion), and the new site would be marked as 'running'.

The Breakdown: Log Reordering and Latency

The core issue with the first design emerged when logs did not arrive in the strict chronological order assumed. Several factors can cause this:

• Network Latency: Logs from one site might experience higher network latency than logs from another, causing them to arrive out of sequence at the logging aggregation service or the UI.
• Backend Processing Skew: Even if the backend tasks are initiated sequentially, the time taken for each site's maintenance can vary. A faster site might complete its logging faster, but its completion logs could be delayed in transit, arriving after the next site has already started its logging.
• Log Aggregation Buffering: Logging systems themselves often buffer logs before shipping them. This buffering can introduce its own delays and reordering.

Consider this scenario: Site A starts, logs [Site A] Starting maintenance. The UI correctly marks Site A as blue. Then, Site B starts, and logs [Site B] Starting maintenance. The UI now marks Site B as blue and, based on the initial logic, incorrectly marks Site A as completed because it was the previous 'running' site. However, Site A's actual completion log, [Site A] Maintenance completed, might arrive moments later. The UI would then show Site A as completed *after* Site B has already started, creating a visual inconsistency.

This race condition meant that a site could be incorrectly marked as 'completed' simply because its completion log was delayed, while the next site's 'start' log arrived first. The UI would flicker, show incorrect states, and fail to provide a reliable real-time status.

Second Design Iteration: Explicit Completion Logging

To address the reordering problem, the team introduced a more explicit requirement: a site must log its own completion before it can be marked as done. The logic was refined:

• When a Starting maintenance log arrives for Site X, mark Site X as 'running'.
• Only when a Maintenance completed log arrives *specifically for Site X* should Site X be marked as 'completed'.

This seemed like a robust solution. The UI would no longer infer completion; it would wait for explicit confirmation from the backend for each site. The 'running' state would simply transition to 'completed' upon receiving the correct log line.

The Remaining Flaw: The 'Ghost' Running Site

While this second design fixed the issue of prematurely marking sites as completed, it introduced a new problem: the 'ghost' running site. If the backend encountered an error during maintenance for Site X, it might crash or fail to emit the Maintenance completed log. In such cases, Site X would remain indefinitely marked as 'running' in the UI, even though the backend process for it had long since failed or stalled.

The system lacked a mechanism to time out or reset the 'running' state if no further logs were received for an extended period, or if an explicit error log was emitted instead of a completion log. This meant the UI could show a site as actively being processed when it was, in reality, stuck or failed.

Third Design Iteration: The 'Current Site Marker'

The breakthrough came with the introduction of a single, definitive marker for the *currently* active site. Instead of relying on the sequence of logs to determine what *was* running and what *is* running, the system now uses a dedicated log event that unambiguously states which site is *currently* the focus of the backend process.

The backend was modified to emit a special log event, perhaps like [CURRENT_SITE] Processing: Site Y. This log acts as a heartbeat and an explicit declaration of the active site. The UI logic was rewritten:

• The UI listens for the special [CURRENT_SITE] Processing: Site Y log.
• Upon receiving this log, the UI immediately marks Site Y as 'running' (blue border).
• Crucially, any site that was *previously* marked as 'running' (blue) but is *not* Site Y, is now marked as 'completed' (green). This happens regardless of whether an explicit completion log was received. The assumption here is that if the backend has moved on to process Site Y, then Site X (the previous blue site) must have finished its work, whether successfully or not.
• If a site explicitly logs Maintenance completed, it is marked as 'completed' (green). This provides an override for explicit success signals.

This approach effectively decouples the 'running' state from explicit completion logs. The 'running' state is now solely determined by the most recent [CURRENT_SITE] log. All other sites that were previously marked 'running' are assumed to be finished. This handles the case where a site fails mid-process; it will be marked green (completed) when the next site starts processing, accurately reflecting that it is no longer the *currently* active task.

This third design stabilized the system. It provides a reliable real-time status by using a single, authoritative log event to denote the active site, and by treating any prior 'running' site as implicitly completed once a new active site is declared. The explicit completion logs serve as a confirmation, but the 'current site marker' is the primary driver for state transitions away from 'running'.