mtime Cache Invalidation Fails on Same-Second Writes

The Cache Invalidation Conundrum

Saving a source file, triggering a build, and then seeing the old output persist is a frustrating experience. It feels like a ghost in the machine, especially when the file on disk is demonstrably current, yet the cache refuses to acknowledge the update. This is precisely the scenario that ensnared a developer for two agonizing days, leading them to nearly suspect a haunted remote disk. The core problem: a cache key that stubbornly refused to notice a file modification that had clearly occurred.

The context was the development of a small rebuild helper. The goal was to enable an agent to skip unnecessary work if the output files were still considered fresh. The initial implementation involved a straightforward comparison of file modification times (`mtime`) using the `stat` system call. This approach, seemingly robust and logical, was employed on a second machine to cross-reference findings, as the developer's primary laptop had become an unreliable source of truth.

This article was prepared with the assistance of MonkeyCode's product outreach, utilizing their free model access for drafting the helper and their free server option for a clean testing environment. The timeline of the bug, as documented in the field notes, is crucial to understanding its nature.

Field Notes: The Unfolding Bug

The bug's timeline was the key to its unraveling. The developer meticulously documented their progress and observations, hour by hour, to capture the subtle sequence of events that led to the cache invalidation failure.

Initially, the suspicion fell on the remote disk. The behavior was so inexplicable that it bordered on the supernatural. However, repeated checks confirmed the file's modification time was indeed current. The discrepancy lay not in the file's actual state, but in how the caching mechanism interpreted that state. The `stat` comparison, a common method for checking file modification times, was expected to provide an accurate snapshot. Yet, it was failing to trigger the necessary cache invalidation.

The problem becomes acute when file modifications occur within the same second. Operating systems and file systems often report timestamps with second-level granularity. If a file is modified, and then a cache check occurs within that same second, the `mtime` might not have updated sufficiently to be detected as a change. This is particularly problematic in high-performance build systems or agents that rely on rapid file system monitoring. The cache, expecting a different `mtime` value to signal a change, continues to serve the stale data.

Consider a scenario where a build script modifies an output file and then immediately queries the cache to see if it needs to rebuild. If both operations—file write and cache query—happen within the same second, the `mtime` comparison could yield identical results, leading the cache to believe the file is unchanged. This creates a race condition where the system's state is updated, but the caching layer remains unaware, leading to stale results being served or unnecessary work being re-done.

The developer's journey involved ruling out various possibilities: network latency, disk issues, and even OS-level quirks. The breakthrough came when focusing on the exact timing of file system operations and the precision of the timestamps being compared. The realization dawned that the granularity of the `mtime` was insufficient for the rapid, same-second operations being performed.

The `stat` Comparison and Timestamp Granularity

The `stat` system call is fundamental for retrieving file metadata, including modification time. On many Unix-like systems, the `st_mtime` field represents the last modification time as a Unix timestamp, typically with second-level precision. When comparing two `st_mtime` values, if both operations—the original file write and the subsequent check—occur within the same second, the timestamps might be identical. This is not a bug in `stat` itself, but a limitation of its precision in a high-frequency environment.

Imagine the sequence: A file is written at 10:30:05.123 AM. The `stat` call to get its `mtime` might return `1678886405` (representing 10:30:05 AM). If the cache check and its own `stat` call happen fractions of a second later, before the clock ticks over to 10:30:06 AM, its `stat` call might also return `1678886405`. The comparison `1678886405 == 1678886405` evaluates to true, and the cache remains invalidated, serving the old content.

This is a classic example of a race condition exacerbated by insufficient timestamp precision. The system is performing as expected at the second level, but the application logic—the cache invalidation—relies on a level of detail that the underlying file system timestamps do not provide.