The Problem: Cache Expiration and Concurrent Requests
At podbor-minuta.ru, the team faced a critical issue: their database, while healthy, became unresponsive under load. The symptom was a queue of requests hitting a connection limit, resulting in 500 errors for users. The root cause wasn't a database overload in the traditional sense, but a subtle flaw in their caching strategy. When a cached item expired, and multiple requests for that same item arrived simultaneously, each would independently attempt to fetch the data from the database. This 'cache stampede' or 'thundering herd' problem, as it's known, can quickly exhaust database connection pools, bringing the entire system to a halt.
Imagine a popular listing of apartments. The data is cached for speed. When that cache entry expires, and 200 users request that same listing within the same millisecond, each user's request sees an empty cache. Without a mechanism to coordinate, all 200 requests hit the database simultaneously. This is like 200 people trying to squeeze through a single doorway at once – chaos ensues. The database connection pool, typically designed for a manageable number of concurrent operations, gets completely depleted. Subsequent legitimate requests, even for unrelated data, find no available connections and time out, leading to the observed 500 errors.
The Naive Approach and Its Downfall
The initial logic was straightforward: check the cache. If data exists, serve it. If not, query the database, store the result in the cache, and then serve it. This works perfectly fine for low-traffic scenarios or when cache expiration is infrequent and requests are spread out. However, under high load, especially when a popular item's cache expires, this naive approach becomes a vulnerability. Every request that finds the cache empty initiates a separate, identical, and often resource-intensive database query. The system, intended to be sped up by caching, inadvertently creates a DDoS attack against its own database.
Introducing the Single-Flight Cache Pattern
The solution lies in a pattern often referred to as 'single-flight' or 'deduplication'. Instead of each concurrent request independently going to the database, the system needs a way to recognize that multiple requests are attempting to fetch the same missing data and ensure only one of them actually performs the database operation. The others wait for that single operation to complete and then receive the result.
This can be implemented using various synchronization primitives. In a multi-process or distributed system, this might involve a distributed lock manager (like Redis with SETNX or a dedicated locking service) or a message queue where only one worker picks up the 'fetch data' task. In a single-process application, a simple in-memory map combined with promises or futures can achieve the same effect. When the first request for a missing key arrives, it initiates the database fetch and stores a promise/future associated with that key. Subsequent requests for the same key, arriving before the first fetch completes, find the existing promise and simply wait for it to resolve. Once the database fetch is done, the result is cached, and all waiting requests are served from the cache (or directly from the result of the single fetch).
Implementation Details and Considerations
Implementing a single-flight cache requires careful consideration of several factors. The choice of synchronization mechanism depends heavily on the application's architecture. For a monolithic application, an in-memory lock or a concurrent map is usually sufficient. For distributed systems, a more robust solution like Redis or ZooKeeper is necessary to ensure that only one instance across all application nodes performs the fetch operation. The duration of the lock or the wait time for the promise must be managed to prevent indefinite blocking.
The time-to-live (TTL) for the cache entry becomes crucial. It needs to be long enough to provide performance benefits but short enough to reflect data freshness requirements. The 'single-flight' mechanism should ideally be tied to the cache's TTL. If a request arrives after the cache has been populated but before the TTL expires, it hits the cache directly. If it arrives after expiration but before a fetch is initiated, it triggers a new single flight. If it arrives while a flight is in progress, it waits.
Error handling is also paramount. What happens if the single database fetch fails? The promise/future should reject, and ideally, the system should log the error. The other waiting requests should also be notified of the failure. Depending on the desired resilience, the system might retry the fetch after a delay or return an error to the user. The critical aspect is that all waiting requests should receive a consistent outcome, whether success or failure, preventing multiple failed attempts from overwhelming the database.
Broader Implications for System Design
The single-flight pattern is not just about caching; it's a fundamental technique for managing concurrent access to shared, potentially expensive resources. It applies to scenarios beyond simple data retrieval, such as calling external APIs, performing complex computations, or even updating shared states where duplicate operations are undesirable or harmful. By deduplicating concurrent requests for the same resource, developers can build more resilient systems that gracefully handle bursts of traffic without sacrificing performance or stability. This pattern is a key tool in the arsenal for engineers building high-throughput, distributed applications, preventing a single hot key from becoming a system-wide Achilles' heel.