The Problem: Cache Stampedes and Database Connection Exhaustion

Many applications rely on caching to speed up database queries. Frequently accessed data, like apartment lists or aggregated numbers for listing pages, is stored in a cache. When this data is present in the cache, requests are served rapidly. However, the system faces a critical vulnerability the moment the cache expires.

Consider a scenario where a cache for a frequently accessed resource expires. Simultaneously, a surge of requests for that same resource arrives. If the caching logic is naive—checking the cache, finding it empty, then proceeding to query the database, compute the result, and populate the cache—each of these concurrent requests will independently perform the same heavy database operation. This is precisely what happened on podbor-minuta.ru, leading to a batch of 500 responses timing out. The database itself remained operational, but the queries to it began to queue up, rapidly draining the connection pool and eventually hitting the connection limit. The consequence is that subsequent requests, even those that could be served quickly if a connection were available, are left waiting and ultimately time out.

This phenomenon is known as a cache stampede, or a thundering herd problem. It occurs when an expiring cache entry triggers a flood of identical requests to the origin data source. For a single request, the logic of checking the cache, and if absent, fetching from the database and updating the cache, is sound. But under high concurrency, this simple approach becomes a systemic risk. When hundreds of requests hit an empty cache at the same moment, they all initiate the same expensive database query. The connection pool, a finite resource, is quickly depleted. As connections are exhausted, new requests cannot be fulfilled, leading to cascading failures and service degradation.

Diagram illustrating a cache stampede with multiple requests hitting an empty cache simultaneously

Introducing Single-Flight Caching

The solution to this cache stampede problem is a pattern called single-flight caching, also known as cache de-duplication or request coalescing. The core idea is to ensure that when multiple identical requests for a cacheable item arrive around the same time, only one of them is allowed to actually fetch the data from the origin (in this case, the database). The other requests are temporarily blocked, waiting for the first request to complete its data fetch and populate the cache. Once the cache is updated, all waiting requests can then be served from the cache.

Implementing single-flight caching involves a mechanism to track ongoing requests for specific cache keys. When a request arrives:

  • First, it checks if the data is already in the cache. If it is, the data is returned immediately.
  • If the data is not in the cache, the system checks if another request for the same key is already in progress.
  • If another request for this key is already in progress, the current request is put into a waiting state. It will be notified and receive the data once the in-progress request completes.
  • If no other request for this key is in progress, the current request is designated to fetch the data. It proceeds to query the database, compute the result, and crucially, update the cache.
  • Once the data is fetched and cached, the system notifies all requests that were waiting for this key. These waiting requests then retrieve the newly cached data and return it to their respective callers.

This approach effectively transforms a potential stampede of hundreds of database queries into a single, controlled query. The database connection pool is protected, preventing exhaustion and maintaining service stability even under heavy load.

Technical Implementation Considerations

Implementing single-flight caching requires careful consideration of several factors:

Concurrency Control: The mechanism for tracking in-progress requests and blocking/unblocking subsequent requests must be thread-safe and efficient. In many programming languages, this can be achieved using constructs like mutexes, semaphores, or specialized concurrency primitives. For distributed systems, distributed locks or coordination services like ZooKeeper or etcd might be necessary.

Cache Invalidation Strategy: While single-flight caching addresses the stampede issue, it doesn't inherently solve cache invalidation. The system still needs a robust strategy for determining when cached data becomes stale and needs to be refreshed. Time-based expiration (TTL) is common, but event-driven invalidation might be required for more dynamic data.

Timeout Management: The duration for which waiting requests are held must be carefully managed. If the primary data fetch takes too long, the waiting requests might time out before the cache is populated. This timeout should ideally be longer than the expected maximum time for a single data fetch and cache update, but not so long that it negatively impacts user experience.

Error Handling: What happens if the single request responsible for fetching data fails? The waiting requests need to be informed of the failure. Depending on the application's requirements, this might involve returning an error to all waiting clients, or retrying the fetch operation. Robust error handling is critical to prevent downstream failures.

Key Granularity: The effectiveness of single-flight caching depends on the granularity of the cache keys. If keys are too broad, legitimate differences in data requirements might be treated as identical, leading to incorrect data being served. If keys are too specific, the benefit of deduplication might be limited.

When to Use Single-Flight Caching

Single-flight caching is most beneficial in scenarios where:

  • Expensive Data Fetches: The underlying data retrieval operation is computationally intensive, time-consuming, or resource-heavy (e.g., complex database queries, external API calls, heavy computations).
  • High Concurrency on Cache Expiration: The application experiences significant traffic spikes, and it's probable that multiple requests for the same data will arrive shortly after a cache misses or expires.
  • Shared Cache: The cache is shared across multiple instances of an application or service, meaning a cache miss on one instance can be experienced by others concurrently.
  • Database Connection Limits: The database has a finite number of connections, and a cache stampede can easily exhaust these resources, leading to service outages.

By implementing single-flight caching, developers can significantly improve the resilience and scalability of their applications. It acts as a buffer, preventing a single point of contention—the database—from becoming overwhelmed by synchronized, cache-miss-driven requests. This pattern is akin to a busy restaurant manager handling multiple identical reservations that arrive simultaneously; instead of seating everyone at once and overwhelming the kitchen, they might ask some parties to wait a moment while they confirm availability and seat the first party, ensuring a smoother overall dining experience for everyone.