Understanding Redis Sentinel Failover Time

When a primary Redis instance goes offline, Redis Sentinel steps in to promote a replica and ensure application availability. But how long does this critical process actually take? A recent series of tests aimed to quantify this downtime, measuring the exact window during which writes to the database fail. The findings reveal a surprisingly consistent outage duration, largely dictated by Sentinel's configuration and a fixed election/promotion overhead.

The experiments involved intentionally terminating a primary Redis instance and timing the interval until a replica could successfully accept writes. Across multiple runs, various down-after-milliseconds settings (1000ms, 2000ms, 3000ms), and different in-memory database engines—Redis 8, Valkey 8, and Dragonfly—the results converged. The core takeaway is that the write outage is approximately the configured down-after-milliseconds value plus an additional two seconds. This two-second buffer accounts for Sentinel's detection, leader election among Sentinels, and the promotion of a new primary.

Table showing Redis Sentinel failover outage times for different down-after-milliseconds settings

The Experiment Setup and Findings

The test environment comprised a standard setup: one primary Redis instance, two replicas, and three Sentinel instances, all running within Docker containers. The primary instance was deliberately killed to trigger a failover. The measurement captured the gap between the primary's termination and the first successful write operation on the newly promoted replica.

Across three distinct down-after-milliseconds configurations—1000ms (1 second), 2000ms (2 seconds), and 3000ms (3 seconds)—the mean write outage times were recorded. For a down-after-milliseconds of 1000ms, the average write outage was 2.5 seconds. This held true even when the duration was increased. With down-after-milliseconds set to 2000ms, the outage averaged 3.5 seconds. Similarly, at 3000ms, the outage clocked in at 4.5 seconds on average. These figures consistently demonstrate that the total failover time is the sum of the down-after-milliseconds setting and approximately two seconds.

The engine used—Redis 8, Valkey 8, or Dragonfly—did not significantly alter this failover duration. This suggests that the bottleneck is primarily within the Sentinel consensus and promotion mechanism rather than the speed of the underlying in-memory data store itself in handling the failover transition.

Understanding the Failover Stages

A Redis Sentinel failover is a multi-stage process, and understanding each step helps explain the observed timings:

  • Primary Failure Detection: Sentinels periodically ping the primary. If a primary fails to respond within the configured down-after-milliseconds interval, it is marked as potentially down.
  • Sentinel Leader Election: Once a majority of Sentinels agree the primary is down, they initiate a leader election process to decide which Sentinel will manage the failover.
  • Replica Selection: The elected leader Sentinel identifies the best replica to promote. This involves checking replica health, replication lag, and other factors.
  • Replica Promotion: The leader Sentinel sends a REPLICAOF NO ONE command to the chosen replica, making it the new primary.
  • Configuration Update: The leader Sentinel then informs all other Sentinels and connected clients about the new primary. Clients must reconfigure themselves to point to the new primary.

The observed two-second overhead is largely consumed by the Sentinel leader election and the time it takes for the chosen replica to acknowledge its new role and for Sentinels to confirm this state.

Production Gotchas and Considerations

While the lab results provide a clear baseline, production environments introduce complexities that can extend failover times or complicate the process:

  • Network Latency: Increased network latency between Sentinels and between Sentinels and replicas can delay failure detection and leader election. High latency can also impact the speed at which the new primary's status is propagated.
  • Resource Contention: If Sentinels or replicas are running on overloaded hosts, their ability to respond promptly to pings or to execute failover commands can be severely degraded. This can lead to Sentinels timing out and marking instances as down prematurely, or replicas being too slow to be promoted effectively.
  • Replication Lag: A replica that has fallen significantly behind the primary might not be a suitable candidate for promotion. The Sentinel leader will wait for a sufficiently up-to-date replica, potentially increasing the perceived failover time if all replicas are lagging.
  • Sentinel Quorum: Sentinel requires a majority (quorum) to agree on a failover. If network partitions or node failures prevent a quorum from forming, failover can be delayed or may not occur until the quorum is re-established.
  • Client Reconnection: The time it takes for client applications to detect the primary failure and reconnect to the new primary is not included in the Sentinel failover time itself but is critical for overall application availability. Ensure clients are configured with appropriate connection timeouts and retry mechanisms.

The surprising detail here is not the consistency of the failover time in a controlled environment, but how easily production factors can disrupt this predictable window. For instance, a network blip that causes intermittent packet loss could trigger multiple Sentinel timeouts, leading to a cascade of unnecessary leader elections and delays.

Optimizing Sentinel Configuration

To minimize downtime, tuning Sentinel's configuration is crucial. The down-after-milliseconds setting is the most direct lever. A lower value means faster detection but increases the risk of false positives during transient network issues. A higher value reduces false positives but extends the actual downtime when a real failure occurs. Finding the right balance depends on the stability of your network and the tolerance for downtime.

Other relevant Sentinel settings include:

  • failover-timeout: The maximum time Sentinel will wait for a failover to complete.
  • parallel-syncs: The number of replicas that can be reconfigured to sync with the new primary simultaneously.

If you run a high-availability Redis setup, understanding these timings and potential pitfalls is essential for robust architecture. The measured 2.5-second outage is a best-case scenario; production deployments must account for network variability and resource availability.