Kubernetes Probes: Readiness vs Liveness Explained for Production

Kubernetes Probes: Failure Detectors with Different Effects

Kubernetes probes act as distributed-systems failure detectors, but their control effects differ significantly. A readiness probe failure tells Kubernetes that a Pod is not yet ready to serve traffic, leading to its removal from Service endpoints. Conversely, a liveness probe failure signals to the kubelet that the container is in an unrecoverable state and should be restarted. This distinction is crucial because probe detection is inherently imperfect. Aggressive thresholds can reduce detection latency, but they also increase the risk of false positives during transient load spikes or temporary network issues.

The core difference lies in their purpose: readiness probes manage traffic flow, while liveness probes manage container health and lifecycle.

Readiness: Protecting Traffic Flow

A readiness probe answers the question: “Can this replica safely accept new work right now?” This assessment should consider the Pod’s ability to handle incoming requests, which might involve checking local state and the health of indispensable downstream dependencies. However, indiscriminately probing every single dependency can lead to cascading failures. Imagine a scenario where a single slow database causes one replica to become unready. If all replicas are configured to depend on this same database for their readiness check, all Pods could be marked as unready simultaneously. This eliminates all available capacity precisely when the system might need graceful degradation the most, exacerbating the problem.

A well-designed readiness probe ensures that traffic is only directed to Pods that are fully capable of processing requests. This includes ensuring that background initializations (like loading large datasets into memory, establishing database connections, or waiting for configuration updates) have completed. If a Pod is undergoing such an initialization or is experiencing temporary issues that prevent it from serving traffic correctly, its readiness probe should fail. Kubernetes will then stop sending new traffic to that Pod until the probe starts succeeding again. This is vital for zero-downtime deployments and for maintaining service availability during planned maintenance or unexpected degradations.

Liveness: Repairing Irrecoverable States

A liveness probe answers a narrower question: “Is the process irrecoverably stuck such that it cannot recover on its own?” If a liveness probe fails, Kubernetes does not try to route traffic away from the Pod; instead, it takes a more drastic action: it restarts the container. This is intended for situations where an application process has entered a deadlock, is consuming excessive resources due to a bug, or is otherwise unresponsive and beyond the ability of the application itself to recover.

Common scenarios for liveness probe failures include:

• An application thread is permanently blocked waiting for a resource that will never become available.
• A memory leak has consumed all available memory, causing the application to crash or become unresponsive.
• A critical background process within the application has stopped functioning, rendering the entire application useless.
• The application is stuck in a restart loop or a fatal error state.

It is critical to understand that liveness probes should not be used to detect transient issues. If a liveness probe fails due to a temporary network glitch or a brief spike in CPU load, the container will be restarted unnecessarily, potentially leading to service disruption. This is why the distinction between readiness and liveness is so important. Readiness is about traffic, liveness is about restarting a broken process.

Configuring Probes Effectively

Kubernetes offers three main types of probes:

• Exec Probes: Execute a command inside the container. A non-zero exit code indicates failure.
• HTTP Probes: Send an HTTP GET request to a specified path. A non-2xx or non-3xx status code indicates failure.
• TCP Probes: Attempt to open a TCP connection to a specified port. A failure to establish a connection indicates failure.

When configuring probes, several parameters are key:

• `initialDelaySeconds`: The number of seconds after the container has started before liveness or readiness probes are initiated. This is crucial to give applications time to start up without triggering false failures.
• `periodSeconds`: How often (in seconds) to perform the probe.
• `timeoutSeconds`: How long (in seconds) to wait for the probe to complete. If the probe fails to return within this time, it is considered a failure.
• `successThreshold`: Minimum consecutive successes for the probe to be considered successful after having failed. Defaults to 1.
• `failureThreshold`: Minimum consecutive failures for the probe to be considered failed after having succeeded. Defaults to 3. This value determines how quickly Kubernetes reacts to a persistent failure.

The Cascading Risk of Misconfiguration

The interplay between these probes and Kubernetes’s control plane can lead to unexpected behaviors if not configured thoughtfully. For instance, a readiness probe that is too sensitive to transient network latency could cause a Pod to be repeatedly removed from Service endpoints. If this happens frequently, the Service might end up with very few, or even zero, healthy Pods available to handle traffic. This is essentially a denial-of-service condition caused by misconfigured probes, not by an actual application failure.

The surprising detail here is not the complexity of the probes themselves, but how easily they can inadvertently cause outages if tuned too aggressively or if their failure modes are not fully understood. Aggressively low `failureThreshold` or `timeoutSeconds` values, especially on readiness probes, can lead to a Pod being taken out of service for minor, temporary network blips. Conversely, if liveness probes are too lenient, a truly stuck application might run for a long time, consuming resources and failing to serve requests, without being automatically restarted.