CoreDNS Bug: One Dead Pod Caused API Gateway Outages

The Phantom Connection

Imagine calling a friend, but instead of their phone ringing, it just sits there, silently holding the line open. You can't make other calls, and you can't hang up. That's the essence of the bizarre bug that plagued our API gateway, leading to intermittent "Upstream Service Unavailable" errors for approximately one-third of all incoming requests. The frustrating part? The issue would vanish after a gateway restart, only to reappear hours or days later, with no discernible changes in code, deploys, or traffic patterns. The backends themselves were consistently healthy.

The culprit was deceptively simple: a single, dead CoreDNS pod in our Kubernetes cluster. This pod, no longer actively serving requests, still held open a network socket. This open, unresponsive socket created a silent black hole for any traffic attempting to resolve hostnames through it. When the API gateway needed to connect to a backend service, it would query DNS. If it happened to hit this specific dead pod, the DNS resolution would hang indefinitely, eventually timing out and resulting in the "Upstream Service Unavailable" error. The gateway wasn't truly unable to reach the upstream service; it was unable to *resolve* the service's address because the DNS query was getting stuck.

The CoreDNS Black Hole Explained

CoreDNS is the default DNS server for Kubernetes. It's designed to be highly available and performant, resolving service names within the cluster. When a pod dies in Kubernetes, its IP address is typically removed from service discovery mechanisms. However, this bug revealed a subtle failure mode: a lingering, open network connection to a non-existent endpoint. This is akin to leaving a phone line perpetually off the hook. Any subsequent call attempting to use that line would be met with silence, never reaching its intended destination.

When the API gateway initiated a request that required DNS resolution, it would query the CoreDNS service. If the load balancer or internal routing directed the query to the specific pod that had died but still held a socket open, the request would stall. The gateway, expecting a swift DNS response, would instead wait for an unresponsive socket. This hang would eventually trigger a timeout, but not before consuming resources and preventing the gateway from attempting other resolutions or forwarding the original request. The gateway itself remained operational, but its ability to resolve backend service addresses was intermittently crippled by this single point of failure in the DNS layer.

The intermittent nature of the problem stemmed from the ephemeral nature of pod scheduling and network routing. A dead pod might remain in a terminating state for a period, and network connections could persist longer than expected. The restart of the API gateway would clear its connection pool and DNS cache, allowing it to establish new, healthy connections. However, without addressing the root cause – the lingering open socket on the dead CoreDNS pod – the issue was bound to recur as soon as the gateway's internal state expired or new connections were established that hit the problematic DNS entry.

Identifying the Culprit

The debugging process was arduous precisely because the symptoms pointed away from the actual cause. Backend logs showed no errors, application performance monitoring indicated healthy upstream services, and the API gateway's own metrics only showed the downstream error, not the upstream resolution failure. The key was realizing that the error message, "Upstream Service Unavailable," was a misnomer. The service was available; the gateway simply couldn't find it due to a DNS resolution hang.

The breakthrough came from correlating the gateway restarts with the problem's disappearance. This suggested an issue with the gateway's state or its external dependencies. By meticulously examining network traffic and DNS query logs during an outage, the team could pinpoint the stalled DNS resolution. Further investigation into the CoreDNS pods revealed one pod in a `Terminating` state with an unusually high number of open connections, some of which were directed at non-existent cluster IPs. This was the smoking gun.

The problem wasn't a failure of CoreDNS to respond, but a failure of a specific instance to gracefully release its network resources upon termination. This is a critical distinction. It highlighted a potential vulnerability in how Kubernetes handles network connections for pods in abnormal states, especially when those pods are critical infrastructure components like DNS resolvers.

Mitigation and Prevention

The immediate solution involved manually terminating the offending CoreDNS pod, forcing the release of its network sockets. This resolved the immediate outage. However, to prevent recurrence, several measures were implemented:

• Increased CoreDNS Replicas: Ensuring a higher number of CoreDNS replicas provides better fault tolerance. Even if one pod fails, others can pick up the load without a single point of failure causing resolution hangs.
• Proactive Health Checks: Implementing more aggressive and comprehensive health checks for CoreDNS pods. These checks should not only verify that the DNS server is running but also that it can successfully resolve common internal and external hostnames.
• Network Policy Review: Examining Kubernetes Network Policies to ensure they don't inadvertently block necessary inter-pod communication for DNS or other essential services during pod termination or rescheduling.
• Resource Limits and Requests: Ensuring appropriate resource limits and requests are set for CoreDNS pods to prevent them from becoming resource-starved and entering unstable states.
• Kubernetes Version and CoreDNS Version Updates: Staying up-to-date with the latest stable versions of Kubernetes and CoreDNS, as these issues are often addressed through patches and improvements in newer releases.

This incident serves as a potent reminder that in distributed systems, failures are not always loud and obvious. Sometimes, the most disruptive issues stem from silent, lingering problems in seemingly innocuous components like DNS. The CoreDNS black hole highlights the intricate dependencies within microservice architectures and the importance of robust error handling and deep visibility into network communication, even when services appear to be functioning nominally.

What remains unaddressed is the precise mechanism within the Linux kernel or CoreDNS that allows a network socket to remain in an `ESTABLISHED` state while pointing to an IP address that is no longer routable or associated with an active process. Understanding this specific kernel behavior could lead to more robust network stack designs across all distributed systems.