The Illusion of Idle Capacity
In cloud infrastructure, especially with managed Kubernetes services like Amazon EKS, it's common to assume that underutilized worker nodes mean excess capacity. The natural inclination is to right-size instances, reducing costs. However, a closer look at metrics in a production EKS cluster revealed a counterintuitive problem: one node was pegged at 98-100% CPU utilization, while its memory usage hovered around a mere 12-15%. This clearly indicated that CPU, not memory, was the performance bottleneck. The cluster, despite appearing to have plenty of available resources on paper, was unable to consolidate workloads onto fewer, smaller nodes due to a specific Kubernetes scheduling constraint: resource requests.
The Engineering Build Notes series aims to document real-world engineering decisions, platform improvements, and infrastructure trade-offs. This particular situation, detailed in Engineering Build Notes #2, highlights how seemingly simple cost optimization efforts can become complex when underlying scheduling dynamics are not fully understood. The apparent abundance of resources was a mirage, masking a critical CPU saturation issue that prevented efficient node consolidation.
Understanding Kubernetes Scheduling and Resource Requests
Kubernetes uses a scheduler to determine which node a new pod should run on. This scheduler relies heavily on the CPU and memory requests defined in a pod's specification. These requests are not just hints; they are hard reservations. When a pod is scheduled, the Kubernetes scheduler checks if a node has enough allocatable CPU and memory to meet the pod's requested resources. If a node's available resources fall below a pod's request, that pod will not be scheduled on that node, even if the node has plenty of *available* CPU or memory that is not currently allocated.
In this EKS cluster, the issue stemmed from the cumulative CPU requests of the pods running on the heavily utilized node. While the node had ample free memory that could have theoretically accommodated more pods or larger workloads, the CPU requests of existing and potential new pods were already maxed out. This meant that even though the node wasn't actively *using* all its CPU, it couldn't accept new pods that requested more CPU than was considered 'allocatable' by Kubernetes, after accounting for the existing requests. This is akin to a restaurant with many empty tables but no single table large enough to seat a new party of eight; the individual seats (memory) might be available, but the configuration (CPU requests) prevents the party from being seated.
The Impact of High CPU Requests
The consequence of these high CPU requests was that the Kubernetes scheduler perceived the node as 'full' from a CPU perspective, even if its actual CPU usage was only moderately high. This prevented Kubernetes from consolidating workloads onto this node or other similarly configured nodes. Instead of allowing the scheduler to place pods onto nodes based on actual, real-time utilization, the rigid adherence to predefined resource requests created artificial scarcity. This inefficiency led to more worker nodes being provisioned than were strictly necessary, driving up AWS costs.
The problem wasn't that the nodes were undersized for their *actual* workload; it was that their CPU requests were set too high, preventing efficient bin-packing of pods. This situation is particularly tricky because CPU is a non-compressible resource. Unlike memory, where Kubernetes can reclaim unused memory pages, CPU is used moment-to-moment. If a pod requests 1 CPU core, Kubernetes ensures that 1 CPU core is available for it, even if the pod is only actively consuming 0.1 CPU at that instant. This over-reservation of CPU capacity is a common pitfall that leads to underutilization and increased costs.
Investigating and Resolving the Bottleneck
The initial assumption that larger EC2 instances meant higher costs (as explored in the previous build notes) was correct, but the inverse wasn't necessarily true. Simply assuming smaller instances would automatically reduce costs ignored the Kubernetes scheduling layer. To resolve this, the engineering team needed to adjust the CPU requests of the pods. This involved a careful analysis of actual pod CPU consumption, not just their declared requests.
Tools like `kubectl top pods` and cluster monitoring solutions (e.g., Prometheus, CloudWatch Container Insights) are crucial here. By observing the actual CPU usage of pods over time, developers can determine more accurate and realistic CPU requests. Setting these requests closer to the observed average or peak usage (with a reasonable buffer) allows Kubernetes to schedule pods more effectively. This process is iterative: set new requests, observe the impact on scheduling and utilization, and adjust as needed.
The key takeaway is that optimizing Kubernetes costs requires looking beyond raw EC2 instance metrics and diving into the specifics of pod resource requests and limits. It's about tuning the Kubernetes scheduler's perception of resource availability to match the reality of application behavior. This is not a one-time fix; as applications evolve and workloads change, resource requests need to be re-evaluated to maintain optimal cluster efficiency and cost-effectiveness.
Broader Implications for Kubernetes Cost Management
This scenario underscores a fundamental challenge in managing Kubernetes at scale: the gap between declared resource requests and actual resource consumption. Developers often err on the side of caution when setting requests, leading to over-provisioning. This is compounded by the fact that, by default, Kubernetes doesn't aggressively enforce CPU limits, but it *does* use requests for scheduling. The result is a cluster that appears to have plenty of capacity but is inefficiently packed, leading to higher cloud bills.
For organizations running EKS or other Kubernetes distributions, a proactive approach to resource request optimization is essential. This involves establishing continuous monitoring of pod resource utilization, implementing policies that encourage accurate request setting, and periodically auditing resource configurations. The goal is to achieve a state where the Kubernetes scheduler can effectively pack workloads, minimizing the number of nodes required and thus reducing infrastructure costs. It requires a shift in mindset from simply