Kubernetes Node Storage: 200Gi Too Much for Some EKS Workloads

The Storage Illusion in EKS

In a real-world production environment, a common pitfall is misinterpreting resource utilization metrics. This was the case in a recent analysis of an Amazon EKS cluster where nodes appeared to have an excess of storage capacity. Initially, the observation suggested an opportunity to optimize costs by downsizing worker nodes. However, a closer examination of the metrics revealed a more nuanced reality: the bottleneck wasn't disk space, but CPU. This article details the investigation and the subsequent simplification of capacity planning decisions.

The context for this investigation stemmed from a previous analysis (Engineering Build Notes #1) where an apparent opportunity to reduce worker-node capacity was explored. The premise was that the nodes were provisioned with more resources than the workloads actually consumed. This led to the hypothesis that downsizing the instances could yield cost savings. However, the data painted a different picture. One specific node was consistently operating at its limits, showing 98–100% CPU utilization. Meanwhile, its memory usage hovered around a mere 12–15%. This stark contrast clearly indicated that CPU was the limiting factor, not memory, and certainly not storage.

The initial assumption that 200Gi of storage per node was excessive proved to be a distraction. While it's true that many workloads on Kubernetes do not require vast amounts of local storage, especially when persistent volumes are managed externally, the focus on storage capacity masked the true performance constraint. In this specific EKS cluster, the workloads were CPU-intensive. The high CPU load meant that the node's ability to process tasks was severely hampered, regardless of how much free disk space was available. Allocating more storage than needed does not inherently improve performance; it simply increases cost without addressing the core performance issue.

Revisiting Capacity Planning

The realization that CPU was the bottleneck, not storage or even memory, necessitated a shift in strategy. Instead of attempting to downsize the nodes, the correct approach was to scale up the instance type. This meant selecting an EC2 instance that offered a higher CPU-to-memory ratio, or simply more raw CPU power, to accommodate the demanding workloads. This decision, driven by accurate metric analysis, led to a more stable and performant cluster. The illusion of over-provisioned storage was dispelled by focusing on the actual performance limiter.

This scenario highlights a critical aspect of cloud-native infrastructure management: the importance of granular and accurate monitoring. It's easy to get fixated on one metric, like storage, especially when it appears to be significantly underutilized. However, in a complex environment like Kubernetes, performance is often dictated by the interplay of multiple resources. For instance, a CPU-bound process might indirectly impact disk I/O performance due to scheduling delays or increased buffer activity. Similarly, insufficient memory can lead to excessive swapping, which drastically degrades disk performance.

The decision to increase node size, rather than merely reallocating or reducing storage, was validated by observing the post-change metrics. The CPU utilization dropped to more manageable levels, and the overall cluster responsiveness improved. This allowed the workloads to execute efficiently without hitting performance ceilings. The key takeaway is that while storage is a critical resource, its perceived abundance does not equate to optimal performance if other resources are constrained.

Simplifying Decisions with Accurate Metrics

The investigation ultimately led to a simpler capacity decision-making process. By understanding that the primary constraint was CPU, the team could confidently provision nodes that met the computational demands of the workloads. The 200Gi of storage, while still potentially more than some individual pods might directly access, became a non-issue. It was a fixed cost associated with the chosen instance type, and its 'excess' capacity did not negatively impact performance or incur additional direct costs beyond the instance pricing itself. The focus shifted from optimizing a perceived surplus (storage) to addressing the actual deficit (CPU).

This approach contrasts with scenarios where storage is indeed the bottleneck. For example, if a database workload or a stateful application were experiencing slow read/write operations, then investigating storage IOPS, throughput, and capacity would be paramount. In such cases, selecting EBS volumes with higher performance characteristics (e.g., `gp3` or `io2` instances) or ensuring sufficient local NVMe storage on EC2 instances would be the correct path. However, in the described EKS cluster, these considerations were secondary to the fundamental need for more processing power.

The series,