The Problem: Fleet Instability from Uncoordinated Autoscaling

An infrastructure team faced a critical issue: their fleet of stateful worker instances, managed by an AWS Auto Scaling Group (ASG), began exhibiting extreme instability. The fleet size would fluctuate dramatically, resizing every 30 to 90 seconds. This wasn't a matter of fine-tuning existing parameters; it was a fundamental problem stemming from how two distinct autoscaling policies interacted—or rather, failed to interact.

Initial Setup: CPU-Based Scaling

The fleet's architecture was standard for stateful workloads. It comprised a baseline of On-Demand instances for guaranteed capacity and Spot instances to handle bursts. Each instance was designed to serve a fixed pool of concurrent stateful sessions. The initial autoscaling strategy employed a TargetTrackingScaling policy based on the average CPU utilization across the ASG. The goal was simple: scale out when the fleet's overall CPU usage reached a predefined target, ensuring sufficient capacity to handle incoming requests.

This CPU-based scaling worked reasonably well until it didn't. The core limitation of CPU as a sole metric for stateful workloads became apparent. CPU utilization is a fleet-wide average. An individual instance could be at 100% capacity, rejecting new sessions, while the ASG's average CPU remained within acceptable limits if other instances were underutilized. This disconnect meant that even with seemingly healthy average CPU, the fleet could be overloaded at the instance level, leading to poor user experience and dropped sessions.

Introducing a Second Signal: Session Count

To address this gap, the team introduced a second autoscaling signal: the number of active sessions. This new metric promised to provide a more direct measure of the actual workload each instance was handling. The intention was to scale out when the total number of active sessions across the fleet approached a limit, ensuring that no single instance became a bottleneck.

This addition, however, was implemented without any coordination mechanism between the two policies. The ASG was configured with both the CPU-based TargetTrackingScaling policy and a new policy triggered by session count. Each policy operated independently, evaluating the fleet's state and issuing scaling commands based on its own set of conditions.

The Cascade Effect: Conflicting Signals, Rapid Resizing

The lack of coordination between the CPU and session count policies led to a dangerous feedback loop. Consider a scenario where the fleet is moderately busy. The CPU utilization might be hovering around the target, but not triggering a scale-out. Suddenly, a batch of new sessions arrives, pushing the session count above its threshold. The session count policy triggers a scale-out, adding new instances.

These new instances, initially with low CPU and session loads, begin to dilute the fleet-wide averages. The CPU utilization across the ASG drops. If this drop falls below the target threshold for the CPU policy, it triggers a scale-in event, removing instances. Now, the remaining instances, which might still be handling a high number of sessions, see their session counts rise again, potentially crossing the session count threshold once more, triggering another scale-out. This cycle repeats, causing the fleet to rapidly add and remove instances in a matter of seconds or minutes.

The core issue wasn't that either metric was inherently bad. CPU is a good indicator of processing load, and session count is a direct measure of active work for stateful services. The problem was that the ASG was receiving conflicting directives from two independent policies that didn't understand each other's actions. It was like having two navigators in a car, each with a different map and destination, constantly fighting over the steering wheel. The vehicle (the fleet) ends up swerving erratically instead of moving forward.

Why Tuning Alone Wouldn't Fix It

Many might assume this is a tuning problem—adjusting cooldown periods, step adjustments, or scaling metrics. However, the fundamental flaw here is the lack of policy coordination. Even with extended cooldown periods, the underlying conflict would eventually manifest. If the CPU policy scales in, and the session count policy immediately scales out again, the cooldown is effectively reset or bypassed.

AWS Auto Scaling Groups offer features like StepScaling and TargetTrackingScaling, but they are designed to react to specific metrics. When multiple TargetTrackingScaling policies are active, they can operate on different schedules and thresholds, leading to unpredictable interactions, especially with stateful workloads where instance state is critical. The ASG itself doesn't have a built-in mechanism to understand that a scale-in triggered by CPU might starve an instance that is critically important for session management, or that a scale-out triggered by session count might be immediately undone by a CPU dip.

The Path Forward: Coordinated Scaling Strategies

Resolving this requires a more sophisticated approach than simply adding more signals. Several strategies can mitigate this issue:

  • Single, Comprehensive Metric: Design a single metric that encapsulates both CPU and session load. This could involve a custom CloudWatch metric or a weighted combination of existing metrics.
  • Scheduled Scaling with Overrides: Use scheduled scaling for predictable baseline capacity and then allow a single reactive policy (e.g., based on session count) to handle deviations, with clear upper and lower bounds.
  • Step Scaling with Conditional Logic (Advanced): While complex, one could potentially use Step Scaling policies with CloudWatch Alarms that trigger Lambda functions to evaluate multiple conditions before issuing a scale-up or scale-down command. This allows for custom logic but significantly increases operational overhead.
  • External Orchestration: For highly complex stateful systems, consider using more advanced orchestration tools or custom logic outside the ASG that can make holistic decisions about fleet size based on a comprehensive understanding of the workload and instance state.

The key takeaway is that simply layering more autoscaling signals without considering their interdependencies, especially for stateful workloads, can lead to instability rather than improved performance. A coordinated strategy is essential.