The Breaking Point: When Heartbeats Became a Deluge
The system, designed to track user connectivity via a heartbeat mechanism, faced an unprecedented surge in traffic in January 2026. For four weeks, the steady stream of heartbeats transformed into a terrifying deluge, overwhelming the architecture. This wasn't a gradual creep; it was a sudden, system-breaking event that exposed the hidden ceiling of the original design.
The architecture relied on a timer-based heartbeat sent by clients. Crucially, the heartbeat interval wasn't fixed on the client side. Instead, an API dictated the next call, defaulting to a 30-second interval. This seemingly minor detail became a critical factor when the system's capacity was tested. The core components of the original design included:
- A timer-driven heartbeat mechanism in the client.
- An Azure Event Hub to ingest these heartbeats.
- A worker service that processed Event Hub messages and wrote session data into Redis.
- Redis key expiration to drive the facility's online/offline status.
This setup had functioned reliably for two years. However, the surge in January 2026 pushed the Event Hub and the downstream worker services to their absolute limits. The sheer volume of incoming heartbeats overwhelmed the partition throughput of the Event Hub, and the worker instances struggled to keep up with the message processing and Redis writes. This bottleneck resulted in dropped heartbeats, inaccurate status reporting, and ultimately, a system that stopped working.
The problem wasn't a single point of failure, but a cascading overload. As the Event Hub partitions became saturated, message latency increased. The worker services, designed to scale based on incoming message volume, couldn't provision new instances fast enough to handle the peak load, especially if they were already struggling to process existing messages and maintain Redis connections. Redis itself, while fast, could also become a bottleneck under extreme write pressure from a massive number of worker instances trying to update session data concurrently.
The root cause was identified as insufficient Event Hub partitioning. The original configuration, adequate for normal operations, lacked the elasticity to absorb the sudden, massive spike in traffic. This directly impacted the throughput of the entire pipeline, from ingestion to status reporting.
The Fix: More Partitions, Smarter Scaling, and Automated Orchestration
The solution involved a multi-pronged approach, addressing the ingestion bottleneck, the processing scalability, and the orchestration of worker instances. The key changes were implementing more Event Hub partitions, migrating the worker services to Azure Container Apps (ACA), and leveraging the Kubernetes Event-Driven Autoscaling (KEDA) operator.
Event Hub Partitioning: Increasing Ingestion Throughput
The first and most direct fix was to increase the number of partitions in the Event Hub. Event Hubs partitions are the units of parallelism. By increasing the number of partitions, the system gains higher ingress and egress throughput. This allows more concurrent connections and higher message ingestion rates. The exact number of partitions required was determined through load testing and analysis of the peak traffic observed during 'The Storm'. A significantly higher partition count ensured that the Event Hub itself would not be the bottleneck during future traffic surges. This is akin to adding more lanes to a highway toll booth; it increases the number of cars that can pass through simultaneously.
Azure Container Apps (ACA): A Scalable Runtime for Workers
Migrating the worker services from their previous hosting environment to Azure Container Apps was a critical step for improving scalability and management. ACA provides a serverless container platform that automatically scales applications based on demand. Unlike traditional virtual machine-based scaling, ACA can scale down to zero and scale up rapidly in response to workload changes. This is essential for handling spiky traffic patterns like the one experienced. ACA also simplifies deployment and management, reducing operational overhead.
The worker services were refactored to be containerized. ACA's built-in scaling capabilities, when combined with KEDA, offered a potent combination for dynamic resource allocation. This move meant that the worker fleet could now grow and shrink automatically, ensuring that processing capacity always matched the incoming message load from the Event Hub.
KEDA: Event-Driven Autoscaling for Event Hub Consumers
The final piece of the puzzle was KEDA, which acted as the intelligent autoscaler for the ACA-hosted worker applications. KEDA integrates with various event sources, including Azure Event Hubs, and can trigger the scaling of Kubernetes deployments (or ACA environments, which are built on Kubernetes) based on the length of event queues or streams. For this use case, KEDA monitored the Event Hub consumer group's lag. When the lag (the number of messages not yet processed) exceeded a defined threshold, KEDA would instruct ACA to scale up the number of worker instances. Conversely, when the lag decreased, KEDA would scale the instances back down, even to zero if there were no messages to process.
KEDA's ability to directly read from Event Hub metrics made it the ideal orchestrator. It provided a reactive scaling mechanism that was precisely tuned to the Event Hub's throughput. This ensured that the worker services were always provisioned with just enough capacity to keep up with the event stream, preventing both overload and wasted resources. The configuration involved defining a KEDA scaler for Event Hub, specifying the target consumer group, and setting minimum and maximum replica counts for the ACA environment. This setup transformed the worker deployment from a static or manually scaled resource into a dynamically adjusting, event-driven system.
Lessons Learned and Future Considerations
The four-week outage served as a stark reminder that even well-architected systems can have hidden scaling ceilings. The decision to rely on API-driven heartbeat intervals, while offering flexibility, also meant that the client's behavior was indirectly tied to the system's perceived capacity. When the system became overloaded, the API would have likely continued to send the 30-second interval, exacerbating the problem rather than signaling a need for clients to back off.
The combination of increased Event Hub partitions, the elastic nature of Azure Container Apps, and KEDA's intelligent event-driven autoscaling proved to be a robust solution. This architecture now provides a much higher ceiling for user connectivity traffic and gracefully handles significant spikes. The system is no longer a fixed-capacity pipeline but a fluid, adaptive one.
A key takeaway is the importance of designing for scale from the outset, even for components that appear stable. Load testing that simulates extreme, albeit improbable, traffic spikes is crucial. Furthermore, understanding the interplay between different cloud services and scaling mechanisms is paramount. KEDA's ability to bridge the gap between event sources and container orchestration is a powerful pattern for building resilient, scalable applications.
What remains to be seen is the long-term cost implications of such a highly dynamic scaling setup. While it prevents outages, running potentially hundreds of container instances during peak events requires careful cost monitoring and optimization strategies. The trade-off between guaranteed availability and operational expenditure is a constant consideration in cloud-native architectures.