Rekuiper 0.500: Pushing Performance Past 200k Messages Per Second
Rekuiper, the Rust reimplementation of the LF Edge eKuiper project, has achieved a significant performance leap in its latest release, version 0.500. The core achievement is moving the engine's hot path – the most frequently executed code – entirely into RAM. This strategic shift bypasses disk I/O bottlenecks, enabling rekuiper to sustain throughput of up to 200,000 messages per second (msg/s) on a single CPU core. This represents a doubling of its previously benchmarked stable performance.
Previous benchmarks for rekuiper demonstrated its memory efficiency, with memory usage staying between 5 and 10 MB compared to Go-based engines that consumed hundreds of megabytes or even failed under load. At 100,000 msg/s, rekuiper was stable. However, when offered 200,000 msg/s, it began dropping packets or backlogging upstream, leaving a critical performance gap. The exact ceiling was unknown, and the reasons for choking when the CPU wasn't fully maxed out remained a mystery.
Detailed profiling under burst loads revealed the unexpected culprit: disk I/O. Not stream parsing or window calculations, but the physical storage was the primary limiter. This insight directly informed the development strategy for rekuiper 0.500.
Eliminating Disk I/O: The RAM-Centric Approach
The fundamental change in rekuiper 0.500 involves re-architecting the engine's internal data handling. Instead of relying on disk for intermediate storage or logging critical paths, all high-frequency operations now occur in volatile memory. This includes message buffering, processing queues, and internal state management for stream processing logic.
This approach treats the engine's core processing loop as a temporary, in-memory workspace. Data enters the system, undergoes transformations and analysis within RAM, and then exits to its destination. This eliminates the latency inherent in disk read/write operations, which can become a significant bottleneck, especially under high-throughput conditions. Disk I/O latency, even on fast SSDs, is orders of magnitude higher than RAM access times. For a system processing tens of thousands of messages per second, even microsecond-level delays on disk operations can cascade into dropped packets and reduced throughput.
Think of it like a chef preparing a complex meal. Before, the chef had to fetch ingredients from a pantry (disk) for every single step, slowing down the entire process. Now, the chef keeps all necessary ingredients and tools on a large, well-organized countertop (RAM), allowing for rapid, uninterrupted preparation.
Finding the Exact Physical Limits: Benchmarking Methodology
To validate the impact of the RAM-centric architecture and determine the new physical limits, the rekuiper team conducted rigorous benchmarking. The tests focused on identifying the precise throughput ceiling under various real-world MQTT workload conditions. The goal was not just to achieve a high number, but to understand the precise point where the system saturates.
The benchmark setup involved a single rekuiper instance running on a single, pinned CPU core. This isolation is crucial for accurately measuring the engine's raw processing capability without interference from other processes or the complexities of multi-core scheduling. Messages were injected at increasing rates, and the system's response was meticulously monitored. Key metrics included messages processed per second, packet loss rate, upstream backlog, and CPU/memory utilization.
The results confirmed that rekuiper 0.500 can sustain 200,000 msg/s with minimal packet loss and no upstream backlog, provided the necessary CPU resources are available. The previous choke point – disk I/O – has been effectively removed from the critical path. While CPU utilization will naturally increase at these higher rates, it no longer hits an artificial wall imposed by storage speed.
Implications for Edge Gateways and IoT
The performance gains in rekuiper 0.500 have substantial implications for edge computing scenarios, particularly for IoT hubs and gateways. These environments often deal with massive volumes of data from numerous sensors and devices, requiring stream processing engines that are both performant and resource-efficient.
By eliminating disk I/O from its hot path and achieving high throughput with low memory footprint, rekuiper is positioned as a strong candidate for deployment on resource-constrained edge devices. The ability to process data locally at high speeds reduces latency, enables real-time decision-making, and can decrease reliance on cloud connectivity for certain tasks.
For developers building IoT solutions, this means they can deploy more sophisticated analytics and processing logic directly at the edge, without being limited by the throughput of their stream processing middleware. It also suggests that other stream processing engines that still rely on disk-bound operations for their hot paths may have significant untapped performance potential waiting to be unlocked through similar memory-centric optimizations.
Future Directions and Unanswered Questions
While the 200k msg/s mark on a single core is a significant achievement, it raises further questions. What happens when multiple cores are utilized? How does rekuiper scale across a cluster of edge devices? And what are the implications for fault tolerance and persistence when the core processing is entirely in RAM? Ensuring data durability and recovery in the event of a power failure or system crash without disk-based logging in the hot path will be a key area for future development and validation.
The team has effectively found the physical limits of the engine's hot path by moving it to RAM. The next frontier is to understand how this high-performance, memory-bound engine behaves under more complex, distributed, and persistent workloads, pushing the boundaries of what's possible at the edge.