Designing for Resilient Realtime Delivery
Scaling realtime event delivery for applications like e-commerce delivery tracking maps demands a robust architecture. The core challenge is ensuring that every user, even those with intermittent connectivity, receives accurate, up-to-date information. This isn't just about pushing data; it's about guaranteeing compatibility and reliability. The key lies in treating each browser connection not as a permanent pipeline, but as a transient projection of a stable, ordered log of events specific to each delivery. This approach ensures that a shopper who loses connection in a tunnel can reconnect on a different network and seamlessly pick up where they left off, even if they also joined a support chat for that delivery.
The ultimate constraint is presence accuracy. While an online indicator is useful, its expiry rules must be clearly understood. However, the delivery state itself must remain correct regardless of the online indicator's status. This means the underlying data source of truth must be immutable and ordered, capable of replaying events to any connected client.
The Durable Log and Disposable Projection Model
The recommended architecture hinges on two main concepts: a durable, ordered log per delivery and treating browser connections as disposable projections of that log. This contrasts with the common approach of relying on long-lived connections to guarantee event delivery exactly once.
For release compatibility, versioned envelopes are crucial. Each event should carry metadata indicating its version, allowing clients to gracefully handle changes. Resume cursors enable clients to specify how far into the log they have processed events, facilitating seamless resumption after disconnections. An explicit resync path ensures that clients can always fetch a complete, consistent state if the cursor mechanism fails or is insufficient.
Scaling is achieved through partitioning the event stream by delivery ID. This distributes the load across multiple processing units. Updates are then coalesced at the edge, meaning that multiple location updates for the same delivery might be batched and sent as a single, more efficient message to connected clients. This prevents overwhelming individual connections with a flood of granular updates.
Key Components for Reliability
Several architectural components are essential for this model to function effectively:
- Versioned Envelopes: Each event is wrapped in a structured envelope that includes a version number. This allows the system to evolve without breaking existing clients. For example, a new version might include additional telemetry data, while older clients would simply ignore it.
- Resume Cursors: Clients maintain a cursor indicating the last processed event ID or timestamp for a given delivery log. When reconnecting, they provide this cursor to the server, which then streams only the new events from that point onwards. This is analogous to bookmarking a specific point in a video stream.
- Explicit Resync Path: In scenarios where cursors are lost or unreliable (e.g., after a very long disconnection or client-side data corruption), an explicit resync endpoint allows the client to request the entire current state for a delivery. This might involve fetching the last known location, estimated time of arrival, and any critical status updates directly from a dedicated API, bypassing the event stream for a full refresh.
- Partitioning by Delivery ID: The core scaling mechanism. Instead of a single monolithic event stream, events are sharded based on the unique identifier for each delivery. This allows for horizontal scaling of the event processing and storage infrastructure. Each partition can be handled by a separate set of consumers and producers.
- Edge Coalescing: At the network edge, close to the clients, updates for a specific delivery can be aggregated. If a delivery has multiple location updates within a short time frame (e.g., seconds apart), the system can consolidate these into a single, more efficient message sent to the map clients. This reduces network traffic and client-side processing load. Think of it like a traffic reporter giving a summary of congestion on a highway stretch, rather than reporting every single car's movement.
Presence and State Management
Presence information – whether a user is actively viewing a map or in a support chat – can inform capacity planning and fan-out strategies. For instance, if analytics show a surge in users opening maps for a particular delivery, the system can proactively scale up resources for that delivery's partitions. However, presence must never dictate the existence of a location update.
The delivery's state is the immutable truth. A user's presence is a secondary signal. If a user is offline, their map connection is lost. But the delivery itself continues to generate location updates. When the user reconnects, the system must be able to deliver those missed updates, ensuring the map reflects the true, current state of the delivery, not just the state when the user was last online.
This architecture ensures that even with 10,000 concurrent, reconnecting maps, the system remains stable and delivers accurate, real-time information. It prioritizes the integrity of the delivery state over the ephemeral nature of individual client connections.