The Inevitable Duplicate Event in Kafka

It's 2 a.m., and your on-call phone buzzes. A customer was charged twice for the same order. You dig into the logs. The OrderCreated event was processed, then processed again seconds later. Nothing crashed. No exception was thrown. Kafka did exactly what it promised: it delivered the message at least once. The bug isn't in Kafka. It's an assumption that your consumer would ever see each event only once.

Kafka's core guarantee is at-least-once delivery. This means that a message sent to a Kafka topic will be delivered to a consumer at a minimum of one time. However, due to network issues, consumer crashes, or broker restarts, a message might be delivered more than once. The 'least' is doing a lot of work, and it's the root of the problem: duplicates are not an edge case; they are a normal part of Kafka's operation.

The common consumer flow looks like this:

  1. The Kafka broker sends a message to the consumer.
  2. The consumer receives the message.
  3. The consumer processes the message (e.g., charges the customer).
  4. The consumer commits the offset to Kafka, signaling that the message has been successfully processed.

Now, consider what happens if the consumer crashes after processing the message but before committing the offset. The broker, not having received the acknowledgment, will assume the message was not delivered and will resend it. The consumer, upon restarting, will re-process the same message. This is the birth of a duplicate event in your application.

The surprising detail here is not that duplicates happen, but that many developers build systems assuming they don't. This assumption leads to silent bugs that only surface under load or during transient failures, manifesting as issues like double charges, duplicate data entries, or incorrect state updates.

Diagram illustrating Kafka's at-least-once delivery mechanism and potential duplicate processing

Making Your Consumer Idempotent

The solution is to make your consumer idempotent. An idempotent operation is one that can be applied multiple times without changing the result beyond the initial application. Think of it like clicking a 'save' button: clicking it once saves your work. Clicking it a second, third, or tenth time has the same effect as the first click – your work is saved, and the state doesn't change further.

For Kafka consumers, idempotency means that processing the same message multiple times results in the same outcome as processing it once. This typically involves adding logic to your consumer to detect and ignore duplicate messages.

A common strategy involves using a unique identifier for each event and storing processed event IDs. When a message arrives:

  1. Extract a unique identifier from the message payload (e.g., an event ID, a transaction ID, or a combination of fields that uniquely identifies the operation).
  2. Check if this identifier has already been processed. This check usually involves querying a fast, in-memory data store (like Redis or a local cache) or a dedicated table in your database.
  3. If the identifier has already been processed, skip the message.
  4. If the identifier has not been processed, proceed with the business logic.
  5. After successful processing, store the identifier in your processed list and then commit the Kafka offset.

This approach ensures that even if a message is redelivered, the consumer will recognize the already-processed identifier and discard the duplicate, preventing unintended side effects.

Implementing Idempotency in Spring Boot with Kafka

Let's consider a Spring Boot application consuming OrderCreated events. We need to ensure that an order is only created once, even if the event is processed multiple times.

First, define your message structure. Each message should contain a unique ID, for example, a UUID representing the order creation request:


public class OrderCreatedEvent {
    private UUID orderId;
    private String customerId;
    private List<String> itemIds;
    private Instant timestamp;
    // getters and setters
}

Next, you'll need a mechanism to track processed order IDs. A simple in-memory cache can work for many scenarios, but for durability across application restarts, a persistent store like Redis or a database table is recommended. For this example, we'll outline a conceptual in-memory approach, acknowledging its limitations.

You'll need a service to manage this state. This service will check if an orderId has been processed and mark it as processed after successful handling.


import org.springframework.stereotype.Service;
import java.util.UUID;
import java.util.concurrent.ConcurrentHashMap;
import java.util.Map;

@Service
public class ProcessedOrderTracker {

    // In a real-world scenario, use Redis or a database for persistence.
    private final Map<UUID, Boolean> processedOrders = new ConcurrentHashMap<>();

    public boolean isOrderAlreadyProcessed(UUID orderId) {
        return processedOrders.containsKey(orderId);
    }

    public void markOrderAsProcessed(UUID orderId) {
        processedOrders.put(orderId, true);
    }
}

Now, modify your Kafka listener to use this tracker:


import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.support.Acknowledgment;
import org.springframework.kafka.support.KafkaHeaders;
import org.springframework.messaging.handler.annotation.Header;
import org.springframework.stereotype.Component;

import java.util.UUID;

@Component
public class OrderConsumer {

    private final OrderService orderService;
    private final ProcessedOrderTracker tracker;

    public OrderConsumer(OrderService orderService, ProcessedOrderTracker tracker) {
        this.orderService = orderService;
        this.tracker = tracker;
    }

    @KafkaListener(topics = "order.events", groupId = "order-service-group")
    public void listen(OrderCreatedEvent event,
                       @Header(KafkaHeaders.ACKNOWLEDGMENT) Acknowledgment acknowledgment) {

        UUID orderId = event.getOrderId();

        if (tracker.isOrderAlreadyProcessed(orderId)) {
            System.out.println("Duplicate order detected, skipping: " + orderId);
            acknowledgment.acknowledge(); // Acknowledge to avoid reprocessing
            return;
        }

        try {
            orderService.createOrder(event);
            tracker.markOrderAsProcessed(orderId);
            acknowledgment.acknowledge(); // Commit offset only after successful processing AND tracking
            System.out.println("Successfully processed order: " + orderId);
        } catch (Exception e) {
            // Handle exceptions appropriately - potentially log and do not acknowledge to trigger retry
            System.err.println("Error processing order " + orderId + ": " + e.getMessage());
            // Do NOT acknowledge here if you want Kafka to redeliver
        }
    }
}

In this consumer logic:

  • We extract the unique orderId.
  • We check the ProcessedOrderTracker. If the order is already processed, we log it, acknowledge the message (so Kafka doesn't redeliver it), and return.
  • If it's a new order, we proceed with orderService.createOrder(event).
  • Crucially, after the order is successfully created and after we've marked it as processed in our tracker, we call acknowledgment.acknowledge(). This commits the Kafka offset.
  • Error handling is vital. If an exception occurs during order creation, we do not acknowledge the message. This tells Kafka that the message failed and should be redelivered.

Beyond Simple Duplicates: Transactional Idempotency

While the above approach handles duplicate message processing, it doesn't guarantee transactional atomicity between your Kafka acknowledgment and your business logic. What if your application crashes between marking the order as processed and calling acknowledgment.acknowledge()? You've marked the order as processed, but Kafka might redeliver it.

For true exactly-once processing semantics (though Kafka itself only guarantees at-least-once delivery), you need to ensure the business operation and the offset commit are atomic. Kafka producers and consumers can participate in transactions. A more robust solution involves using Kafka transactions:

  1. Start a Kafka transaction.
  2. Process the message (e.g., create the order).
  3. Update your tracking mechanism (e.g., save the processed ID) within the same transaction.
  4. Send a