Exactly-Once Delivery

How the three delivery semantics work

Every distributed message bus delivers messages with one of three semantics, and only one of those three is actually free.

At-most-once. The producer sends the message and doesn't retry. If the network drops the packet, the message is lost. If the consumer crashes after receiving but before processing, the message is lost. Zero duplicates, zero guarantees. Fire-and-forget UDP. Acceptable for periodic state updates where the next update obsoletes the missed one — sensor telemetry, presence pings.

At-least-once. The producer retries until it gets an ACK. The consumer processes, then ACKs. If a crash happens between processing and ACK, the producer retries and the message is processed twice. Duplicates are the default. Every "reliable" message queue without explicit EOS is at-least-once — Kafka, RabbitMQ, SQS, NATS JetStream, Pub/Sub. The right default for almost everything.

Exactly-once. The message is processed once and only once. Two Generals proved it impossible to achieve over an unreliable network with a finite protocol. What's actually achievable is exactly-once processing: the consumer's side effects happen exactly once, even if the wire delivers the message multiple times. Two production paths get you there: Kafka's transactional EOS, or an idempotent consumer with a dedup key.

Path A — Idempotent consumers with a dedup key

The pragmatic answer used in 90% of real systems. Every message carries a stable, producer-generated ID. The consumer checks an idempotency store before processing:

function handle(msg):
    if seen.contains(msg.id):
        ack(msg)              // duplicate — skip processing
        return
    process(msg)              // do the side effect
    seen.add(msg.id)           // remember we did it
    ack(msg)

The dedup store is the load-bearing piece. Redis with SETNX and a TTL is the standard cheap option — 100 bytes per ID, one extra RTT (~1 ms), and IDs age out after a day. A Postgres table with a unique index on the message ID gives exactness at higher cost. A counting Bloom filter trades a tiny false-positive rate (and occasional silent drops) for 10× memory savings.

This protects against duplicates on the consumer. It does not protect against duplicates on external side effects. If the consumer does charge_card then seen.add and crashes between them, the next delivery will charge again. For safe external calls you push idempotency further out: the payment gateway accepts an Idempotency-Key header and rejects duplicate keys at its own boundary.

Path B — Kafka transactional EOS

Kafka 0.11 (2017) introduced transactional producers and the processing.guarantee=exactly_once_v2 consumer mode. The flow is:

producer.initTransactions()
loop:
    record = consumer.poll()
    producer.beginTransaction()
    result = process(record)
    producer.send(outputTopic, result)
    producer.sendOffsetsToTransaction({inputPartition: record.offset+1})
    producer.commitTransaction()

The transaction coordinator on the broker bundles the produced message AND the consumer's offset commit into one atomic operation. Either both land or neither does. Consumers on the output topic with isolation.level=read_committed only see messages from committed transactions. If the consumer crashes mid-process, the transaction aborts, the offset stays at the old value, and the message gets re-processed cleanly.

The cost: every transaction adds ~50–100 ms of latency for the commit RTT and writes transaction markers to every partition. Throughput drops by roughly 5% versus at-least-once; you trade some headroom for genuine end-to-end exactness within Kafka. It does not extend to external sinks — writing to Postgres or making an HTTP call still requires application-level dedup.

When you actually need exactly-once

• Side effects that cost money. Credit card charges, payouts, refunds. Duplicates here are felt directly on the bottom line.
• External messaging. SMS, email, push notifications. Receivers experience duplicates as a bad product.
• Counters and aggregations. Incrementing a balance or a view count by 1 is not idempotent — duplicates inflate the number.
• Workflow orchestration. A saga step that "create order in external system" should not create two orders.

You don't need exactly-once when duplicates are no-ops: setting a flag to true, upserting a record with the same key, refreshing a cache to a current value. Frame your handler as set state X = Y rather than increment X by 1 and at-least-once becomes safe for free.

At-most-once vs at-least-once vs effectively-once

	At-most-once	At-least-once	Effectively-once (dedup)	Kafka EOS
Lost messages	Possible	No (retry until ACK)	No	No
Duplicate processing	Never	Yes (on retry)	No (filtered)	No
Producer cost	Send and forget	Retry until ACK	Retry + stable ID	TX init + commit
Consumer cost	None	None	Dedup lookup per msg	read_committed iso
Latency overhead	0	~0 in happy path	+1 ms / msg (Redis)	+50–100 ms / TX
Throughput penalty	None	None	~2–5% (lookup)	~5%
External sinks	n/a	n/a	Needs idem-key at sink	Needs idem-key at sink
Right for	Telemetry, presence	Idempotent upserts	Cards, emails, sagas	Stream-to-stream (Kafka-only) pipelines

Effectively-once via dedup is the default in microservice and saga architectures because it composes — every service brings its own dedup store and is robust to upstream duplicates. Kafka EOS dominates inside a single Kafka topology (stream-processing apps with Kafka Streams or Flink) where the transactional boundary covers the entire pipeline.

What exactly-once actually costs

The Kafka EOS overhead has been measured by Confluent: roughly 5% lower max throughput compared to at-least-once on the same hardware, plus an extra 50–100 ms of commit latency. For most workloads at 10k msgs/sec, that means peak drops from say 100k to 95k — usually irrelevant. The latency hit hurts only for low-RPS interactive flows.

The dedup approach has a per-message cost: one Redis lookup (~1 ms over the local network) plus the storage for the ID. Storage cost is bounded by your TTL. At 1k msgs/sec with a 24-hour TTL and 100 bytes per ID, that's 8.6 GB of dedup state — fits in a single Redis instance with room to spare.

The operational cost is the hidden one. Every "exactly-once" claim depends on the dedup store being available. If Redis is partitioned or down, the consumer must decide: fail open (skip dedup, accept duplicates) or fail closed (reject the message). Both are bad. Some teams replicate the dedup store, others fail closed and let messages pile up. None of those choices is free.

Pseudo-code

// PATH A — idempotent consumer with Redis dedup.

handle(msg):
    // Atomic check-and-set with a TTL — Redis SETNX EX 86400
    if not redis.setNX(key="dedup:" + msg.id, value="1", ttl=86400):
        ack(msg)             // already processed → skip
        return
    try:
        process(msg)
        ack(msg)
    except Error:
        redis.del("dedup:" + msg.id)   // allow retry
        nack(msg)

// PATH B — Kafka EOS transactional consume-process-produce.

producer.initTransactions()
while true:
    records = consumer.poll(100ms)
    producer.beginTransaction()
    try:
        for record in records:
            out = transform(record)
            producer.send(outputTopic, out)
        producer.sendOffsetsToTransaction(
            consumer.committedOffsets(),
            consumer.groupMetadata()
        )
        producer.commitTransaction()
    except KafkaException:
        producer.abortTransaction()
        consumer.seek(records.beginOffset)

Python: idempotent consumer with Postgres dedup

import psycopg2
from contextlib import closing

SCHEMA = """
CREATE TABLE IF NOT EXISTS processed_messages (
  message_id  UUID PRIMARY KEY,
  processed_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
"""

def handle(conn, msg, side_effect):
    """Process exactly-once via Postgres unique constraint."""
    with conn:
        with conn.cursor() as cur:
            try:
                cur.execute(
                    "INSERT INTO processed_messages (message_id) "
                    "VALUES (%s)",
                    (msg.id,),
                )
            except psycopg2.errors.UniqueViolation:
                # Already processed — drop silently.
                return "duplicate"

            # Side effect runs INSIDE the transaction.
            # If side_effect raises, the row insert rolls back too,
            # so retries are safe.
            side_effect(msg)
            return "processed"

def cleanup(conn, days=7):
    """Periodic GC of old dedup rows."""
    with conn:
        with conn.cursor() as cur:
            cur.execute(
                "DELETE FROM processed_messages "
                "WHERE processed_at < now() - INTERVAL '%s days'",
                (days,),
            )

The unique constraint plus transactional side-effect gives exactness as long as the side effect is itself transactional (DB write, atomic file write). For external HTTP calls, push the Idempotency-Key header to the gateway and let it dedupe at the boundary — that's how Stripe's API has worked for a decade.

Variants and extensions

• Idempotent producer (Kafka 0.11+). The producer attaches a sequence number to each batch. The broker rejects duplicate sequences. Cheaper than full EOS — eliminates producer-side retries-as-duplicates without transaction overhead.
• Bloom filter dedup. 10 bits per item gives 99.9% accuracy. Memory-cheap when the ID space is huge and a tiny false-positive miss rate is acceptable.
• External idempotency keys. Stripe, AWS, GitHub — every serious API accepts an Idempotency-Key header and dedupes server-side. Your consumer becomes effectively-once at the boundary instead of needing local dedup.
• Saga-step dedup. Each saga step is dedicated to a saga instance ID + step number; replay re-runs the step idempotently because the saga state machine ignores already-completed steps.
• Flink / Spark Structured Streaming EOS. Internal Kafka-to-Kafka EOS via transactional sinks. The framework hides the transaction lifecycle.
• Two-phase event publishing (transactional outbox + CDC). Combine the outbox pattern with idempotent consumers and you have effectively-once end-to-end without Kafka EOS.

Common pitfalls and edge cases

• Non-deterministic processing. If your handler calls now() or random(), the same input message produces different output on each retry — your "idempotent" function isn't. Capture timestamps and random IDs in the producer, not the consumer.
• Side effects after ACK. Process, then ACK, then trigger an external HTTP call — a crash between ACK and HTTP loses the side effect entirely (at-most-once behaviour even on at-least-once delivery). Always ACK after the side effect succeeds.
• Dedup store TTL too short. If a duplicate arrives 25 hours later (rare but possible with retried producer batches across rebalancing), your 24-hour TTL won't catch it. Set TTL generously beyond any plausible retry window.
• External system without idempotency keys. Calling a payment API that doesn't accept idempotency keys means every retry charges again. Wrap such APIs with an internal proxy that maintains its own dedup table.
• Mixing EOS and non-EOS in one pipeline. If part of your Kafka topology is transactional and part is not, the non-transactional portion will see uncommitted messages and your "exactly-once" claim is fiction. Audit the entire pipeline.
• Forgetting consumer group rebalance. During rebalance, a partition can be reassigned mid-flight; the new owner re-reads from the last committed offset. Without dedup, the old owner's in-flight work duplicates.