Publish-Subscribe Pattern

Why pub/sub is everywhere

Three parts: a publisher, a topic, and one or more subscribers. The publisher writes a message into a topic. The broker keeps a list of subscribers for that topic and delivers the message to all of them. Done.

What you get from these three primitives is enormous. Decoupling: publisher and subscribers never address each other. Fan-out: N subscribers each receive every message. Independence: subscribers come and go without disturbing the publisher. Substitution: replace a subscriber, the publisher doesn't notice. Replay (in log-based brokers): subscribers can rewind.

Pub/sub is the messaging primitive at the heart of every event-driven system. The Observer pattern is its in-process cousin: same shape — one Subject, many Observers, broadcast on state change — minus the network and the broker.

Pub/sub vs message queue

The single most important distinction in messaging:

• Queue — each message goes to one consumer. Multiple consumers compete; the broker load-balances across them. Used for work distribution (10 worker processes draining a queue of jobs).
• Topic (pub/sub) — each message goes to every subscriber. Used for broadcast (one event triggering many independent reactions).

Kafka unifies both with consumer groups. Within a group, partitions are split among consumers (queue semantics). Multiple groups on the same topic each get the full feed (pub/sub semantics). RabbitMQ separates them explicitly: queues vs exchanges. Redis has both PUBLISH/SUBSCRIBE (pub/sub) and BLPOP-style queues.

A worked example: notification fan-out

Your app's notification service publishes a NotificationRequested event to topic notifications. Four subscribers register:

• EmailSender: renders an email and sends via SES.
• SMSSender: calls Twilio API.
• PushSender: sends to APNs / FCM.
• InAppNotifier: writes a row to the in-app feed.

One publish call — broker.publish("notifications", payload) — and four downstream services react independently. Add Slack later? Subscribe SlackSender to the topic; nothing else changes. Drop SMS? Unsubscribe SMSSender; nothing else changes. The notification service is in blissful ignorance of how many subscribers exist.

Pub/sub in Kafka

Kafka is the most-deployed pub/sub system in production. Five concepts to know:

1. Topic. A named log: orders.OrderPlaced. Append-only, durable, replicated.
2. Partition. Topics are split into N partitions for parallelism. Messages with the same key always go to the same partition (preserves order per key).
3. Producer. Appends messages to a topic. Picks the partition (by key or round-robin).
4. Consumer. Reads messages from one or more partitions, tracks its own offset.
5. Consumer group. A set of consumers cooperating to consume a topic; each partition is assigned to one consumer in the group. Two consumer groups on the same topic each get the full message stream — that's the pub/sub fan-out.

Throughput: 1M+ msg/sec/broker on commodity hardware. Latency: 5-10 ms p99 publish, 1-5 ms tail read. Retention: configurable (default 7 days; many shops set to forever for event sourcing).

from confluent_kafka import Producer, Consumer

# Publisher
producer = Producer({"bootstrap.servers": "kafka:9092"})
producer.produce(
    topic="notifications",
    key=user_id,
    value=json.dumps({"type": "order_placed", ...}),
)
producer.flush()

# Subscriber (one of many)
consumer = Consumer({
    "bootstrap.servers": "kafka:9092",
    "group.id": "email-sender",        # group = isolation per subscriber type
    "auto.offset.reset": "earliest",
})
consumer.subscribe(["notifications"])
while True:
    msg = consumer.poll(1.0)
    if msg and not msg.error():
        send_email(json.loads(msg.value()))
        consumer.commit(msg)

Redis Pub/Sub

Redis offers a much simpler pub/sub: in-memory, no persistence, no replay. Latency: sub-millisecond. Throughput: ~1M msg/sec per server. Limitation: subscribers must be connected at publish time — a subscriber that connects after a publish never sees the missed messages.

import redis
r = redis.Redis()

# Publisher
r.publish("notifications", json.dumps({"type": "order_placed"}))

# Subscriber
pubsub = r.pubsub()
pubsub.subscribe("notifications")
for msg in pubsub.listen():
    if msg["type"] == "message":
        handle(json.loads(msg["data"]))

Use Redis Pub/Sub for low-latency in-process broadcast (live dashboards, chat-room presence). Use Kafka for durable, replayable event streams. Redis Streams (newer) sits between — persistent and consumer-group-aware.

Pub/sub brokers compared

	Kafka	Redis Pub/Sub	RabbitMQ	NATS	GCP Pub/Sub	MQTT
Durability	Disk + replication	None (in-memory)	Disk-optional	Optional (JetStream)	Managed durable	Optional (QoS 1/2)
Throughput	1M+ msg/sec/broker	~1M msg/sec/server	~50k msg/sec/node	1M+ msg/sec	Autoscale	~10k msg/sec/broker
p99 latency	5-10 ms	<1 ms	1-10 ms	<10 ms	~50 ms	10-100 ms
Replay history	Yes (offsets)	No	No (consumed)	JetStream: yes	Yes (snapshots)	Retained: yes
Subscription model	Consumer groups	Channel SUBSCRIBE	Queue bindings	Subject subscription	Subscription resource	Topic + wildcard
Wildcards	Regex consumers	PSUBSCRIBE	Topic exchange	Subject wildcards	Subscription filter	+ and # wildcards
Typical use	Stream + ES	Real-time fan-out	Work queues	In-cluster messaging	Managed event ingest	IoT / edge

The Observer pattern connection

Pub/sub's in-process ancestor is the Observer pattern (GoF 1994). Same shape, different scope:

class Subject:
    def __init__(self):
        self._observers = []
    def subscribe(self, observer):
        self._observers.append(observer)
    def publish(self, event):
        for observer in self._observers:
            observer.notify(event)

class EmailObserver:
    def notify(self, event):
        send_email(event)

class AnalyticsObserver:
    def notify(self, event):
        record_metric(event)

subject = Subject()
subject.subscribe(EmailObserver())
subject.subscribe(AnalyticsObserver())
subject.publish({"type": "order_placed"})

This is exactly pub/sub minus the network, the durability, and the cross-process scope. Every UI framework's event model (DOM events, React state subscriptions, RxJS) is Observer-shaped. Every distributed messaging system is pub/sub-shaped. The pattern scales from "function call list" to "trillion messages per day" by changing the transport, not the model.

Ordering, delivery, and exactly-once

• Per-key ordering. Kafka guarantees order within a partition. Use the entity ID (order_id, user_id) as the partition key to preserve order for that entity.
• At-least-once by default. Subscribers may see duplicates due to network retries. Idempotency keys at the consumer are mandatory.
• Kafka EOS (2017). Transactional producer + transactional consumer + transactional sink — exactly-once within Kafka. Doesn't extend to external sinks unless the sink also participates.
• Effectively-once. The pragmatic target: at-least-once delivery + idempotent consumers = no duplicate observable effect.
• Backpressure. Fast publisher + slow consumer = lag. Kafka shows this as consumer-group lag metrics; alert when lag exceeds threshold.

Real-world pub/sub at scale

• LinkedIn Kafka. Invented Kafka. ~7 trillion msg/day across activity events, feed updates, ML feature streams.
• Slack. ~3.5B real-time events/day flow through Kafka and Redis Pub/Sub for the WebSocket fan-out.
• Twitter. Original timeline fan-out was Redis Pub/Sub; later moved to a hybrid push/pull architecture for scale.
• AWS SNS. Cloud pub/sub at scale; ~1 trillion msg/month across SNS topics globally.
• MQTT broker fleets. Tesla, smart-home platforms (Home Assistant, AWS IoT) use MQTT pub/sub for device-to-cloud at millions of devices per region.
• Google Cloud Pub/Sub. Manages every YouTube view event for analytics — hundreds of billions/day.

Common misconceptions

• "Pub/sub equals broadcast equals fan-out." Yes — and that means every subscriber pays the cost. If you only want one consumer to process, use a queue.
• "Kafka and Redis Pub/Sub are interchangeable." No. Kafka persists; Redis Pub/Sub doesn't. Choose based on durability requirements.
• "Subscribers always see every message." Only if they're connected (Redis) or within retention (Kafka). Missed messages outside the window are lost.
• "Ordering is global." No — per-partition only (in Kafka). Use partition keys to scope ordering to an entity.
• "Subscribers process in real time." They process when they consume. Lag is normal. Monitor it.
• "Pub/sub gives you exactly-once." At-least-once is the default. Combine with idempotent consumers for effectively-once.
• "Adding a subscriber is free." Each subscriber processes every message — that's CPU and IO. Free for the publisher; not free for the broker or subscriber.

Performance characteristics

• Publish latency: 1-50 ms typical depending on broker. Redis fastest, Kafka durable, MQTT highest for QoS 2.
• Fan-out cost: O(N) per message at the broker — N subscribers means N delivery operations. Brokers like Kafka push this to consumers (pull model), Redis pushes from broker (push model).
• Subscriber state: Kafka tracks offsets, Redis Pub/Sub tracks active connections, RabbitMQ tracks queue bindings. State proportional to (topics × subscribers).
• Storage (Kafka): ~500 bytes/event compressed; 250 GB / 1B events. Retention configurable.
• Replay cost: rewinding a Kafka offset to 0 replays the whole topic — useful for new consumers backfilling history.