Messaging infrastructure comparison

Apache Kafka vs RabbitMQ

Two dominant messaging systems with very different philosophies. Kafka is a distributed log built for high-throughput event streaming; RabbitMQ is a traditional broker built for reliable task distribution and flexible routing. Choosing between them shapes your entire event architecture.

TL;DR

  • Kafka is a distributed commit log. Messages are retained for days/weeks, consumers read at their own pace using offsets, and replay is a first-class feature. Built for millions of messages per second.
  • RabbitMQ is a traditional message broker (AMQP). Messages are routed through exchanges to queues; consumers acknowledge receipt; messages are deleted once consumed. Excellent for task queues, complex routing, and RPC patterns.
  • Kafka for streaming and event sourcing; RabbitMQ for task queues and routing. Many organisations run both.

Side-by-side comparison

AspectApache KafkaRabbitMQ
ModelDistributed log / event streamMessage broker (AMQP)
ThroughputVery high — millions of msg/sHigh — hundreds of thousands of msg/s
PersistenceRetained by offset (configurable duration)Removed on acknowledgement
OrderingGuaranteed within partitionFIFO within a single queue
ReplayYes — consumers seek to any offsetNo (built-in); partial via Streams plugin
Consumer modelConsumer group; each partition → one consumerCompeting consumers; any worker picks up message
Routing logicBasic (topic name, partition key)Rich — direct, fanout, topic, headers exchanges
ProtocolCustom binary (Kafka protocol)AMQP 0-9-1, STOMP, MQTT
Operational complexityHigher — cluster, ZooKeeper/KRaftLower for small setups; management UI included
Best forEvent streaming, analytics, event sourcingTask queues, microservice RPC, complex routing

Code side-by-side

Producing and consuming a message:

Kafka (Node.js kafkajs)

// Producer
await producer.send({
  topic: 'order-events',
  messages: [
    { key: 'ord_123',
      value: JSON.stringify({ event: 'placed' }) },
  ],
});

// Consumer (group: analytics-service)
await consumer.subscribe({
  topic: 'order-events',
  fromBeginning: false,  // or replay with true
});
await consumer.run({
  eachMessage: async ({ message }) => {
    console.log(message.value.toString());
  },
});

RabbitMQ (amqplib)

// Producer
channel.assertQueue('order_tasks', {
  durable: true,
});
channel.sendToQueue(
  'order_tasks',
  Buffer.from(JSON.stringify({ orderId: 'ord_123' })),
  { persistent: true },
);

// Consumer
channel.consume('order_tasks', (msg) => {
  if (msg !== null) {
    processOrder(JSON.parse(msg.content));
    channel.ack(msg);  // removes from queue
  }
});

When to use Kafka

  • High-throughput event streaming. Clickstreams, IoT sensors, application logs — millions of events per second from many producers.
  • Event sourcing and CQRS. Kafka's retained log is the perfect event store; services rebuild read models by replaying the topic from offset 0.
  • Multiple independent consumers. Analytics, ML pipelines, and audit services all read the same events at different speeds without interfering with each other.
  • Stream processing. Kafka Streams and ksqlDB process data in motion — joins, aggregations, windowing — without a separate processing framework.

When to use RabbitMQ

  • Task queues with workers. Email sending, image resizing, PDF generation — workers compete for tasks; acknowledgement ensures each job runs exactly once.
  • Complex routing logic. Route messages to different queues based on routing keys, headers, or patterns without writing routing code in your application.
  • Request/reply (RPC) over messaging. RabbitMQ's reply-to and correlation-id headers support synchronous RPC patterns over async messaging.
  • Simpler operations for small teams. A single RabbitMQ node with the management UI handles most small-to-medium workloads without cluster complexity.

English phrases engineers use

Kafka conversations

  • "We produce events to the order-events topic."
  • "The analytics service is a separate consumer group."
  • "We can replay from offset zero to rebuild the read model."
  • "The partition key is the user ID — same user, same partition."
  • "We're using Kafka Streams to join orders with inventory in real time."

RabbitMQ conversations

  • "Messages go through the fanout exchange to all bound queues."
  • "The worker acks the message after successful processing."
  • "Failed messages end up on the dead-letter exchange."
  • "We set prefetch count to 1 so workers don't grab more than they can handle."
  • "The queue is durable — it survives a broker restart."

Quick decision tree

  • High throughput event streaming (millions/sec) → Kafka
  • Task queue with competing workers → RabbitMQ
  • Need to replay past events → Kafka
  • Complex routing (topic/header/pattern) → RabbitMQ
  • Multiple independent consumers of same events → Kafka
  • RPC over messaging → RabbitMQ
  • Small team, simpler operations → RabbitMQ
  • Event sourcing / CQRS → Kafka

Frequently asked questions

What is Apache Kafka in plain English?

Kafka is a distributed commit log. Producers append messages to topics; consumers read from any point in that log using an offset. Messages are retained for a configurable period (days/weeks/forever) regardless of whether they have been consumed. This enables replay, event sourcing, and multiple independent consumer groups reading the same data at their own pace.

What is RabbitMQ?

RabbitMQ is a traditional message broker implementing the AMQP protocol. Producers send messages to exchanges; exchanges route them to queues based on routing keys and bindings; consumers receive from queues and acknowledge receipt. Messages are typically removed once acknowledged. Designed for task distribution, RPC, and complex routing logic.

Which has higher throughput?

Kafka. Its sequential disk write design (append-only log) achieves millions of messages per second. RabbitMQ is highly capable — hundreds of thousands per second — but Kafka is purpose-built for high-throughput streaming and wins on raw volume.