Messaging infrastructure comparison
Apache Kafka vs RabbitMQ
Two dominant messaging systems with very different philosophies. Kafka is a distributed log built for high-throughput event streaming; RabbitMQ is a traditional broker built for reliable task distribution and flexible routing. Choosing between them shapes your entire event architecture.
TL;DR
- Kafka is a distributed commit log. Messages are retained for days/weeks, consumers read at their own pace using offsets, and replay is a first-class feature. Built for millions of messages per second.
- RabbitMQ is a traditional message broker (AMQP). Messages are routed through exchanges to queues; consumers acknowledge receipt; messages are deleted once consumed. Excellent for task queues, complex routing, and RPC patterns.
- Kafka for streaming and event sourcing; RabbitMQ for task queues and routing. Many organisations run both.
Side-by-side comparison
| Aspect | Apache Kafka | RabbitMQ |
|---|---|---|
| Model | Distributed log / event stream | Message broker (AMQP) |
| Throughput | Very high — millions of msg/s | High — hundreds of thousands of msg/s |
| Persistence | Retained by offset (configurable duration) | Removed on acknowledgement |
| Ordering | Guaranteed within partition | FIFO within a single queue |
| Replay | Yes — consumers seek to any offset | No (built-in); partial via Streams plugin |
| Consumer model | Consumer group; each partition → one consumer | Competing consumers; any worker picks up message |
| Routing logic | Basic (topic name, partition key) | Rich — direct, fanout, topic, headers exchanges |
| Protocol | Custom binary (Kafka protocol) | AMQP 0-9-1, STOMP, MQTT |
| Operational complexity | Higher — cluster, ZooKeeper/KRaft | Lower for small setups; management UI included |
| Best for | Event streaming, analytics, event sourcing | Task queues, microservice RPC, complex routing |
Code side-by-side
Producing and consuming a message:
Kafka (Node.js kafkajs)
// Producer
await producer.send({
topic: 'order-events',
messages: [
{ key: 'ord_123',
value: JSON.stringify({ event: 'placed' }) },
],
});
// Consumer (group: analytics-service)
await consumer.subscribe({
topic: 'order-events',
fromBeginning: false, // or replay with true
});
await consumer.run({
eachMessage: async ({ message }) => {
console.log(message.value.toString());
},
}); RabbitMQ (amqplib)
// Producer
channel.assertQueue('order_tasks', {
durable: true,
});
channel.sendToQueue(
'order_tasks',
Buffer.from(JSON.stringify({ orderId: 'ord_123' })),
{ persistent: true },
);
// Consumer
channel.consume('order_tasks', (msg) => {
if (msg !== null) {
processOrder(JSON.parse(msg.content));
channel.ack(msg); // removes from queue
}
}); When to use Kafka
- High-throughput event streaming. Clickstreams, IoT sensors, application logs — millions of events per second from many producers.
- Event sourcing and CQRS. Kafka's retained log is the perfect event store; services rebuild read models by replaying the topic from offset 0.
- Multiple independent consumers. Analytics, ML pipelines, and audit services all read the same events at different speeds without interfering with each other.
- Stream processing. Kafka Streams and ksqlDB process data in motion — joins, aggregations, windowing — without a separate processing framework.
When to use RabbitMQ
- Task queues with workers. Email sending, image resizing, PDF generation — workers compete for tasks; acknowledgement ensures each job runs exactly once.
- Complex routing logic. Route messages to different queues based on routing keys, headers, or patterns without writing routing code in your application.
- Request/reply (RPC) over messaging. RabbitMQ's reply-to and correlation-id headers support synchronous RPC patterns over async messaging.
- Simpler operations for small teams. A single RabbitMQ node with the management UI handles most small-to-medium workloads without cluster complexity.
English phrases engineers use
Kafka conversations
- "We produce events to the order-events topic."
- "The analytics service is a separate consumer group."
- "We can replay from offset zero to rebuild the read model."
- "The partition key is the user ID — same user, same partition."
- "We're using Kafka Streams to join orders with inventory in real time."
RabbitMQ conversations
- "Messages go through the fanout exchange to all bound queues."
- "The worker acks the message after successful processing."
- "Failed messages end up on the dead-letter exchange."
- "We set prefetch count to 1 so workers don't grab more than they can handle."
- "The queue is durable — it survives a broker restart."
Quick decision tree
- High throughput event streaming (millions/sec) → Kafka
- Task queue with competing workers → RabbitMQ
- Need to replay past events → Kafka
- Complex routing (topic/header/pattern) → RabbitMQ
- Multiple independent consumers of same events → Kafka
- RPC over messaging → RabbitMQ
- Small team, simpler operations → RabbitMQ
- Event sourcing / CQRS → Kafka
Frequently asked questions
What is Apache Kafka in plain English?
Kafka is a distributed commit log. Producers append messages to topics; consumers read from any point in that log using an offset. Messages are retained for a configurable period (days/weeks/forever) regardless of whether they have been consumed. This enables replay, event sourcing, and multiple independent consumer groups reading the same data at their own pace.
What is RabbitMQ?
RabbitMQ is a traditional message broker implementing the AMQP protocol. Producers send messages to exchanges; exchanges route them to queues based on routing keys and bindings; consumers receive from queues and acknowledge receipt. Messages are typically removed once acknowledged. Designed for task distribution, RPC, and complex routing logic.
Which has higher throughput?
Kafka. Its sequential disk write design (append-only log) achieves millions of messages per second. RabbitMQ is highly capable — hundreds of thousands per second — but Kafka is purpose-built for high-throughput streaming and wins on raw volume.
Can I replay messages in RabbitMQ?
Not natively. Once a RabbitMQ message is acknowledged it is gone. You can achieve a form of replay by not acknowledging messages, using the Shovel plugin to copy messages, or using the streams feature (added in RabbitMQ 3.9), which provides a log-like append-only structure similar to Kafka.
Which is easier to operate?
RabbitMQ is generally considered easier to operate for standard use cases — a single broker handles most needs, and the management UI is excellent. Kafka requires a cluster (or at minimum careful single-broker configuration), has more moving parts, and Zookeeper / KRaft mode adds operational overhead. Managed services (Confluent, MSK, CloudAMQP) reduce this for both.
Does Kafka guarantee message ordering?
Yes, within a partition. A Kafka topic can have multiple partitions; messages in a single partition are strictly ordered. If you need global ordering of all messages in a topic, you must use a single partition (which sacrifices parallelism). RabbitMQ guarantees FIFO ordering within a single queue.