Message Queues Vocabulary

A topic is a named stream or category of messages — a logical channel that publishers write to and consumers subscribe to. It decouples producers from consumers: a producer need not know who reads the data, and many consumers can independently read the same topic. In systems like Apache Kafka, a topic is the top-level unit of organization, typically split into multiple partitions for scalability. Topics let you separate different kinds of events (e.g. orders, payments) into distinct, independently managed streams.

A partition is a horizontal slice of a topic — an ordered, append-only log of messages. Splitting a topic into partitions lets the system distribute data and load across brokers and allows multiple consumers to read in parallel, which is the key to throughput and scalability. Ordering is guaranteed only within a single partition, not across the whole topic; messages sharing a partition key land in the same partition to preserve their relative order. The number of partitions effectively caps how many consumers in a group can process a topic concurrently.

A consumer group is a set of consumer instances that work together to consume a topic, with the partitions divided among them so that each partition is read by exactly one member of the group. This delivers load balancing and scalability — adding consumers (up to the partition count) increases throughput — and provides fault tolerance, since partitions are reassigned (a rebalance) if a member fails. Importantly, each group tracks its own progress independently, so multiple groups can each receive a full copy of the topic for different purposes.

An offset is a monotonically increasing integer that uniquely identifies the position of a message within a partition's ordered log. Consumers track which offsets they have processed by committing them, so on restart or rebalance they resume from the last committed position rather than reprocessing or skipping data. Because offsets are just positions in a retained log, a consumer can also rewind to an earlier offset to replay messages or skip ahead. Managing offset commits correctly is central to achieving at-least-once or exactly-once processing semantics.

A dead-letter queue (DLQ) is a secondary queue where messages are routed after they cannot be processed successfully — for example after exceeding a maximum retry count, failing validation, or expiring. Diverting these "poison" messages prevents them from blocking the main queue and endlessly consuming resources, while preserving them for later inspection, debugging, or manual reprocessing. DLQs are a key reliability pattern: they isolate failures so the healthy message flow continues, and they give operators visibility into what went wrong and why.