Kafka Operations Vocabulary

Consumer group lag is the difference between the latest offset written to a topic partition and the current offset of the consumer group. A lag of 50K means the consumer is 50K messages behind — it's not processing fast enough. High lag indicates the consumer needs scaling or the producer volume has spiked.

In Kafka, each partition has one leader broker that handles all reads and writes for that partition. Other brokers may hold replicas (followers). If the leader broker fails, Kafka elects a new leader from the in-sync replicas. Knowing which broker is the partition leader helps diagnose performance bottlenecks.

A consumer group rebalance reassigns which partitions are consumed by which consumer instances. When a new consumer joins (or leaves), Kafka rebalances to distribute partitions evenly. During a rebalance all consumers briefly pause — this is called 'stop the world' and can cause temporary processing delays.

Kafka topic retention is time-based (or size-based) and controls how long messages are kept on brokers. A 7-day retention means messages are available for replay or new consumers to read for 7 days. After that, they are deleted. Retention is independent of consumption — even read messages stay until retention expires.

Replication factor 3 means each partition has 3 copies distributed across 3 different brokers. With 3 brokers and replication factor 3, the cluster can tolerate up to 2 broker failures without losing data. This is a common production configuration — it balances fault tolerance with storage cost.