Real-Time & Streaming Data Language Exercises

A Kafka topic is a named, durable log to which producers write records and from which consumers read them. Engineers describe topics as "append-only logs" that are "partitioned for parallelism" and "replicated for fault tolerance." Common phrases include "publishing to a topic," "consuming from a topic," and "topic retention policy" to set how long records are kept.

A consumer group is a set of consumer instances that collectively read a topic, with each partition assigned to exactly one consumer in the group at a time. Engineers use this concept to explain horizontal scaling: "adding consumers to the group" increases throughput, while "rebalancing" describes the process of redistributing partition assignments when group membership changes.

An offset is an integer that marks a consumer's position within a partition. Engineers say consumers "commit offsets" to record progress, allowing them to "resume from where they left off" after a restart. Key phrases include "auto-commit," "manual offset commit," "seek to beginning," and "lag" — the difference between the latest offset and the committed offset — which indicates how far behind a consumer is.

Stream processing vocabulary includes "stateful operators," "windowing," "event time vs. processing time," and "watermarks." Engineers say a Flink job "processes unbounded streams" with "low-latency stateful transformations," while Spark Streaming "micro-batches" records. Key terms are "exactly-once processing," "checkpointing," and "backpressure" — the mechanism that slows producers when consumers fall behind.

Windowing groups records into finite sets for aggregation. Engineers describe "tumbling windows" (non-overlapping fixed-size intervals), "sliding windows" (overlapping intervals), and "session windows" (gaps between events define boundaries). The phrase "window closes" means the system emits a result for that window. "Late data" refers to records that arrive after a window has closed.

Exactly-once semantics (EOS) guarantees that each record is processed and its effect reflected in output exactly one time, even if failures cause retries. Engineers contrast it with "at-most-once" (records may be lost) and "at-least-once" (duplicates are possible). In Kafka, EOS is enabled through idempotent producers and transactional APIs, and engineers say it "eliminates duplicates end-to-end."

Event time is when an event actually occurred (recorded in the event's timestamp), while processing time is when the streaming engine processes it. Engineers emphasise using event time for accurate analytics because processing time is affected by network delays and system load. "Watermarks" are used to track progress in event time and signal when a window can safely be closed despite out-of-order arrivals.

Producer vocabulary includes "acknowledgement mode" (acks=0, acks=1, acks=all), "batch size," "linger.ms" (how long to wait before sending a batch), "compression codec," and "idempotence." Engineers say a producer "flushes its buffer" when the batch is full or the linger time expires. "Producer throughput" and "producer latency" are traded off against each other via these settings.

Schema management in streaming systems uses terms like "Schema Registry," "schema evolution," "backward compatibility," and "forward compatibility." Engineers say producers "serialize with Avro or Protobuf" and schemas are "registered" with the registry. "Schema drift" describes the problem where producers and consumers use incompatible versions, causing deserialization failures downstream.

Stream-table duality is the idea that a stream can be interpreted as a table (the latest value per key) and a table can be interpreted as a stream of changes (a changelog). Kafka Streams and ksqlDB use this concept to enable "materialised views" from streams and "stream-stream joins." Engineers describe "KStream" as an unbounded sequence of events and "KTable" as a compacted, key-value changelog.