Prometheus & PromQL

Prometheus has distinct metric types. A counter is a cumulative value that only ever increases, resetting to zero only on process restart — ideal for totals like requests served or errors. Because the raw number is meaningless on its own, you wrap counters in rate() to get per-second change. A gauge represents a value that can rise and fall — temperature, memory in use, queue depth — so you read it directly. A histogram additionally buckets observations for latency distributions.

rate() calculates the per-second average rate of increase of a counter across a specified range window, e.g. rate(http_requests_total[5m]). It automatically handles counter resets (treating a drop as a restart) and extrapolates at window edges. Because it averages over the window, rate() smooths spikes; for fast-moving alerting you might use irate(), which uses only the last two samples. Applying rate() to a gauge is meaningless — use it exclusively on counters.

A label is a key-value pair attached to a metric that adds a dimension, such as method="GET" or status="500". Every unique combination of metric name and labels forms its own time series. Labels let you slice and aggregate data — for instance summing request rates across all instances or filtering to one endpoint. Beware high cardinality: putting unbounded values (user IDs, request IDs) in labels explodes the number of series and can overwhelm Prometheus's memory.

Prometheus follows a pull model: on a configured interval it scrapes each target by making an HTTP request to its /metrics endpoint, which exposes current metric values in a simple text exposure format. Targets are discovered statically or via service discovery (Kubernetes, Consul, etc.). Each scrape's success and duration are themselves recorded as metrics. For short-lived batch jobs that cannot be scraped, a Pushgateway acts as an intermediary the job pushes to and Prometheus scrapes.

A recording rule evaluates a PromQL expression at a fixed interval and stores the result as a brand-new time series. This precomputes expensive or frequently used queries — like an aggregated request rate across all instances — so dashboards and alerts read a cheap, ready-made metric instead of recomputing it each time. Recording rules live in rule files loaded by Prometheus and follow a naming convention (e.g. level:metric:operation). They contrast with alerting rules, which evaluate a condition and fire alerts to Alertmanager.