English for Prometheus and Metrics

Metrics vocabulary gets mixed up constantly — “counter” and “gauge” sound interchangeable to someone new to Prometheus, but they behave completely differently and mixing them up produces genuinely broken dashboards. Precise language here matters especially during an incident, when a team needs to interpret a graph correctly under time pressure rather than guess at what a metric actually represents.

Key Vocabulary

Counter A metric type that only ever increases (or resets to zero on restart), used for things like total requests served or total errors — you typically graph its rate of change, not its raw value. Example: “This is a counter, so the raw value climbing steadily is expected — what we actually care about is the rate of increase over the last five minutes, not the absolute number.”

Gauge A metric type that can go up or down freely, used for things like current memory usage or the number of active connections at a given moment. Example: “Unlike the request counter, this is a gauge — it’s meant to fluctuate, and a sudden drop to zero is the concerning signal, not a steady climb.”

Histogram A metric type that samples observations (like request durations) into configurable buckets, allowing you to compute quantiles like p95 or p99 latency after the fact. Example: “We’re using a histogram for request latency so we can calculate p99 later, rather than only ever knowing the average, which hides tail latency problems.”

Scrape The process by which Prometheus pulls metrics from a target at a regular interval, rather than the target pushing metrics to Prometheus. Example: “This service’s metrics aren’t showing up because Prometheus isn’t configured to scrape its metrics endpoint — the target isn’t in our scrape config yet.”

Label (metric label) A key-value pair attached to a metric that allows it to be sliced and filtered along a dimension, like endpoint or status_code, without needing a separate metric per value. Example: “Instead of a separate metric per endpoint, we use a single counter with an endpoint label, so we can filter or aggregate by that dimension in queries.”

Cardinality The number of distinct label value combinations a metric can produce, which directly affects memory and storage cost — high-cardinality labels (like raw user IDs) are a common source of performance problems. Example: “We accidentally added a raw user ID as a label on this metric, which exploded cardinality and is why memory usage on the Prometheus server spiked.”

Alerting rule A condition, expressed as a query, that triggers a notification when it evaluates to true for a defined duration, distinguishing a transient blip from a genuinely sustained problem. Example: “We added a for: 5m clause to this alerting rule so a brief latency spike doesn’t page anyone unless it’s sustained for five minutes.”

PromQL The query language used to select, filter, and aggregate metrics stored in Prometheus, used both for dashboards and for defining alerting rules. Example: “This PromQL query is summing the rate of errors across all instances, but we actually want it broken down per instance to find which one is failing.”

Common Phrases

In code reviews:

• “This metric is being instrumented as a gauge, but it only ever increases — should it actually be a counter instead?”
• “This label includes a raw request ID, which will create unbounded cardinality — can we drop it or bucket it into something coarser?”
• “The alerting rule doesn’t have a for duration, so a single scrape spike will page someone even if it resolves on its own within seconds.”

In standups:

• “Yesterday I converted this misclassified gauge to a proper counter with rate-based alerting; today I’m validating the new dashboard against yesterday’s incident data.”
• “I’m blocked on a cardinality explosion — a label we added last week is generating far more time series than expected, and the Prometheus server is under memory pressure.”
• “I finished adding a histogram for this endpoint’s latency, so we can finally track p95 instead of only having an average.”

In incident retrospectives:

• “The dashboard showed a counter’s raw value climbing, which looked alarming, but that’s just expected counter behavior — the actual signal we needed was the rate, which was flat.”
• “We didn’t get paged for this because the alerting rule’s threshold was set against an average, and the p99 latency was already well past what users were experiencing as slow.”
• “High cardinality on this label caused the query to time out during the incident, right when we needed to look at the dashboard the most.”

Phrases to Avoid

Saying “the metric is wrong” for expected counter behavior. Say instead: “this is a counter, so the raw value only ever increases — we should be looking at its rate, not the absolute number” — this is one of the most common points of confusion for anyone new to reading Prometheus dashboards.

Saying “add more labels” without considering cardinality. Say instead: “check whether this label could produce unbounded values before adding it — high cardinality labels are a common cause of Prometheus performance problems.” Naming cardinality explicitly signals you’re aware of the tradeoff, not just adding dimensions freely.

Saying “alert on average latency” as a default. Say instead: “alert on a specific quantile, like p95 or p99, since averages hide the tail latency that actually affects real users” — this is a frequent, consequential mistake in alerting rule design.

Quick Reference

Term	How to use it
counter	”This counter only increases; graph its rate, not the raw value.”
gauge	”The gauge should fluctuate normally; a drop to zero is the concern.”
histogram	”The histogram lets us compute p99 latency, not just an average.”
scrape	”The target isn’t in our scrape config, so its metrics aren’t collected.”
cardinality	”A raw user ID label exploded cardinality on this metric.”
alerting rule	”The for: 5m clause prevents a brief blip from paging anyone.”

Key Takeaways

• Distinguish counters from gauges precisely — a climbing counter is expected behavior, while a fluctuating gauge dropping unexpectedly is the actual signal to watch.
• Prefer histograms over simple averages for latency metrics, since averages hide tail latency that matters most to real users.
• Watch cardinality carefully before adding a new label — unbounded label values are a leading cause of Prometheus performance and cost problems.
• Use a for duration in alerting rules to distinguish a transient blip from a sustained, genuinely actionable problem.
• When a dashboard looks alarming, check whether the metric type explains the shape (a counter’s steady climb) before treating it as an incident signal.

Navigating Nuance: English for Metric Interpretation

As DevOps engineers increasingly rely on Prometheus and its associated metrics, a precise understanding of terminology becomes paramount. It’s not just about reading numbers; it’s about communicating those numbers effectively – to colleagues reviewing your code, explaining a sudden spike in latency, or crafting a clear alert rule. For non-native English speakers, this can be particularly challenging due to subtle differences in phrasing and the highly specialized vocabulary surrounding monitoring. Often, what seems like a simple statement can carry significant implications depending on the context and the level of detail provided. Let’s consider a scenario: a code review comment. Imagine you’ve written a new service that exposes a metric tracking request latency. During review, your colleague, Sarah, leaves this comment: “This request_duration_seconds metric seems unusually high during peak hours. Could you investigate the potential bottleneck?” The immediate challenge isn’t just understanding what she’s saying – it’s interpreting the implied urgency and the expectation of further action. “Unusually high” doesn’t simply mean above a threshold; it suggests a deviation from normal behavior that warrants attention. Similarly, framing an alert rule requires careful consideration. A poorly worded rule might trigger false positives or fail to capture critical issues. For example, instead of “Alert if CPU usage exceeds 80%,” a more precise phrasing would be: “Trigger an alert when sustained CPU utilization across all nodes in the cluster reaches 80% for more than five minutes.” The difference is subtle but crucial – the latter clearly defines the duration and scope of the alert.

Another common situation arises during PR descriptions. When detailing changes to a monitoring system, clarity and precision are vital. Vague statements like “Improved metric collection” won’t cut it. Instead, developers should strive for detailed explanations, outlining why a change was made and what impact it’s expected to have. For instance: “Implemented new histogram buckets for request latency to provide more granular data visualization and facilitate root cause analysis during peak load.” This demonstrates an understanding of the underlying metrics and the value they bring. Furthermore, remember that technical discussions often involve quantifying observations – “the increase in error rates was approximately 15%” is much clearer than simply stating “error rates increased”.

Finally, don’t be afraid to ask for clarification. If you’re unsure about a term or phrase used by colleagues, politely request an explanation. Asking questions demonstrates your commitment to understanding the system and avoids misunderstandings that could lead to inaccurate interpretations or ineffective solutions. The key is to focus on communicating intent and context alongside the technical details.

Here’s an example of how you might use promtool to investigate a metric:

promtool query 'rate(http_requests_total[5m])'

This command queries for the rate of HTTP requests over a 5-minute window, which could be used to identify sudden surges in traffic. Analyzing this output – and understanding the implications of the rate() function and the chosen time window – is just as important as executing the command itself.