Observability Engineering Language

The three pillars of observability are metrics, logs, and traces. Metrics are numeric time-series measurements—counters, gauges, and histograms—that capture aggregate system behaviour over time. Logs are timestamped records of discrete events, ideally structured as JSON to enable querying. Traces represent the path of a single request through a distributed system, connecting spans across multiple services. Engineers often say "we need visibility into the system" when they mean they need at least one of these three signal types.

A Service Level Indicator (SLI) is a specific metric that measures service behaviour—such as request success rate or p99 latency. A Service Level Objective (SLO) is an internal target for that metric—for example, "99.9% of requests succeed within 500ms over a 30-day window." A Service Level Agreement (SLA) is a contractual commitment made to customers, typically less strict than the internal SLO to provide an error budget buffer. Engineers discuss "burning the error budget" when SLI performance approaches or breaches the SLO threshold.

OpenTelemetry (OTel) is a vendor-neutral open-source observability framework that provides APIs, SDKs, and a collector for instrumenting applications to emit traces, metrics, and logs in a standardised format. Key vocabulary includes instrumentation library, auto-instrumentation, manual instrumentation, the OTLP protocol (OpenTelemetry Line Protocol), the OTel Collector (an agent for receiving, processing, and exporting telemetry), and semantic conventions (standardised attribute names for common operations). The project is governed by the CNCF (Cloud Native Computing Foundation).

Distributed tracing is the technique of tracking a single request as it propagates through multiple microservices, capturing timing and contextual information at each step. A trace is composed of spans—individual units of work within a service—and each span carries attributes (key-value metadata), events (point-in-time annotations), and a status. The trace context is propagated between services via headers (W3C TraceContext or B3 format), enabling the assembly of a complete request waterfall diagram in tools like Jaeger, Zipkin, or Grafana Tempo.

Cardinality refers to the number of unique label or tag value combinations for a given metric. High-cardinality metrics—those with labels like user_id, request_id, or IP address—can generate millions of unique time series, causing memory exhaustion and query performance problems in systems like Prometheus. Engineers speak of "cardinality explosion," "high-cardinality labels," and "label pruning" when optimising metric schemas. Understanding cardinality is essential for cost-effective, scalable metrics infrastructure.

Alerting strategy vocabulary includes terms like threshold alert, anomaly detection, alert fatigue (too many low-signal notifications), alert suppression, dead man's switch (an alert that fires if a signal stops arriving), severity levels (P1/P2/P3 or critical/warning/info), oncall rotation, escalation policy, and runbook. Engineers discuss the difference between symptom-based alerting (alerting on user-visible impact) and cause-based alerting (alerting on internal system states), generally recommending the former for paging and the latter for dashboards.

Structured logging is the practice of emitting log records as machine-parseable key-value pairs—typically JSON—rather than free-form text strings. Engineers contrast it with unstructured logging, where information is embedded in human-readable sentences that require regex to query. Key vocabulary includes log level (DEBUG, INFO, WARN, ERROR, FATAL), log correlation (linking logs to traces via a trace ID field), log aggregation (centralising logs in a system like Loki, Elasticsearch, or CloudWatch), and log retention policy.

An error budget is the maximum allowable amount of downtime or failure permitted by an SLO within a given period. If a service's SLO is 99.9% availability over 30 days, the error budget is 0.1%—approximately 43 minutes of allowable downtime. Teams "spend" the error budget when reliability incidents occur and "refill" it at the start of each window. Common phrases include "we've burned through our error budget," "the error budget is nearly exhausted," and "we need to freeze feature work to protect the budget."

The OpenTelemetry Collector is a vendor-agnostic proxy that receives telemetry data from instrumented services, processes it (filtering, batching, sampling, attribute enrichment), and exports it to one or more observability backends. Engineers describe pipelines composed of receivers (ingestion), processors (transformation), and exporters (output). The Collector can be deployed as an agent sidecar, a gateway, or both in a hierarchical topology, and it decouples application instrumentation from the choice of backend.

Dashboard and on-call vocabulary includes golden signals (latency, traffic, errors, saturation), service health overview, drill-down, time range selector, panel, variable, annotation, and threshold line. In on-call contexts, engineers use phrases like "triaging an alert," "following the runbook," "escalating to the owning team," "declaring an incident," and "handing off to the next on-call." Understanding this vocabulary is essential for reading dashboards, writing runbooks, and communicating effectively during incident response.