Vocabulary for Chaos Engineers

Chaos engineering is the discipline of deliberately introducing failures into a system to discover weaknesses before they cause unplanned outages. It was pioneered by Netflix and has become a core practice in site reliability engineering. The vocabulary of chaos engineering is specific and precise — mastering it will help you design experiments, communicate risk to stakeholders, and participate in game days with confidence.

Chaos Engineering

Chaos engineering is the practice of running controlled experiments on a system in order to build confidence in the system’s ability to withstand turbulent, unexpected conditions in production.

“Chaos engineering is not about breaking things randomly — it is about running principled experiments that test specific hypotheses about system resilience.” “We introduced chaos engineering after our third unexpected database failover caused an outage. We needed to know whether our recovery mechanisms actually worked.”

Steady State

The steady state is the normal, healthy operating condition of a system — defined by measurable metrics such as error rate, throughput, and latency. A chaos experiment validates that the system returns to steady state after a disruption.

“Before running an experiment, we define our steady state: p99 latency under 200ms, error rate below 0.1%, and throughput above 1,000 requests per second.” “A successful chaos experiment is one where the system maintains or returns to steady state despite the injected failure. If it does not, we have found a weakness.”

Experiment Hypothesis

An experiment hypothesis in chaos engineering follows the format: “If we inject X failure, the system will maintain steady state because of Y safeguard.” The hypothesis is specific, falsifiable, and tied to measurable outcomes.

“Our hypothesis: if we terminate 50% of the instances in the payment service, the load balancer will route traffic to the remaining instances within 30 seconds and the error rate will not exceed 1%.” “We were wrong about our hypothesis. When we terminated the instances, the health checks took 90 seconds to kick in — far longer than we assumed.”

Failure Injection

Failure injection is the act of deliberately introducing a fault into a system — such as killing a process, introducing network latency, filling a disk, or corrupting a configuration file. Failure injection tools include Chaos Monkey, Gremlin, and LitmusChaos.

“We use Gremlin to inject network latency between the order service and the inventory service. We simulate 500ms of added latency and observe whether the order service’s circuit breaker triggers correctly.” “Failure injection should be gradual. Start with a small blast radius in a non-production environment before moving to production.”

Blast Radius

Blast radius refers to the scope of impact — how many users, services, or components will be affected if a particular failure occurs. Controlling the blast radius is a safety practice in chaos engineering: start small and expand incrementally.

“We limit the blast radius of our initial experiments to 5% of production traffic. If we see unexpected behaviour, we stop before it affects more users.” “The blast radius of a database failure is larger than the blast radius of a single application instance failure — we run database experiments with much more caution.”

Game Day

A game day is a scheduled, facilitated chaos engineering exercise in which an engineering team deliberately triggers failures and practises responding to them — similar to a fire drill.

“We run a quarterly game day where we simulate a range of failure scenarios: availability zone failure, external API outage, and database leader election. Each team must detect, respond, and restore within the agreed SLA.” “Game days expose gaps in runbooks, alerting, and incident response processes that would otherwise only surface in a real incident — at the worst possible moment.”

Turbulence

Turbulence is a broader metaphor used in chaos engineering to describe the unpredictable, chaotic conditions of production — unexpected traffic spikes, hardware failures, network partitions, and software bugs that combine in ways not anticipated in testing.

“Our staging environment is stable. Our production environment is turbulence. Chaos engineering helps us close the gap between what we think will happen and what actually happens.”

Abort Conditions

Abort conditions (or stop conditions) are pre-defined thresholds that automatically halt a chaos experiment if things go worse than expected. They are a safety mechanism to prevent controlled experiments from becoming uncontrolled outages.

“We set an abort condition: if the error rate exceeds 5%, the experiment is automatically halted and the injected failure is reversed.” “Never run a chaos experiment without abort conditions. The point is to find weaknesses — not to cause an incident.”

Observability in Chaos Engineering

Chaos experiments are meaningless without observability — the ability to measure the system’s behaviour during and after the experiment. You need metrics, logs, and traces to verify whether the hypothesis held.

“Before running the experiment, ensure your observability stack is in place. If you cannot measure the steady state, you cannot determine whether the experiment passed or failed.”

Practical Phrases for Chaos Engineers

• “Let’s define the steady state before we design the experiment.”
• “The hypothesis is: if we inject 200ms of latency on the payment gateway, the checkout flow degrades gracefully with a user-visible warning rather than a silent failure.”
• “We’ll start with a blast radius of 10% of traffic and expand if the system holds.”
• “The game day is scheduled for next Thursday. Teams should review their runbooks beforehand.”
• “The abort condition is: total error rate exceeds 2%. If we hit that, we stop immediately.”
• “The experiment passed — the system maintained steady state throughout the failure injection period.”

Chaos engineering vocabulary reflects the discipline’s philosophy: systematic, scientific, and safety-first. Mastering these terms will help you design rigorous experiments, communicate risk to product and business stakeholders, and build genuinely resilient systems that survive the turbulence of real-world production environments.

Navigating the Chaos: A Primer for Engineers

This article aims to equip you with essential terminology used within the field of chaos engineering – specifically focusing on how to think about system resilience through controlled disruption. It’s not a prescriptive guide but rather an exploration of common concepts and approaches.

The core idea behind chaos engineering isn’t wanton destruction, but rather a systematic approach to understanding and improving a system’s ability to withstand unexpected events. We achieve this by deliberately introducing failures – what we call “failure injection” – to test our assumptions about stability and resilience. Understanding the nuances of terms like “blast radius” and “steady state” is crucial for effective experimentation.

Let’s delve into some key vocabulary:

1. Blast Radius: This isn’t literally an explosion, but rather the scope of a failure. It represents the extent to which a problem can propagate within a system if left unchecked. A small blast radius means a localized issue easily contained; a large one indicates potential cascading failures. Think of it as the reach of a ripple in a pond – the wider the splash, the further the disturbance.

2. Steady State: The desired operational condition for a system. It’s not necessarily perfect, but it’s the baseline we’re trying to protect against disruption. Maintaining this “steady state” requires constant vigilance and proactive testing. It’s worth noting that ‘steady state’ is rarely truly static - there will always be some level of ‘turbulence’.

3. Experiment Hypothesis: Every failure injection must start with a clear hypothesis – what are we trying to learn? For example, “If we introduce this latency spike during peak hours, will the caching layer still effectively mitigate the impact?” This is about formulating testable predictions.

4. Failure Injection: The deliberate introduction of faults or errors into a system to observe its behavior and identify vulnerabilities. This isn’t about breaking things randomly; it’s about systematically probing the system’s resilience.

5. Game Day: A specific period during which you actively monitor your system, focusing intently on potential failure injection points. It’s the execution phase of your experiment, demanding heightened awareness and rapid response capabilities.

6. Turbulence: Represents the normal fluctuations and variations within a system – things that aren’t necessarily failures but can still disrupt operations if they escalate or are combined with injected faults. Think network latency, high user load, or unexpected traffic patterns.

# Example: Introducing Latency via a Network Emulator (using `tc`)

# This command simulates increased network latency to test caching performance.
# Requires root/sudo privileges and knowledge of your network topology.
#  This is for illustrative purposes only - use with caution!
sudo tc qdisc add dev eth0 root netem delay 100ms loss 5%

7. Recovery Time Objective (RTO) & Point-in-Time Restore (PITR): These are operational metrics, not chaos engineering vocabulary per se, but vitally important to understand alongside the philosophical concepts. RTO defines how long it takes to restore a service after an outage. PITR refers to restoring a system to a specific point in time, often useful when data corruption is suspected.

Now, let’s consider a practical scenario:

Slack Message: “Hey team, running a ‘game day’ experiment with the new database schema. We’re injecting latency and sporadic connection errors to test our alerting thresholds. Blast radius is currently limited to the reporting dashboard – we’ll monitor closely for any cascading failures. Hypothesis: If we increase the error rate beyond 10%, the automated rollback won’t trigger fast enough.”

PR Description: “Implementing feature X, introducing a controlled failure injection via synthetic traffic generation to validate resilience against high load scenarios. Monitoring key metrics – latency, error rates, and resource utilization – will inform adjustments to the steady state configuration.”

Understanding these terms—and continually refining your thinking around system stability and disruption—is what elevates you from simply managing systems to engineering resilience.