Chaos Engineering Language Exercises

The steady state is the normal, measurable behaviour of the system before any fault is introduced — for example, 99.5% request success rate, p99 latency under 200 ms, and zero error-page responses. Defining it before the experiment gives you a concrete baseline to compare against. If the system's behaviour during fault injection deviates from this baseline, the experiment has revealed a weakness. In a pre-experiment brief: "Our steady-state hypothesis is that payment processing maintains a success rate above 99% and median latency below 150 ms. We will abort the experiment if either threshold is breached."

Blast radius describes the scope of potential impact if a chaos experiment causes unexpected damage — which users, services, or data could be affected. You intentionally start with a small blast radius and expand only when confidence is established. Planning language: "We'll scope the first run to 1% of production traffic in a single availability zone. Once we've validated that the circuit breaker fires correctly, we'll increase the blast radius to 10%." This term is also used in incident retrospectives to describe the actual scope of an outage.

A Game Day (or GameDay) is a structured, facilitated exercise where teams intentionally introduce failures into a system to practice detection and response. Communication for leadership: "We're running a Game Day on Thursday afternoon. The engineering and on-call teams will rehearse real failure scenarios in a controlled environment — similar to a fire drill. The goal is to surface gaps in our runbooks, alerts, and communication procedures before those failures happen in an unplanned way."

Common fault injection types include: network latency injection (adding artificial delay to service calls), packet loss (dropping a percentage of network requests), CPU or memory pressure (consuming resources to simulate a stressed host), process kill (terminating a service instance), disk full simulation, and dependency unavailability (making a downstream service return errors or time out). In an experiment proposal: "For this run we'll inject 500 ms of latency on all calls from the order service to the inventory service to validate that our timeout and retry configuration is correct."

A chaos hypothesis follows the structure: "We believe that [system] will [maintain steady state] when [fault is introduced]." Example: "We believe that the checkout service will continue processing orders with a success rate above 98% when the recommendations service is made unavailable, because checkout does not have a hard dependency on recommendations." A well-formed hypothesis makes the experiment falsifiable and gives the team a clear pass/fail criterion.

Key sections and terms: experiment summary (what fault was injected, duration, scope), steady-state verification (baseline metrics before the run), observations (what actually happened during the fault), findings (gaps discovered — missing circuit breakers, slow alert thresholds, incomplete runbooks), and recommendations (prioritised action items). Useful phrases: "The system degraded gracefully / failed to degrade gracefully," "Alerting fired within the expected window / with a 4-minute delay," "Fallback behaviour was triggered as designed / was not triggered."

Load testing validates that a system can handle expected or peak traffic volume — it tests capacity and performance under stress. Fault injection validates that a system behaves correctly when individual components fail — it tests reliability and resilience design patterns like circuit breakers, retries, and fallbacks. In conversation: "Load testing told us our service handles 10,000 requests per second. Chaos engineering told us what happens when our primary database becomes unavailable at that load — two different risk dimensions."

Chaos Monkey, originally developed by Netflix, is a tool that randomly terminates virtual machine or container instances in production to ensure the system can tolerate unexpected instance failures. The "Simian Army" extended this concept with tools targeting latency (Latency Monkey), security (Security Monkey), and compliance. In an introduction: "Chaos Monkey is essentially an automated adversary — it randomly kills services to prove that our system is resilient enough to survive the kinds of failures that happen naturally in distributed infrastructure."

Observability refers to how well you can understand the internal state of a system from its external outputs — logs, metrics, and traces. In chaos engineering, high observability is a prerequisite: if you cannot tell what the system is doing during a fault injection, you cannot learn anything from the experiment. Useful language: "Before running this experiment, we need to verify we have adequate observability — specifically, distributed tracing on the payment service and a dashboard that shows per-service error rates in real time."

Maintain a dedicated communication channel (Slack or Teams thread) for the duration of the experiment. Designate roles: experiment lead (coordinates), safety observer (watches for abort criteria), and scribe (documents observations in real time). Announce clearly before starting: "Experiment is live — fault injection active as of 14:03 UTC." Call the abort clearly if thresholds are breached: "Aborting experiment — success rate has dropped to 94%, below the 98% abort threshold. Fault injection stopped at 14:11 UTC." Document the timeline for the retrospective.