chaos-engineer

Overview

Chaos engineering is the disciplined practice of injecting failure to verify that systems handle it — finding weaknesses in business hours instead of at 3am. Done right, it's careful science: hypothesis, blast radius control, and abort conditions. Done wrong, it's just breaking production. This skill covers the methodology that makes it the former.

When to use

• Designing your first chaos experiments (kill pods, inject latency, drop traffic)
• Building a chaos program: scope, safety, and cadence
• Running game days for critical user journeys
• Validating failover, autoscaling, and degradation behavior
• Convincing stakeholders that controlled failure injection is worthwhile

Core concepts

Hypothesis-driven, not random. Every experiment states a hypothesis: "We believe killing 30% of pods in zone A keeps p99 latency under 500ms because the HPA scales and the LB drains." You run the experiment to confirm or refute — the learning is the point, including (especially) when the hypothesis is wrong.

Blast radius starts tiny. Begin in staging, then production with minimal scope (one pod, one AZ, 1% of traffic), expanding only as confidence grows. Controls: target precisely, limit duration, and always have an instant abort (a big red button, automatic halt on SLO burn).

Steady-state first. Define normal (the metrics that characterize health) before injecting failure, so you can measure deviation objectively. Without a baseline, every experiment result is an anecdote.

The five classic failure modes to test: process death (crash recovery), network latency/packet loss (degraded dependencies), resource exhaustion (CPU/memory/disk pressure), dependency failure (what if the cache/DB/3rd party is down?), and zone/region loss (failover actually working). Most outages are one of these wearing a costume.

Game days for the human system. Scheduled simulations where the team practices response to a scenario — they test runbooks, communication, and coordination, not just technology. The debrief often surfaces more value than the experiment itself.

Practical workflow

1. Pick a critical user journey and map its dependencies (services, datastores, third parties, DNS, CDN) — experiments target real paths.
2. Write the hypothesis with a falsifiable steady-state claim and success criteria tied to SLIs.
3. Design for safety: minimal blast radius, short duration, automatic abort on guardrail metrics (error budget burn, p99 latency), manual abort procedure, and execution during business hours with the owning team present and aware.
4. Run in expanding scope: staging → production-canary → broader production — only advance when the previous scope behaved as hypothesized.
5. Debrief blamelessly: what happened vs hypothesis, what broke unexpectedly, what the fix is — file action items with owners.
6. Codify the program: a backlog of experiments prioritized by risk, regular cadence (monthly game day, continuous small experiments), and results tracked like any engineering work.

Common pitfalls

• Starting in production with wide blast radius — the fastest way to lose organizational trust; earn scope with small wins.
• No abort conditions — experiments without automatic halt on SLO burn turn "controlled" into "incident we caused on purpose".
• Testing at 2am to "avoid impact" — defeats the purpose (fewer eyes, slower response); run during business hours with the team ready.
• Surprising stakeholders — product, support, and leadership should know experiments are happening; surprises erode trust even when nothing breaks.
• One-off experiments with no follow-through — finding that failover takes 8 minutes is useless if nobody fixes it; the action items are the product.
• Chaos as theater — killing pods that Kubernetes replaces in seconds proves nothing about your actual risks; target the scary unknowns (region loss, dependency outage, data corruption paths).