qa-resilience
Chaos engineering, fault injection, and resilience drills: 7 skills (chaos-experiment-author, chaos-mesh, toxiproxy-chaos, failure-injection-test-author, chaos-drill-protocol, dr-drill-runner, error-budget-tests). Chaos-experiment design with steady-state pre-flight validation and LitmusChaos/Gremlin routing, the chaos-drill run protocol with abort gates, Chaos Mesh, Toxiproxy, and combined HTTP+TCP fault injection, plus the DR-drill discipline (backup verification + restore-time / RTO measurement in references) and error-budget gating with burn-rate alerting and incident metrics.
Install this plugin
/plugin install qa-resilience@testland-qaPart of role bundle: qa-role-performance
qa-resilience
Chaos engineering, fault injection, and resilience drills. The unrehearsed half: structured chaos-experiment authoring with steady-state pre-flight validation and per-tool routing (Chaos Mesh standalone; LitmusChaos and Gremlin as references), the chaos-drill run protocol with abort gates and recovery validation, Toxiproxy (TCP-level), and combined HTTP+TCP fault injection. The rehearsed half: measured, scheduled DR drills with backup verification and restore-time SLAs worked in references, plus error budgets and MTTR/MTBF tracking.
Components
| Type | Name | Description |
|---|---|---|
| Skill | chaos-experiment-author | Build-an-X workflow for a chaos experiment per the Principles of Chaos Engineering - steady-state hypothesis, real-world event choice, blast radius, automation, verdict. Includes the five-check pre-flight hypothesis validation with hard-reject rules, and routes tool choice: LitmusChaos + Gremlin setup live in references/, chaos-mesh stays standalone. |
| Skill | chaos-mesh | Configures Chaos Mesh for Kubernetes-native chaos engineering - picks fault types (PodChaos, NetworkChaos, StressChaos, IOChaos, TimeChaos, DNSChaos, KernelChaos, HTTPChaos), targets via label selectors, controls blast radius via namespace whitelists + selector filters, schedules via CronJobs, observes via dashboard. |
| Skill | toxiproxy-chaos | Configures Toxiproxy for TCP-level fault injection - runs as a sidecar / proxy between client and upstream, applies toxics (latency, bandwidth, slow_close, timeout, slicer, limit_data, reset_peer) via control API. Use for TCP-precise fault injection in development / integration environments without K8s or commercial tooling. |
| Skill | failure-injection-test-author | Build-an-X workflow that combines WireMock fault stubs (HTTP-level fault: 500s, malformed JSON, slow responses) with Toxiproxy (TCP-level: latency, packet loss, reset) into one orchestrated test scenario - the test starts both, applies fault per scenario, runs the SUT against the impaired endpoints, verifies the SUT's resilience patterns. |
| Skill | chaos-drill-protocol | The run protocol and run workflow for an already-designed experiment: four pre-flight gates, a derived blast-radius bound, abort criteria fixed before injection, per-runner inject/abort commands with refuse-to-start rules, and recovery validation with tolerance and timeout. |
| Skill | dr-drill-runner | The full DR-drill discipline: per-tier RTO + RPO; pre-drill checklist; drill workflow (announce → fail-over → verify → fail-back → cleanup); the supervised run protocol (refuse rules, RTO/RPO monitor, abort-on-breach); post-drill report; cold/warm/hot patterns; cadence. Backup-integrity verification and restore-time / RTO measurement live in references/. |
| Skill | error-budget-tests | SLI calculation; budget consumption; multi-window multi-burn-rate alerting; freeze-trigger when budget exhausted; rolling-window reset; weekly stakeholder reporting. The MTTR / MTTA / MTTD / MTBF incident-metrics schema + formulae live in references/. |
Install
/plugin marketplace add testland/qa
/plugin install qa-resilience@testland-qaSkills
chaos-drill-protocol
Run protocol and run workflow for a chaos experiment that has already been designed: the four pre-flight gates (non-production target, measured healthy baseline, live observability, a rollback that has actually been exercised), how to pick a conservative blast-radius bound, the sampling cadence and abort criteria fixed in writing before injection, the per-runner inject and abort commands (Chaos Mesh / Litmus / Gremlin / Toxiproxy), the refuse-to-start rules (no blast-radius bound, production context, degraded baseline, offline observability, unexercised rollback), and the recovery-validation step with its tolerance and timeout. Owns execution safety only, not experiment design: the steady-state hypothesis, the fault to inject, and the experiment file come from chaos-experiment-author. Use when an experiment definition exists and a fault is about to be injected into a running system, and the go/no-go gates, abort thresholds, and recovery check still need to be agreed and written down before the fault starts.
chaos-experiment-author
Build-an-X workflow for a chaos experiment per the Principles of Chaos Engineering - defines steady-state hypothesis, picks the variables (real-world events: network latency, node failure, region outage), sets the blast radius (which percentage / namespace / user cohort), automates execution, and emits the verdict (steady-state held / didn't hold). Includes the five-check pre-flight validation of the steady-state hypothesis (measurable, baselined, SLI-backed tolerance, defined measurement window, metric moves under the fault) with hard-reject rules, and routes the tool choice: Chaos Mesh has its own standalone skill, while LitmusChaos and Gremlin setup live in this skill's references. Use to scope and pre-flight-validate a chaos experiment before running it via Chaos Mesh / Litmus / Gremlin / Toxiproxy.
chaos-mesh
Configures Chaos Mesh for Kubernetes-native chaos engineering - picks fault types (PodChaos, NetworkChaos, StressChaos, IOChaos, TimeChaos, DNSChaos, KernelChaos, HTTPChaos), targets via label selectors, controls blast radius via namespace whitelists + selector filters, schedules via CronJobs, observes via dashboard. Distinct from Litmus by architecture (Chaos Mesh has its own dashboard + workflow orchestration; Litmus uses ChaosCenter UI). Use when the target system runs on Kubernetes and fault experiments should be declared as CRDs in the cluster alongside the workloads they target.
dr-drill-runner
The full DR-drill discipline for one service: author the runbook (per-tier RTO + RPO), pre-drill checklist (data sync state, alert silencing, customer comms), drill workflow (announce, fail-over, verify, fail-back) with timestamps, the supervised run protocol (refuse without declared RTO/RPO or against production, RTO/RPO monitoring cadence, abort-on-breach), and an auditor-ready post-drill report. Backup-integrity verification (SHA-256 + signature, restore spot checks, cross-region replication, retention, key recovery) and restore-time / RTO measurement (TTF segments, PITR latency, parallel-restore tuning, trend tracking) are worked in references. Per Google Cloud DR planning guide; covers cold / warm / hot standby tier-specific patterns. Use when a scheduled or post-incident failover drill for one service is being planned, executed, or written up, or when a new tier-1 service ships without a drill defined.
error-budget-tests
Build error-budget gate tests - SLO + error-budget calculation per Google SRE workbook ("difference between target uptime and actual uptime"); burn-rate alerting; monthly-budget exhaustion test; freeze-trigger when budget consumed. Per sre.google embracing-risk reference. Includes the incident-metrics reference for MTTR / MTBF / MTTD / MTTA - per-incident record schema, calculation formulae, exclusion rules, dashboards-as-code, and target-vs-actual alerting. Use when an SLO and error budget are written down but nothing verifies that burn-rate alerts fire or that the release freeze engages when the budget runs out, or when MTTR / MTBF dashboards report numbers nobody can reproduce.
failure-injection-test-author
Orchestrates WireMock fault stubs (HTTP-level fault: 500s, malformed JSON, slow responses) with Toxiproxy (TCP-level: latency, packet loss, reset) into a single resilience test scenario - the test starts both, applies fault per scenario, runs the SUT against the impaired endpoints, verifies the SUT's resilience patterns. Use when one test must reproduce a combined network + HTTP failure - a cross-layer failure mode from an incident postmortem that neither pure HTTP fault stubs nor pure TCP chaos can cover alone, because most real failures span both layers.
toxiproxy-chaos
Configures Toxiproxy for TCP-level fault injection - runs as a sidecar / proxy between client and upstream, applies toxics (latency, bandwidth, slow_close, timeout, slicer, limit_data, reset_peer) via control API. Focused on the proxy itself rather than an API-level chaos runner, including non-test usage (chaos in dev environments, integration tests, pre-prod simulation). Use when the team needs TCP-precise fault injection in development / integration environments without K8s or commercial tooling.