api-chaos-runner
Runs the project's existing API tests under injected network chaos - latency, timeouts, dropped connections, bandwidth caps, packet loss - via Toxiproxy (notes on Pumba / Gremlin / LitmusChaos). Builds a per-scenario chaos matrix and reports which assertions break under which conditions, verifying resilience patterns (retry, circuit-breaker, timeout, fallback). Unlike schemathesis-fuzzing and restler-fuzzing, which generate new tests from a schema, this drives your EXISTING example-based suite.
Install with skills.sh (any agent)
npx skills add testland/qa --skill api-chaos-runnerapi-chaos-runner
Overview
Most API tests run against perfect networks: <1ms latency, no packet loss, infinite bandwidth, deterministic ordering. Real production isn't like that. Network chaos testing drives the existing tests under controlled network impairment - the team discovers which retry / circuit-breaker / timeout patterns actually hold up before real customers find out.
The canonical open-source primitive is Toxiproxy - Shopify's "TCP proxy to simulate network and system conditions for chaos and resiliency testing" (toxiproxy-readme (opens in new window)). The pattern: sit Toxiproxy between client and upstream; manipulate toxics (latency, timeout, bandwidth, etc.) during test execution.
This skill is build-an-X - the workflow chains the team's existing API tests (Postman / Karate / RestAssured / Tavern / Schemathesis) through a Toxiproxy-managed connection and orchestrates a per-scenario chaos matrix.
When to use
If the team is just starting API testing and has no resilience patterns to verify, this skill is overkill - start with happy-path coverage via postman-collections or the language-native equivalents first.
How to use
Step 1 - Pick the chaos primitive
| Tool | Layer | Best for |
|---|---|---|
| Toxiproxy | TCP proxy | Per-connection latency / bandwidth / timeout / drop. Most precise. |
| Pumba | Docker container | Container-level chaos (kill, pause, network). |
| Gremlin (commercial) | Multi-platform | Production-grade chaos with audit / approval flow. |
| LitmusChaos | Kubernetes operator | Cloud-native; experiments declared as CRDs. |
tc qdisc (Linux native) | Network interface | Lowest level; most setup; CI-friendly only with --cap-add NET_ADMIN. |
Default recommendation: Toxiproxy for per-API chaos in CI. The others fit when the team is already in those ecosystems (Docker-Compose-heavy projects, Kubernetes-first projects).
Step 2 - Define the chaos matrix
For each existing API test scenario, define a matrix of conditions to run it under:
| Scenario | Toxic | Expected behavior |
|---|---|---|
| Order create (POST /orders) | None (control) | 201 in <500ms |
| Order create | latency=1000ms | 201 in <2s (within timeout budget) |
| Order create | latency=10000ms | 504 with retry-after, OR client gives up |
| Order create | bandwidth=10kbps | 201 (eventual) OR 408 timeout |
| Order create | reset_peer | 502 with retry attempted |
| Order create | timeout | 504; circuit-breaker opens after 3rd |
Per toxiproxy-readme (opens in new window), the canonical toxic types include latency, down (forced failure), bandwidth, slow_close, timeout, slicer, limit_data, reset_peer.
The matrix is the load-bearing artifact: what the team expects under each condition is what differentiates resilience verification from "did the test pass?" The Expected column drives the assertions.
Step 3 - Wire Toxiproxy and run the matrix
Sit Toxiproxy between the app and its upstream, register a proxy per dependency, and add/remove toxics around each test run. Point the app at Toxiproxy's listen ports; it forwards to the real upstream when no toxic is active. A matrix runner loops the scenarios, adding one toxic per row and producing one JUnit XML per scenario to aggregate in the report stage.
Full docker-compose setup, control-API proxy registration, toxiproxy-cli toxic commands, and the runner shell script: references/toxiproxy-wiring.md.
Step 4 - Report what broke under what
A successful chaos run produces a resilience matrix report:
## API Chaos Matrix - verdict: REVIEW
| Scenario | Control | Latency 1s | Bandwidth 10k | Timeout 5s | Reset peer |
|------------------|:-------:|:----------:|:-------------:|:----------:|:----------:|
| POST /orders | ✅ | ✅ | ✅ | ✅ | ❌ |
| GET /orders/:id | ✅ | ✅ | ❌ | ✅ | ✅ |
| DELETE /orders/:id | ✅ | ✅ | ✅ | ✅ | ✅ |
### Failures
| Test | Toxic | Expected | Actual |
|------|-------------|---------------------------------------------------|--------|
| POST /orders | reset_peer | 502 + retry attempted; second attempt succeeds | 502; no retry observed in client logs |
| GET /orders/:id | bandwidth=10k | 200 in <30s | 408 timeout at 10s |A green matrix isn't the goal - finding where resilience is missing is the goal. A failure under a chaos scenario is a feature request, not a bug in the test.
Choosing what to inject
Match toxics to documented resilience requirements:
| Resilience pattern documented | Toxic to inject |
|---|---|
| Retry on 5xx | down (forced 5xx) or reset_peer |
| Timeout after Nms | latency=N+500 (force the timeout) |
| Circuit-breaker after 3 failures | down for ≥3 requests |
| Fallback to cache when upstream unreachable | down indefinitely |
| Bulkhead under load | bandwidth=very-low |
| Slow-loris client | slow_close on response |
Run only the toxics that map to a documented expectation; running every toxic against every endpoint is noise.
Worked example
The orders service documents "retry once on connection reset." The team already has a Postman collection covering POST /orders.
Verdict: REVIEW. The documented retry policy is not actually implemented. The output is a fix request against the client (add the retry), not a bug in the test - exactly the resilience gap the matrix exists to surface.
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Chaos in production | Real users observe; oncall pages. | CI / staging only. Production chaos requires the team's full chaos engineering practice (Gremlin / Litmus + approval flow). |
| Per-PR chaos matrix | Adds 10+ minutes; team disables. | Nightly chaos runs; PR runs only the control row. |
| Asserting "chaos must not break anything" | Every system has a breaking point; the test trivially fails. | Assert specific resilience behavior under specific conditions; document the breaking point as accepted. |
Using down for everything | down forces 5xx; doesn't model real-world latency / bandwidth. | Mix latency, bandwidth, timeout, reset_peer for realistic mixes. |
| Skipping the control row | Without control, the matrix can't distinguish chaos failures from test bugs. | Always run a no-toxic scenario as the baseline. |
Limitations
References
Toxiproxy wiring and matrix runner
View source (opens in new window)Toxiproxy wiring and matrix runner
Full setup for placing Toxiproxy between the client and upstream, registering proxies, adding toxics, and running the chaos matrix. Referenced from the api-chaos-runner skill's Step 3.
Setup (Docker example)
# docker-compose.test.yml
services:
toxiproxy:
image: ghcr.io/shopify/toxiproxy:latest
ports:
- 8474:8474 # control API
- 5432:5432 # proxied DB
- 8080:8080 # proxied API
app:
build: .
environment:
DATABASE_URL: 'postgres://user:pass@toxiproxy:5432/db'
EXTERNAL_API_URL: 'http://toxiproxy:8080'Per toxiproxy-readme (opens in new window), the application points at Toxiproxy's listen ports rather than the upstream. Toxiproxy forwards to the real upstream when no toxic is active.
Define proxies via the control API
# Register the upstream
curl -d '{"name":"orders-api","listen":"0.0.0.0:8080","upstream":"orders-api-real:8080"}' \
http://toxiproxy:8474/proxiesAdd a toxic during a test
Per toxiproxy-readme (opens in new window):
# 1000ms latency on every request through this proxy
toxiproxy-cli toxic add -t latency -a latency=1000 orders-api
# Bandwidth cap at 10 KB/s
toxiproxy-cli toxic add -t bandwidth -a rate=10 orders-api
# Forced timeout
toxiproxy-cli toxic add -t timeout -a timeout=5000 orders-api
# Remove all toxics
toxiproxy-cli toxic remove orders-api -n <toxic-name>For a stateless add-test-remove cycle, the language-native client libraries (toxiproxy-python, toxiproxy-node, toxiproxy-ruby, toxiproxy-go) wrap the HTTP API.
Run the matrix
A minimal runner shell script:
#!/usr/bin/env bash
# scripts/chaos-matrix.sh
set -e
PROXY=orders-api
TEST_CMD="npx newman run collections/orders.postman_collection.json -e environments/chaos.json -r cli,junit --reporter-junit-export results-$1.xml"
run_with_toxic() {
local label="$1"; local type="$2"; local args="$3"
echo "=== $label ==="
toxiproxy-cli toxic remove "$PROXY" -n latency 2>/dev/null || true
toxiproxy-cli toxic remove "$PROXY" -n bandwidth 2>/dev/null || true
toxiproxy-cli toxic remove "$PROXY" -n timeout 2>/dev/null || true
if [ -n "$type" ]; then
toxiproxy-cli toxic add -t "$type" $args "$PROXY"
fi
$TEST_CMD "$label" || true # don't bail; we want the matrix
}
run_with_toxic 'control' '' ''
run_with_toxic 'latency-1s' latency '-a latency=1000'
run_with_toxic 'bandwidth' bandwidth '-a rate=10'
run_with_toxic 'timeout' timeout '-a timeout=5000'The matrix produces one JUnit XML per scenario. Aggregate them in the report stage.
Related skills
api-testing-overview
Teaches API testing from zero: what functional API testing covers, how it differs from contract testing and load testing, and a decision table that picks one tool from observable project facts (language and build file, whether an OpenAPI or GraphQL schema exists, functional vs spec-conformance fuzzing vs stateful security fuzzing, whether non-engineers read the tests). Names the real options (Postman with newman, REST Assured, Karate, Tavern, Schemathesis, RESTler), gives install and first-run commands with what a passing run looks like, and the traps that bite first: asserting only on HTTP status, order-dependent tests sharing server state, and hardcoded environment URLs and secrets. Use when an HTTP API needs automated tests and no tool has been chosen, or when an inherited suite only checks status codes.
karate-testing
Authors Karate `.feature` files using its Gherkin-flavored DSL for HTTP API tests, leverages the `match` keyword with fuzzy validators (#number / #string / #regex / contains / arrays), runs the suite via JUnit 5 plus Maven Surefire, and produces JUnit XML for CI gating. Use when the project is on the JVM and prefers a feature-file authoring flow over Java-DSL fluent chains; for those fluent chains use restassured-testing, for the same YAML-style flow on a Python/pytest stack use tavern-testing.
postman-collections
Authors Postman collections (requests + tests + variables + environments), runs them headless via the Newman CLI, configures reporters (cli / json / junit / html) for CI artifact upload, and uses iteration data files (JSON / CSV) for data-driven runs. Use when the project ships HTTP API tests authored in Postman and the team needs CI execution alongside or instead of the Postman desktop runner.
restassured-testing
Authors REST Assured (Java) API tests using the given().when().then() BDD-style DSL - status code + JSON/XML path assertions + authentication (Basic, OAuth2, API key). Configures Maven / Gradle dependencies, runs via JUnit 5, and emits Surefire / JaCoCo reports for CI gating. Use when the project is on the JVM and wants type-safe API tests in the app's own language; for a Gherkin feature-file flow on the same JVM use karate-testing, for YAML tests on the pytest stack use tavern-testing.
restler-fuzzing
Runs stateful REST API fuzzing using Microsoft's RESTler - infers producer-consumer dependencies from an OpenAPI spec, drives sequences of requests (POST → GET → DELETE chains), and reports 5xx errors, resource leaks, and hierarchy violations. Wraps the canonical 4-stage workflow (compile → test → fuzz-lean → fuzz). Use when the API is stateful (resources are created, queried, modified, deleted) and Schemathesis's stateless fuzzing is missing the multi-step bugs.
schemathesis-fuzzing
Generates property-based API tests automatically from an OpenAPI 2/3.x or GraphQL schema using Schemathesis, runs them via the `schemathesis run` CLI or as a pytest decorator, configures the canonical checks (status_code_conformance, response_schema_conformance, content_type_conformance, response_headers_conformance, not_a_server_error), and gates CI on schema-conformance failures plus 5xx detection. Use when the project ships an OpenAPI or GraphQL schema and the team wants schema-driven coverage that scales as the API evolves.
tavern-testing
Authors Tavern API tests as YAML files (`test_*.tavern.yaml`) with `test_name` + `stages` + `request` + `response` blocks, runs them through the Tavern pytest plugin (auto-discovered), and gates CI on the resulting JUnit XML. Covers RESTful, MQTT, and gRPC variants. Use when the project runs on pytest and prefers YAML over a Python- or Java-DSL; on the JVM use karate-testing or restassured-testing instead, and for schema-driven property-based fuzzing on the same pytest stack use schemathesis-fuzzing.