Testland
Browse all skills & agents

failure-injection-test-author

Orchestrates WireMock fault stubs (HTTP-level fault: 500s, malformed JSON, slow responses) with Toxiproxy (TCP-level: latency, packet loss, reset) into a single resilience test scenario - the test starts both, applies fault per scenario, runs the SUT against the impaired endpoints, verifies the SUT's resilience patterns. Use when one test must reproduce a combined network + HTTP failure - a cross-layer failure mode from an incident postmortem that neither pure HTTP fault stubs nor pure TCP chaos can cover alone, because most real failures span both layers.

Install with skills.sh (any agent)

npx skills add testland/qa --skill failure-injection-test-author
View source

failure-injection-test-author

Overview

Real production failures span layers:

  • HTTP layer: Stripe returns 500; webhook payload malformed; third-party times out.
  • TCP layer: Connection reset; high latency; packet loss.

A test using only WireMock (HTTP fault stubs) misses TCP-level chaos. A test using only Toxiproxy misses payload-level faults. Production failures combine both.

This skill builds a workflow that chains WireMock + Toxiproxy into one orchestrated test scenario - closer to production reality.

When to use

  • A resilience test must verify behavior under both HTTP and TCP faults.
  • An incident postmortem identified a "TCP reset followed by malformed retry response" failure mode that single-tool tests can't reproduce.
  • A team running combined integration + chaos testing wants a single test pattern.

For pure HTTP fault stubs, see wiremock-stubs (in the qa-test-data plugin). For pure TCP chaos, see toxiproxy-chaos.

Step 1 - Topology

[ SUT (App) ] → [ Toxiproxy ] → [ WireMock ] → (returns canned response or 500)
       ↓                ↓                ↓
   resilience       network chaos    HTTP fault stub
   patterns         (latency, etc)   (500, malformed JSON, etc)

The SUT connects to Toxiproxy; Toxiproxy forwards to WireMock; WireMock returns the configured response. The combined chain exercises both layers.

Step 2 - docker-compose setup

# docker-compose.test.yml
services:
  wiremock:
    image: wiremock/wiremock:3
    ports: ["8081:8080"]
    volumes:
      - ./wiremock-mappings:/home/wiremock/mappings

  toxiproxy:
    image: ghcr.io/shopify/toxiproxy:latest
    ports:
      - "8474:8474"
      - "8080:8080"     # what the SUT connects to

  app:
    build: .
    environment:
      EXTERNAL_API_URL: http://toxiproxy:8080

The SUT's EXTERNAL_API_URL points at Toxiproxy:8080; Toxiproxy forwards to wiremock:8080.

Step 3 - Configure the proxy

Once both containers are up:

# Tell Toxiproxy where to forward
curl -d '{"name":"external-api","listen":"0.0.0.0:8080","upstream":"wiremock:8080"}' \
  http://toxiproxy:8474/proxies

Step 4 - Per-scenario test setup

// tests/resilience.spec.ts
import { Toxiproxy } from 'toxiproxy-node-client';
import axios from 'axios';

const toxiproxy = new Toxiproxy('http://toxiproxy:8474');
const wiremockBase = 'http://wiremock:8080';

beforeEach(async () => {
  // Reset both
  await axios.delete(`${wiremockBase}/__admin/mappings`);
  const proxy = await toxiproxy.get('external-api');
  for (const toxic of await proxy.toxics()) {
    await proxy.removeToxic(toxic.name);
  }
});

test('SUT retries on TCP reset followed by 500 then succeeds', async () => {
  // 1. Stub WireMock: first call returns 500, second returns 200
  await axios.post(`${wiremockBase}/__admin/mappings`, {
    request: { method: 'GET', url: '/api/orders/1' },
    response: { status: 500 },
    priority: 1,
    scenarioName: 'retry-test',
    requiredScenarioState: 'Started',
    newScenarioState: 'after-first',
  });
  await axios.post(`${wiremockBase}/__admin/mappings`, {
    request: { method: 'GET', url: '/api/orders/1' },
    response: { status: 200, jsonBody: { id: 1, status: 'fulfilled' } },
    priority: 2,
    scenarioName: 'retry-test',
    requiredScenarioState: 'after-first',
  });

  // 2. Configure Toxiproxy: reset_peer toxic
  const proxy = await toxiproxy.get('external-api');
  await proxy.addToxic({
    name: 'reset-on-first-byte',
    type: 'reset_peer',
    attributes: { timeout: 0 },
  });

  // 3. Trigger SUT
  const result = await sut.fetchOrder(1);

  // 4. Assert: SUT recovered after retry
  expect(result).toEqual({ id: 1, status: 'fulfilled' });

  // 5. Verify the WireMock log shows 2 attempts
  const requests = await axios.get(`${wiremockBase}/__admin/requests`);
  expect(requests.data.requests).toHaveLength(2);
});

The test verifies: SUT made 2 calls (per WireMock log) and the second succeeded - the retry pattern works under TCP-reset + HTTP-500 combined fault.

Step 5 - Scenario catalog

Common scenarios:

Scenario nameTCP toxicHTTP faultVerifies
Slow + 500latency 2000ms500 statusRetry honors timeout + retry-on-5xx
Reset + retry successreset_peer (1 hit)200 (next call)Retry handles connection reset
Slow bodybandwidth 1KB/s200 with large payloadRead timeout fires
Malformed JSON(none)200 + invalid JSONParser handles gracefully
Cascade: timeout + 503timeout 5000ms503Circuit breaker opens after N timeouts
Network partitiontimeout (forever)(n/a)Fallback to cached / null

Step 6 - Verdict

Each scenario produces a per-resilience-pattern verdict:

## Failure injection results - `<sha>`

| Scenario              | SUT behavior                            | Verdict |
|-----------------------|-----------------------------------------|---------|
| Slow + 500             | Retried 3 times; succeeded on 3rd        |   ✅    |
| Reset + retry success  | Retried; succeeded                       |   ✅    |
| Slow body              | Read timeout at 5s; aborted               |   ✅    |
| Malformed JSON          | ParseError thrown; defaulted to empty    |   ✅    |
| Cascade: timeout + 503 | Circuit breaker opened after 3 timeouts  |   ✅    |
| Network partition       | Fell back to cached value                 |   ⚠ partial - fallback returned stale > 1h |

Step 7 - CI integration

- run: docker compose -f docker-compose.test.yml up --wait --wait-timeout 120
- run: npx jest tests/resilience.spec.ts
- run: docker compose -f docker-compose.test.yml down --volumes

Anti-patterns

Anti-patternWhy it failsFix
Mocking the HTTP client instead of using WireMockMock can't simulate TCP-level faults.Real Toxiproxy + WireMock chain (Step 1).
Forgetting to reset toxics + stubs between testsCross-test contamination.beforeEach reset (Step 4).
Single-scenario tests (just 500, no TCP)Real failures combine layers; single-layer tests miss them.Author scenarios spanning both layers (Step 5).
Per-test docker-compose up / downSlow; per-test setup overhead.Per-suite docker-compose up (Step 7).
Not verifying the WireMock request logTest passes even if SUT didn't actually retry (just got lucky).Assert on __admin/requests count (Step 4 example).

Limitations

  • Setup complexity. Two containers + control APIs + scenario state. Not lightweight.
  • TCP-only chaos via Toxiproxy. UDP / QUIC / DNS-level failures need different tools.
  • HTTP fault realism via WireMock. Some payload-level faults (binary protocol corruption) need custom tooling.
  • Doesn't replace production chaos. This is integration-test chaos; production chaos engineering is separate (per chaos-experiment-author).

References

  • wiremock-stubs - HTTP fault stub primitive this skill orchestrates.
  • toxiproxy-chaos - TCP-level fault primitive this skill orchestrates.
  • api-chaos-runner (in the qa-api-testing plugin) - sister: pure Toxiproxy + test-suite matrix.
  • chaos-experiment-author - methodology for the chaos-experiment shape.

Related skills

chaos-drill-protocol

Run protocol and run workflow for a chaos experiment that has already been designed: the four pre-flight gates (non-production target, measured healthy baseline, live observability, a rollback that has actually been exercised), how to pick a conservative blast-radius bound, the sampling cadence and abort criteria fixed in writing before injection, the per-runner inject and abort commands (Chaos Mesh / Litmus / Gremlin / Toxiproxy), the refuse-to-start rules (no blast-radius bound, production context, degraded baseline, offline observability, unexercised rollback), and the recovery-validation step with its tolerance and timeout. Owns execution safety only, not experiment design: the steady-state hypothesis, the fault to inject, and the experiment file come from chaos-experiment-author. Use when an experiment definition exists and a fault is about to be injected into a running system, and the go/no-go gates, abort thresholds, and recovery check still need to be agreed and written down before the fault starts.

chaos-experiment-author

Build-an-X workflow for a chaos experiment per the Principles of Chaos Engineering - defines steady-state hypothesis, picks the variables (real-world events: network latency, node failure, region outage), sets the blast radius (which percentage / namespace / user cohort), automates execution, and emits the verdict (steady-state held / didn't hold). Includes the five-check pre-flight validation of the steady-state hypothesis (measurable, baselined, SLI-backed tolerance, defined measurement window, metric moves under the fault) with hard-reject rules, and routes the tool choice: Chaos Mesh has its own standalone skill, while LitmusChaos and Gremlin setup live in this skill's references. Use to scope and pre-flight-validate a chaos experiment before running it via Chaos Mesh / Litmus / Gremlin / Toxiproxy.

chaos-mesh

Configures Chaos Mesh for Kubernetes-native chaos engineering - picks fault types (PodChaos, NetworkChaos, StressChaos, IOChaos, TimeChaos, DNSChaos, KernelChaos, HTTPChaos), targets via label selectors, controls blast radius via namespace whitelists + selector filters, schedules via CronJobs, observes via dashboard. Distinct from Litmus by architecture (Chaos Mesh has its own dashboard + workflow orchestration; Litmus uses ChaosCenter UI). Use when the target system runs on Kubernetes and fault experiments should be declared as CRDs in the cluster alongside the workloads they target.

dr-drill-runner

The full DR-drill discipline for one service: author the runbook (per-tier RTO + RPO), pre-drill checklist (data sync state, alert silencing, customer comms), drill workflow (announce, fail-over, verify, fail-back) with timestamps, the supervised run protocol (refuse without declared RTO/RPO or against production, RTO/RPO monitoring cadence, abort-on-breach), and an auditor-ready post-drill report. Backup-integrity verification (SHA-256 + signature, restore spot checks, cross-region replication, retention, key recovery) and restore-time / RTO measurement (TTF segments, PITR latency, parallel-restore tuning, trend tracking) are worked in references. Per Google Cloud DR planning guide; covers cold / warm / hot standby tier-specific patterns. Use when a scheduled or post-incident failover drill for one service is being planned, executed, or written up, or when a new tier-1 service ships without a drill defined.

error-budget-tests

Build error-budget gate tests - SLO + error-budget calculation per Google SRE workbook ("difference between target uptime and actual uptime"); burn-rate alerting; monthly-budget exhaustion test; freeze-trigger when budget consumed. Per sre.google embracing-risk reference. Includes the incident-metrics reference for MTTR / MTBF / MTTD / MTTA - per-incident record schema, calculation formulae, exclusion rules, dashboards-as-code, and target-vs-actual alerting. Use when an SLO and error budget are written down but nothing verifies that burn-rate alerts fire or that the release freeze engages when the budget runs out, or when MTTR / MTBF dashboards report numbers nobody can reproduce.

toxiproxy-chaos

Configures Toxiproxy for TCP-level fault injection - runs as a sidecar / proxy between client and upstream, applies toxics (latency, bandwidth, slow_close, timeout, slicer, limit_data, reset_peer) via control API. Focused on the proxy itself rather than an API-level chaos runner, including non-test usage (chaos in dev environments, integration tests, pre-prod simulation). Use when the team needs TCP-precise fault injection in development / integration environments without K8s or commercial tooling.