Chaos Toolkit steady-state-hypothesis block and tolerance forms

Reference for the steady-state hypothesis validation section of chaos-experiment-author. The five pre-flight checks there validate a hypothesis expressed in this schema; this file is the schema and tolerance detail they assume.

The steady-state-hypothesis object

Per chaostoolkit.org/reference/api/experiment/ (opens in new window), the steady-state-hypothesis object requires:

• title (string): human-readable rationale for the hypothesis.
• probes (array): one or more probe objects, each with:
  • type: "probe"
  • name: identifier string
  • provider: execution specification (HTTP, process, or Python)
  • tolerance: the gate value; if the probe's return value does not satisfy the tolerance, the experiment bails before running the method.

Tolerance forms supported

Per chaostoolkit.org/reference/api/experiment/ (opens in new window):

Execution flow

Per chaostoolkit.org/reference/concepts/ (opens in new window): probes run once before the method (baseline check) and once after (deviation check). A probe that fails before the method means the system is already outside its acceptable state; the experiment must not run. A probe that fails after the method means the chaos activity caused the system to leave its steady state.

Chaos experiment authoring reference

Supporting detail for chaos-experiment-author. Step 2 uses the event catalog, Step 7 uses the verdict report template, and the anti-patterns table lists the mistakes each step guards against.

Real-world event catalog (Step 2)

Per principlesofchaos.org (opens in new window) principle 2, vary real-world events: inject what could plausibly happen, not "anything". Pick events the team has already seen in real incidents or realistically expects.

Verdict report template (Step 7)

## Chaos experiment verdict - `checkout-network-latency`

**Date:** YYYY-MM-DD   **Duration:** 5 minutes
**Steady-state hypothesis:** checkout_completion_rate >= 95%
**Verdict:** HELD

| Metric                   | Pre-experiment | During experiment | Post |
|--------------------------|----------------|-------------------|------|
| checkout_completion_rate |     97.2%      |       96.8%       | 97.5% |
| p95 latency              |     245ms      |       380ms       | 240ms |

### Observations
- Latency increased as expected (300ms injected).
- Retry logic worked: ~200 retries observed; no user-visible failures.

### Action items
- (none - system behaved as expected)

### Next iteration
- Increase blast radius from 1% to 5% in next month's run.
- Add a longer-duration variant (15 min) to test fatigue.

Anti-patterns

Gremlin advanced operations - attacks, Scenarios, Reliability Score, CI, compliance

Deep reference for gremlin.md (opens in new window). Consult when picking a specific attack, chaining attacks into a Scenario, wiring Gremlin into CI, reading the Reliability Score, or satisfying an audit / compliance requirement.

Per gremlin-home (opens in new window):

Full attack table

Per gremlin-home (opens in new window) and the broader Gremlin docs, the four attack classes expand into these individual attacks:

Running an attack via the web UI

Web UI workflow:

1. Select target (host / container / service / Lambda).
2. Pick attack type.
3. Configure (e.g., latency 500ms; duration 5min).
4. Optionally schedule.
5. Click "Unleash."

The UI provides safety: blast-radius scoping, abort button, notifications.

Authoring a Scenario

A Scenario chains multiple attacks:

# Pseudo-Scenario config (Gremlin's UI exports JSON; this approximates)
scenario:
  name: "Checkout resilience test"
  attacks:
    - type: latency
      target: { service: checkout }
      length: 5min
      latency: 500ms
    - type: packet-loss
      target: { service: payment }
      length: 5min
      loss-percent: 10
      delay-after-previous: 1min
  abort_conditions:
    - "Sentry error rate > 2%"
    - "Manual abort"

Scenarios match per the chaos-experiment-author "vary real-world events" principle - combinations approximate real incidents.

Reliability Score

Per gremlin-home (opens in new window), Gremlin's differentiator is the "Reliability Score" - "individual services" get scores "based on dependency mapping, risk detection, and failure testing."

Score components (per Gremlin docs):

• Resilience tests passed: % of attacks the service survived
• Dependency map: service-to-service relationships
• Detected risks: configuration drift, hidden dependencies

A service moving from "untested" to "score 80" via passing attacks creates an objective improvement signal.

API + CI automation

Trigger an attack directly from the API:

curl -X POST "https://api.gremlin.com/v1/attacks/new" \
  -H "Authorization: Key $GREMLIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "command": {
      "type": "latency",
      "args": ["-l", "300", "-m", "500", "-c", "5", "-h", "^api\\.example\\.com$"]
    },
    "target": {
      "type": "Random",
      "containers": { "labels": { "app": "checkout" } }
    }
  }'

The API enables CI integration - trigger a saved Scenario, wait for it to run, then evaluate a monitoring-driven verdict:

- name: Trigger Gremlin scenario
  run: |
    curl -X POST "https://api.gremlin.com/v1/scenarios/${{ vars.SCENARIO_ID }}/runs" \
      -H "Authorization: Key ${{ secrets.GREMLIN_API_KEY }}"
- name: Wait + verdict
  run: sleep 600 && ./scripts/datadog-verdict.sh

Compliance + audit

Gremlin's enterprise tier (per gremlin-home (opens in new window)'s positioning) provides:

• Audit logs (who triggered what, when).
• RBAC at organization / team / role level.
• SOC 2 / FedRAMP / etc. compliance posture.

Important for regulated industries where audit is non-negotiable.

References

• gh (opens in new window) - Gremlin overview: enterprise reliability platform, reliability scoring, multi-platform fault injection.
• chaos-experiment-author - methodology Gremlin Scenarios implement.

Gremlin runner

Deep dive for chaos-experiment-author Step 4. Configures Gremlin (commercial) for cross-platform chaos engineering - agent install on Linux / Windows / Kubernetes, the four attack classes, Scenarios, and the Reliability Score. Use when the platform spans multiple environments (bare metal + cloud

• serverless) and the team needs a commercial-supported solution.

Overview

Gremlin is a commercial reliability platform for fault injection across bare metal, on-prem, multi-cloud, and serverless. It assigns each service a forward-looking Reliability Score from repeated resilience tests, so teams fix likely failure points before an incident (per gremlin-home (opens in new window)).

When to use

• The platform spans multiple environments (not just Kubernetes - Gremlin's differentiator vs LitmusChaos / Chaos Mesh).
• Enterprise support is required (compliance, audit, SLA).
• The team wants reliability scoring (vs just per-experiment pass/fail).
• The team is in regulated industry (finance, healthcare) needing the compliance posture.

If the team is K8s-only and OSS-preferred, see litmus.md (opens in new window) or chaos-mesh.

How to use

1. Install the Gremlin agent on the target host or cluster (see Install) and register it with the Gremlin Control Plane.
2. Pick an attack type from the four classes (resource, network, state, request) - the exhaustive per-attack table is in gremlin-advanced-operations.md (opens in new window).
3. Verify before injecting: assert the target is in steady state (error rate and p95 latency healthy on the dashboard) and the blast radius is scoped to a single container in staging; if either check fails, do not inject - fix the scope or wait for steady state to return.
4. Run one scoped experiment end to end against staging - inject a single fault and attach an abort condition that halts the attack the moment the steady-state metric breaches its threshold (see Worked example).
5. Verify the abort path fires: confirm the attack actually stops when the abort condition trips; if it does not halt on breach, fix the abort wiring (monitor query, threshold, or notification hook) before widening the blast radius.
6. Promote passing experiments into a Scenario (chained attacks + abort conditions), wire it into CI via the API, and track each service's Reliability Score - all covered in gremlin-advanced-operations.md (opens in new window).

Install

Linux:

sudo apt install -y gremlin
sudo gremlin auth login --org-id <org-id> --user-id <user-id> --api-token <token>

Kubernetes:

helm repo add gremlin https://helm.gremlin.com
helm install gremlin gremlin/gremlin \
  --namespace gremlin --create-namespace \
  --set gremlin.secret.create=true \
  --set gremlin.secret.teamID=<team-id> \
  --set gremlin.secret.clusterID=<cluster-id> \
  --set gremlin.secret.teamSecret=<secret>

The agent connects to the Gremlin Control Plane (cloud); attacks trigger via web UI or API.

Attack types

Gremlin groups fault injections into four classes (per gremlin-home (opens in new window) and the Gremlin docs):

The full per-attack table (all twelve attacks with their exact effect) lives in gremlin-advanced-operations.md (opens in new window).

Worked example

A single end-to-end experiment: inject 500ms latency into the checkout service, scoped to one container for five minutes.

1. Steady state. Confirm from monitoring that checkout error rate is under 1% and p95 latency is healthy.
2. Hypothesis. A 500ms upstream latency injection keeps the error rate under 2% (retries + timeouts absorb it).
3. Inject the fault via the API, scoped tight (one container, capped at 5 minutes):

curl -X POST "https://api.gremlin.com/v1/attacks/new" \
  -H "Authorization: Key $GREMLIN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "command": {
      "type": "latency",
      "args": ["-l", "300", "-m", "500", "-c", "1", "-h", "^checkout\\..*$"]
    },
    "target": {
      "type": "Random",
      "percent": 10,
      "containers": { "labels": { "app": "checkout" } }
    }
  }'

1. Observe + abort. Watch the error rate for the five-minute window; the UI halt button (or a monitored abort condition) stops the attack the moment error rate crosses 2%.
2. Verdict. Error rate held at 1.2% - checkout tolerates 500ms upstream latency. Record the pass against the service's Reliability Score, then widen the blast radius on the next run.

Scenarios (chaining this latency attack with a downstream packet-loss attack), the Reliability Score model, the full CI workflow, and the compliance / audit posture are in gremlin-advanced-operations.md (opens in new window).

Anti-patterns

Limitations

• Commercial cost. Subscription model; per-team / per-host pricing. Not suitable for OSS budgets.
• Cloud control plane. Air-gapped environments need on-prem deployment.
• Vendor lock-in. Scenarios + Reliability Score data lives in Gremlin; migration cost real.
• Less Kubernetes-deep than Chaos Mesh / Litmus. Gremlin abstracts platform; loses some K8s-specific power.

References

• gh (opens in new window) - Gremlin overview: enterprise reliability platform, forward-looking reliability scores, multi-platform (bare metal / on-prem / multi-cloud / serverless), fault injection + reliability scoring + dependency discovery.
• gremlin-advanced-operations.md (opens in new window) - exhaustive attack table, UI attack workflow, Scenario authoring, Reliability Score model, the API + CI automation workflow, and the compliance / audit posture.
• litmus.md (opens in new window), chaos-mesh - open-source K8s-only alternatives.
• The host chaos-experiment-author SKILL.md - methodology Gremlin Scenarios implement.

LitmusChaos runner

Deep dive for chaos-experiment-author Step 4. Configures LitmusChaos for Kubernetes-native chaos engineering. Prefer over chaos-mesh when the team wants a ChaosCenter web UI for workflow scheduling and ChaosHub catalog browsing; use chaos-mesh for fine-grained network-fault policies via its own CRD family.

Overview

Per litmus-home (opens in new window):

"LitmusChaos is a CNCF-hosted, open-source Chaos Engineering platform that helps teams identify infrastructure weaknesses through safe, controlled chaos tests."

"Kubernetes developers & SREs use Litmus to manage chaos in a declarative manner." (litmus-home (opens in new window))

The architecture: Litmus runs as a Kubernetes operator; experiments are CRDs; results export to Prometheus.

When to use

• The platform is Kubernetes (Litmus is K8s-native).
• The team wants CNCF / open-source chaos tooling (vs commercial Gremlin).
• A chaos experiment's outcome should integrate with existing K8s observability (Prometheus, Grafana).

Step 1 - Install

Per litmus-home (opens in new window):

helm repo add litmuschaos https://litmuschaos.github.io/litmus-helm/
helm install litmuschaos litmuschaos/litmus -n litmus --create-namespace

The Litmus operator + ChaosCenter (web UI) deploy.

Step 2 - Pick a ChaosExperiment from the Hub

Per litmus-home (opens in new window), the ChaosHub is "a repository hosting most of the chaos experiments that are needed for a quick start in Chaos Engineering." Common experiments:

Install per-experiment:

kubectl apply -f https://hub.litmuschaos.io/api/chaos/2.14.0?file=charts/generic/pod-delete/experiment.yaml

Step 3 - Author a ChaosEngine

The ChaosEngine CR runs an experiment against a target:

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: checkout-pod-delete
  namespace: app
spec:
  appinfo:
    appns: app
    applabel: 'app=checkout'
    appkind: deployment
  chaosServiceAccount: pod-delete-sa
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: '60'      # seconds
            - name: CHAOS_INTERVAL
              value: '20'       # seconds
            - name: PODS_AFFECTED_PERCENTAGE
              value: '50'
        probe:
          - name: 'check-checkout-availability'
            type: httpProbe
            httpProbe/inputs:
              url: 'http://checkout.app.svc:8080/health'
              insecureSkipVerify: false
              method:
                get:
                  criteria: '=='
                  responseCode: '200'
            mode: 'Continuous'
            runProperties:
              probeTimeout: 5
              interval: 2
              retry: 3
              probePollingInterval: 1

Per litmus-home (opens in new window), probes "create complete chaos scenarios close to the real application experience upon failure." The probe is the steady-state check per the chaos principles.

Step 4 - Run

kubectl apply -f checkout-pod-delete.yaml

Litmus runs the experiment for TOTAL_CHAOS_DURATION seconds, checking the probe continuously. The verdict (Pass / Fail) lands in chaosengine.status.experimentStatus.verdict.

Step 5 - Read the verdict

kubectl get chaosengine checkout-pod-delete -o jsonpath='{.status.experimentStatus.verdict}'
# Output: Pass | Fail

Step 6 - Probe types

Probes can run in different modes: SOT (start of test), EOT (end of test), Edge (both), Continuous (every N seconds during the experiment).

Step 7 - Observability

Per litmus-home (opens in new window): "chaos observability by exporting Prometheus metrics that highlight and quantify the impact of chaos on the applications or infrastructure in real time."

Key metrics:

• litmuschaos_passed_experiments
• litmuschaos_failed_experiments
• litmuschaos_awaited_experiments

Wire to Grafana for dashboards.

Step 8 - CI integration

- name: Run chaos experiment
  run: |
    kubectl apply -f experiments/checkout-pod-delete.yaml
    kubectl wait --for=condition=Complete chaosengine/checkout-pod-delete --timeout=10m
    VERDICT=$(kubectl get chaosengine checkout-pod-delete -o jsonpath='{.status.experimentStatus.verdict}')
    echo "Verdict: $VERDICT"
    [ "$VERDICT" = "Pass" ]

Anti-patterns

Limitations

• Kubernetes only. No native non-K8s support (vs Gremlin's multi-platform).
• ChaosHub experiments need vetting. Community experiments vary in quality.
• Cluster overhead. Litmus operator + per-experiment pods consume resources.
• Per-tool incompatibility. Litmus ChaosEngines aren't Chaos-Mesh CRDs; experiments don't port.

References

• lh (opens in new window) - LitmusChaos overview, CNCF-hosted, ChaosExperiments
  • ChaosEngine + ChaosHub + probes, Prometheus metrics export.
• chaos-mesh - sibling K8s-native alternative.
• gremlin.md (opens in new window) - commercial multi-platform alternative.
• The host chaos-experiment-author SKILL.md - methodology this tool implements.