schemathesis-fuzzing

Overview

Schemathesis is a property-based API testing tool that "automatically generates property-based tests from your OpenAPI or GraphQL schema and exercises the edge cases that break your API" (schemathesis-readme). The schema is the single source of truth - every endpoint, parameter, and response shape becomes a generator that produces hundreds of targeted variations per run.

This is complementary to example-based API testing (postman-collections, tavern-testing, restassured-testing, karate-testing) - example-based tests verify happy paths; Schemathesis attacks the boundaries the team forgot.

When to use

• The repo has an OpenAPI 2.0 / 3.0 / 3.1 / 3.2 spec or a GraphQL schema (June 2018+).
• The team wants automatic test coverage of new endpoints without authoring per-endpoint tests.
• Negative-scenario coverage ("what happens with a malformed request?") is missing or thin.
• A CI gate on 5xx responses (any unexpected server error) is desired.

If the API has no schema, Schemathesis cannot help - generate a schema first (FastAPI, Flask-Smorest, NestJS Swagger, dropwizard, or hand-author a spec), then return.

Install

pip install schemathesis

(Per schemathesis-readme.)

For the latest CLI without modifying the project's Python env:

uvx schemathesis run <schema-url>

(Adapted from schemathesis-docs - uvx is the uv runner for one-off tool execution.)

Running via CLI

Basic invocation per schemathesis-readme:

schemathesis run <schema-url>

<schema-url> can be:

• Remote URL: https://api.example.com/openapi.json.
• Local file: ./openapi.yaml.

Key flags

Per schemathesis-readme:

Worked example

schemathesis run https://api.example.com/openapi.json \
  --base-url https://staging.example.com \
  --checks status_code_conformance \
  --checks response_schema_conformance \
  --checks not_a_server_error \
  --hypothesis-max-examples 200 \
  --workers 4 \
  --header "Authorization: Bearer $API_TOKEN"

Built-in checks

Per schemathesis-readme, the canonical checks:

Run all checks (default), or restrict via repeated --checks:

# Strict: every check active
schemathesis run <schema>

# Only flag 5xx errors (cheap smoke test)
schemathesis run <schema> --checks not_a_server_error

A failing check produces a deterministic reproduction - Schemathesis prints the exact curl command and Hypothesis seed to reproduce the generated request.

Pytest integration

For projects that want Schemathesis cases as first-class pytest tests (schemathesis-readme):

# tests/api/test_schemathesis.py
import schemathesis

schema = schemathesis.openapi.from_url("https://your-api.com/openapi.json")

@schema.parametrize()
def test_api(case):
    case.call_and_validate()

@schema.parametrize() generates one pytest test per endpoint × method combination. case.call_and_validate() issues the generated request and runs every default check.

For finer-grained integration:

@schema.parametrize()
@schemathesis.hook("before_call")
def add_auth(context, case):
    case.headers["Authorization"] = f"Bearer {os.environ['API_TOKEN']}"

def test_api_with_auth(case):
    case.call_and_validate()

Hooks let the team inject project-specific auth, header signatures, or response post-processing without forking Schemathesis.

CI integration

# .github/workflows/api-fuzz.yml
name: api-fuzz

on:
  pull_request:
  push:
    branches: [main]
  schedule:
    - cron: '0 6 * * *'   # nightly broader run

jobs:
  schemathesis:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5

      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - run: pip install schemathesis

      - name: Schemathesis run
        env:
          API_TOKEN: ${{ secrets.STAGING_API_TOKEN }}
        run: |
          schemathesis run https://staging.example.com/openapi.json \
            --base-url https://staging.example.com \
            --hypothesis-max-examples 50 \
            --workers 4 \
            --header "Authorization: Bearer $API_TOKEN" \
            --junit-xml=results.xml

      - name: Upload report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: schemathesis-results
          path: results.xml
          retention-days: 14

      - name: Surface JUnit
        if: always()
        uses: dorny/test-reporter@v1
        with:
          name: API fuzz
          path: results.xml
          reporter: java-junit

The PR-trigger run uses lower --hypothesis-max-examples (e.g. 50) for fast feedback; the nightly cron run uses higher values (200+) for deeper coverage. This separation keeps PR CI under 10 minutes without sacrificing nightly depth.

Two complementary CI cadences

Hypothesis (the underlying property-based engine) shrinks failures to minimal reproducers automatically - a 200-example PR run is plenty to surface most regressions; nightly depth catches the rare ones.

Anti-patterns

Limitations

• Authentication state. Schemathesis can attach a token but doesn't model multi-step auth flows (login → token → use). Combine with tavern-testing or restassured-testing for the auth chain.
• Stateful sequences. Out of the box, every Schemathesis case is independent. For stateful API fuzzing (POST → GET created resource → DELETE), use restler-fuzzing, which is built for stateful sequences.
• Schema drift. If the schema doesn't match the implementation, every case fails for the wrong reason. Run a Schemathesis baseline pre-PR to catch schema drift early.
• OpenAPI 2.0 limitations. Some OpenAPI 2.0 features (e.g. formData) generate weaker variations than 3.x; consider migrating the schema to 3.x.

References

• schemathesis-readme - main repo: install, CLI flags, built-in checks, pytest integration with @schema.parametrize().
• schemathesis-docs - full docs (CLI reference, hooks, authentication strategies).
• restler-fuzzing - stateful fuzzing complement.
• postman-collections, tavern-testing, restassured-testing, karate-testing - example-based authoring; use alongside Schemathesis for happy-path coverage.