Testland
Browse all skills & agents

negative-test-generator

Generates negative / error-path test cases that mirror happy-path tests - for each happy-path test, produces companions exercising input validation rejection, missing required fields, type mismatches, authorization failures, rate-limit errors, and adversarial payloads from the malicious-payload-bank. Emits cases as parameterized tests in the project's runner format. Use when a feature has happy-path coverage but the rejection / error / unauthorized paths are untested.

Install with skills.sh (any agent)

npx skills add testland/qa --skill negative-test-generator
View source

negative-test-generator

Overview

A typical test suite has a happy-path test for every endpoint: "valid input → 200 / created object / etc." What's missing is the rejection path - what happens with malformed input, missing fields, wrong type, unauthorized user, rate-limited request, adversarial payload?

This skill takes a happy-path test and emits its negative companions - one per failure mode. The result is a paired set where every "should accept X" has a sibling "should reject Y" catalog.

When to use

  • A feature has a happy-path test but no error-path coverage.
  • An API endpoint accepts user input without documented rejection testing.
  • A team is migrating from "test only what works" to "test every documented response code."
  • Pairs with acceptance-criteria-extractor (in the qa-shift-left plugin) output: every Then-clause's positive assertion gets a negative sibling.

Negative-path categories

For each happy-path test of <verb> <path> with body fields <f1, f2, ...>, generate companions across these categories:

  1. Schema violations - missing required, wrong type, out-of-range, wrong enum, bad format, extra/unknown field, null-where-forbidden, excess length.
  2. Authentication failures - no / expired / malformed / cross-tenant token; expect 401.
  3. Authorization failures - missing permission, IDOR, cross-tenant access; expect 403 / 404.
  4. Rate / quota failures - burst over limit (429), plan quota exceeded.
  5. Conflict / state errors - duplicate (409), optimistic-lock (412), invalid state transition (422 / 409).
  6. Adversarial inputs - XSS, SQLi, SSRF, path traversal, ReDoS, Unicode confusables from malicious-payload-bank. Assert reject (4xx) or escaped output - never that the payload merely "didn't crash".
  7. Server errors - upstream unavailable (502 / 503) or timeout (504) via api-chaos-runner.

Per-pattern test tables and the starter adversarial-payload catalog (with CWE / OWASP references): references/negative-path-catalog.md.

Worked example

Given a happy-path test:

def test_create_order_succeeds():
    response = post('/api/orders',
                    headers={'Authorization': f'Bearer {token}'},
                    json={'sku': 'SKU-1', 'qty': 2})
    assert response.status_code == 201
    assert response.json()['order_id']

The skill emits negative companions:

import pytest
from tests.fixtures import token, expired_token, free_tier_token
from tests.payloads import XSS, SQLI

# 1. Schema violations
@pytest.mark.parametrize('body,expected_status,expected_field', [
    ({'qty': 2},                  400, 'sku'),         # missing required
    ({'sku': 'SKU-1'},            400, 'qty'),         # missing required
    ({'sku': 'SKU-1', 'qty': 'two'}, 400, 'qty'),    # wrong type
    ({'sku': 'SKU-1', 'qty': 0},  400, 'qty'),         # below min
    ({'sku': 'SKU-1', 'qty': -1}, 400, 'qty'),         # negative
    ({'sku': 'SKU-1', 'qty': 1000000}, 400, 'qty'),    # above max
    ({'sku': 'INVALID-SKU', 'qty': 2}, 400, 'sku'),     # unknown enum
])
def test_create_order_schema_rejects(body, expected_status, expected_field):
    response = post('/api/orders',
                    headers={'Authorization': f'Bearer {token}'},
                    json=body)
    assert response.status_code == expected_status
    assert expected_field in response.json().get('errors', {})

# 2. Auth failures
def test_create_order_no_token():
    response = post('/api/orders', json={'sku': 'SKU-1', 'qty': 2})
    assert response.status_code == 401

def test_create_order_expired_token():
    response = post('/api/orders',
                    headers={'Authorization': f'Bearer {expired_token}'},
                    json={'sku': 'SKU-1', 'qty': 2})
    assert response.status_code == 401

# 3. Authorization failures
def test_create_order_free_tier_blocked_for_premium_sku():
    response = post('/api/orders',
                    headers={'Authorization': f'Bearer {free_tier_token}'},
                    json={'sku': 'PREMIUM-1', 'qty': 1})
    assert response.status_code == 403

# 6. Adversarial payloads
@pytest.mark.parametrize('payload', XSS + SQLI)
def test_create_order_handles_adversarial_sku(payload):
    response = post('/api/orders',
                    headers={'Authorization': f'Bearer {token}'},
                    json={'sku': payload, 'qty': 1})
    # Either reject (preferred) or escape - never pass through with execution
    assert response.status_code in (400, 404, 422)
    assert '<script>' not in response.text

Output format

## Negative tests for `<endpoint>` - `<verb> <path>`

**Happy path:** `tests/<file>::test_<happy_name>`
**Negative companions generated:** N (across 6 categories)

### Tests by category

| Category               | Count | File                                                |
|------------------------|------:|-----------------------------------------------------|
| Schema violations      |    7  | `tests/<file>::test_<endpoint>_schema_rejects`     |
| Auth failures          |    2  | `tests/<file>::test_<endpoint>_no_token` etc.       |
| Authorization failures |    1  | `tests/<file>::test_<endpoint>_unauthorized`        |
| Rate/quota failures    |    0  | (skipped; rate-limit not yet implemented)           |
| Conflict/state errors  |    1  | `tests/<file>::test_<endpoint>_duplicate`           |
| Adversarial payloads   |    8  | `tests/<file>::test_<endpoint>_handles_adversarial`|

### Skipped categories

- Rate/quota failures: rate limiting not yet implemented; revisit when added.
- Server errors: covered by `api-chaos-runner` in a separate suite.

### Recommended next step

1. Run the new negative tests; expect all to pass given the
   documented behavior.
2. Any failure indicates a real gap - either the validator is
   missing the case OR the assertion is wrong.
3. For passes: commit.

Anti-patterns

Anti-patternWhy it failsFix
Negative tests that assert vague status codes ("not 200")The test passes if the server returns 500 (a bug); should fail.Always assert specific status: 400 for validation, 401 for auth, 403 for authz, 429 for rate, etc.
Skipping the "wrong type" casesThe most common validator bug class.Always include type-mismatch cases for every field.
One mega-test that covers all negativesFailure attribution unclear; one failure cascades.Parameterize per category; one test function per category.
Negative-path coverage but no _synthetic dataReal-looking PII flows through; compliance issue if logs leak.Use synthetic-pii-generator for any field that could be PII.
Asserting 'error' in response.bodyBrittle; error-message text changes; locale-specific bugs.Assert structured field: response.json()['errors']['<field>'] or response.headers['X-Error-Code'].

Limitations

  • Doesn't auto-detect input domain. The team must declare required fields, types, ranges, and enum values for the skill to generate the matching cases. Use acceptance-criteria-extractor
    • non-functional-requirement-extractor upstream.
  • Server-error simulation requires mocking. Categories 5 and 7 need api-chaos-runner or wiremock-stubs / msw-handlers for the upstream failure simulation.
  • Per-language output styling. Test scaffolding adapts to pytest / Jest / xUnit / JUnit; the skill emits the project's preferred runner format.

References

  • acceptance-criteria-extractor - upstream skill producing the happy-path AC.
  • malicious-payload-bank - adversarial payload catalog (category 6).
  • boundary-value-generator - sibling skill for boundary cases.
  • synthetic-pii-generator - for any test data that includes PII.

Negative-path catalog

For each happy-path test of a request <verb> <path> with body fields <f1, f2, ...>, the negative-test set covers these categories.

1. Schema violations (validation rejection)

PatternTest
Missing required fieldSend the request without <f1>; expect 400 with <f1> named.
Wrong typeSend <f1> as wrong type (string where int expected); expect 400.
Out-of-range valueSend <f1> outside [min, max] per boundary-value-generator; expect 400.
Wrong enum valueSend <f1> with a value not in enum_values; expect 400.
Wrong formatString that fails the regex / format constraint; expect 400.
Extra unknown fieldSend field not in the schema; behavior depends on policy (strict reject vs. ignore).
Empty / null where forbiddenSend null or empty string for a non-nullable field.
Excess length / countSend string longer than max-length; collection larger than max-count.

2. Authentication failures

PatternTest
No Authorization headerExpect 401.
Expired tokenExpect 401 with token expired indication.
Malformed tokenExpect 401, no info leak about token shape.
Token from a different tenantExpect 401 / 403.

3. Authorization failures

PatternTest
User without permissionAuthenticated but role lacks required permission; expect 403.
Resource owned by another userRead/write attempt on someone else's resource (IDOR); expect 403 / 404.
Cross-tenant accessAttempt resource from another organization; expect 403 / 404.

4. Rate / quota failures

PatternTest
Burst exceeds rate limitSend N+1 requests in window; expect 429.
Plan quota exceededFree-tier user exceeds Free quota; expect 403 / 429 with quota info.

5. Conflict / state errors

PatternTest
Duplicate creationCreate same resource twice; expect 409.
Optimistic-lock conflictStale If-Match; expect 412.
State machine violationInvalid status transition (e.g. placed → completed skipping shipped); expect 422 / 409.

6. Adversarial inputs

Pull from malicious-payload-bank:

Field typePayload classes
Free-text inputXSS, SQLi, Unicode confusables
URL fieldSSRF, open-redirect
File upload namePath traversal
Search fieldReDoS, SQLi

Starter payloads per class (aligned with the OWASP Top 10 (opens in new window) and CWE Top 25 (opens in new window); the full catalog with encoded variants and per-context selection lives in malicious-payload-bank):

ClassStarter payloads
SQLi (CWE-89)' OR '1'='1' -- · '; DROP TABLE users; -- · admin'--
XSS (CWE-79)<script>alert(1)</script> · <img src=x onerror=alert(1)> · javascript:alert(1)
SSRF (CWE-918)http://169.254.169.254/latest/meta-data/ · http://localhost:6379/ · file:///etc/passwd
Path traversal (CWE-22)../etc/passwd · ..%2fetc%2fpasswd · ....//etc/passwd
ReDoS (CWE-1333)aaaaaaaaaaaaaaaaaaaaaaaa! (vs /^(a+)+$/) · aaaaaaaaaaaaaaaaaaaaaaaa@aaaaaa (vs typical email regexes)
Unicode confusablesаdmin (Cyrillic а U+0430) · gооgle.com · (ff ligature)

Assert reject (4xx) or escaped output - never that the payload merely "didn't crash".

7. Server errors (where applicable)

PatternTest
Upstream dependency unavailableMock the dependency to fail (per api-chaos-runner); expect 502 / 503 with retry hint.
TimeoutMock slow upstream; expect 504 / circuit-breaker fallback.

Related skills

bogus-data

Authors .NET test fixtures using the Bogus library - fluent typed `Faker` builders with `.RuleFor` per property, generation via `Generate()` / `GenerateBetween(min, max)` / `GenerateLazy()`, and `UseSeed()` for reproducibility. Provides the Bogus equivalent of Python's Faker / Ruby's FactoryBot. Use when the project is C# / F# / VB.NET and the team needs typed fixture creation.

boundary-value-generator

Generates boundary-value test cases from typed input specifications - for each input field, produces the canonical 6-point set (one below, at, and above the lower bound; one below, at, and above the upper bound) plus equivalence-class representatives. Emits cases as parameterized test inputs (pytest @parametrize / Jest test.each / xUnit InlineData / etc.). Use when a function or endpoint has numeric / string-length / collection-size constraints and the team needs systematic edge-case coverage.

e2e-test-narrative-builder

Assembles a multi-step end-to-end user-journey test from a list of high-level user intents - translates each intent ("user signs up", "user adds product to cart", "user completes checkout with promo code") into the corresponding test-runner step (Playwright / Cypress / Selenium / Karate), wires shared state across steps via test fixtures, and emits the resulting test as a single Scenario in the project's E2E framework. Use when scaffolding an E2E test that exercises a complete user flow rather than a single page.

factory-bot-data

Authors Ruby FactoryBot factories with traits, associations, sequences, and the three build strategies (build / create / build_stubbed); integrates with RSpec / Minitest test suites; pairs with Faker for randomized field values. Use when the project is Ruby / Rails and needs structured fixture creation with referential integrity.

faker-data

Authors test-data factories using Faker: the Python `faker` library, the `@faker-js/faker` JS port, and the `faker-ruby` gem. Owns the library mechanics end to end: install per language, the provider catalogue (person / internet / location / date / finance / lorem), locale selection and multi-locale mode, and seed-based determinism for reproducible runs. Scope is generating fresh values for tests that start from nothing, not replacing values inside an existing dataset that already holds real records, which raises referential-integrity and re-identification concerns this skill does not address. Prefer this skill when the codebase already uses the Faker family or when cross-language consistency across Python, JS, and Ruby matters; use mimesis-data only when deeper Python locale coverage is the primary requirement. Use when authoring fixtures or factories that need realistic-looking field values.

golden-file-conventions

Reference catalog for snapshot / golden file management - naming conventions, directory layout, when to add / update / remove a baseline, sanitization (timestamps, IDs, PII), per-OS / per-runtime variant strategy, and review workflow for snapshot diffs in PRs. Use when designing a snapshot-testing convention or auditing an existing one for drift.

malicious-payload-bank

Reference catalog of curated adversarial input payloads keyed by attack class - SQL injection, XSS, SSRF, path traversal, command injection, XXE, prototype pollution, regex DoS, Unicode confusables, header injection - plus per-context guidance for which payloads apply (URL parameter / form input / JSON body / file upload). Use when authoring negative-test cases for input validation, fuzz targets, or a security-focused test suite that needs to exercise the OWASP Top 10 attack surface.

mimesis-data

Authors Python test fixtures using mimesis - a fast, type-hinted, locale-aware test-data generator with 46 locales - covering Person / Address / Internet / Datetime providers and the Schema/Field pattern for typed-dict generation. Pairs with factory_boy when referential integrity is needed. Use when the project is Python and the team values speed, type hints, or strong locale coverage over Faker's larger ecosystem.

mountebank-imposters

Authors Mountebank imposters (multi-protocol mock servers - HTTP, HTTPS, TCP, SMTP, LDAP, gRPC, WebSockets, GraphQL, and more) by POSTing JSON definitions to the Mountebank control API on port 2525, configures stubs with predicates and responses, and uses record-playback proxy mode to capture upstream traffic. Use when the project needs a multi-protocol mock server beyond HTTP-only tools like WireMock or MSW.

msw-handlers

Authors Mock Service Worker (MSW) request handlers for both browser and Node.js test environments using the `http.get` / `http.post` / `HttpResponse.json` API, wires them via `setupWorker` (browser) or `setupServer` (Node), and manages the test lifecycle (`server.listen` / `resetHandlers` / `close`). Use when the project uses JavaScript / TypeScript and needs to mock fetch / XHR at the network layer for both Vitest / Jest unit tests and Cypress / Playwright integration tests.

pairwise-test-case-generator

Generates parameterized test inputs combining boundary-value, equivalence-class, and pairwise-combinatorial cases from a typed multi-input specification - produces the cross-product of cases up to a configurable strength (1-wise / 2-wise / N-wise) using all-pairs reduction so the test surface stays tractable. Emits cases in the project's test-runner-native parametrize format. Use when a function or endpoint takes 3+ inputs whose interactions matter and full Cartesian product would explode.

seed-data-curator

Builds a reproducible E2E seed dataset for the project's test environments - picks a representative user / org / data-product cross-section, generates the rows via the project's chosen factory library (FactoryBot / mimesis / Bogus / Faker + factory_boy), persists the dataset as a checked-in fixture (SQL dump / JSON / per-engine seed file), and wires it into the test bootstrap. Use when starting E2E coverage on a project that has no seed strategy, or when an existing seed has drifted.

synthetic-data-tool-selector

Chooses between the four mainstream synthetic test-data generators - Faker (JavaScript), FactoryBot (Ruby), mimesis (Python), Bogus (.NET) - picks the right tool by language and use case (raw value generation vs. typed factory orchestration), shows side-by-side equivalents for the same fixture across all four, and emits the language-appropriate code. Use when starting test-data work on a project and the team wants the "which tool should I use" decision documented.

synthetic-pii-generator

Generates realistic-but-fake personally identifiable information (PII) - emails, phone numbers, SSNs / national IDs, addresses, names, credit-card numbers (test BIN ranges), date-of-birth - for non-production environments. Wraps Faker / mimesis with PII-aware constraints so generated values match real format expectations (Luhn-valid card numbers, region-valid phone formats, ITIN/SSN format) without ever generating real-person data. Use when seeding test environments, building demo data, or replacing real PII in copied datasets.

test-data-patterns

Pure reference catalog of the cross-language object-construction patterns for test data - Test Data Builder (Pryce/Freeman), Factory (with traits and associations), Object Mother, Fixture composition (per-test / per-describe / shared), Snapshot (defers to `golden-file-conventions` for the operational details), and Production-Data Anonymisation. Distinct from per-language data wrappers (`factory-bot-data` Ruby, `faker-data` JS, `mimesis-data` Python, `bogus-data` .NET) which document tool-specific configuration; this catalog is the architecture-tier reference for choosing **which pattern** before reaching for the tool. Use when choosing a test-data construction pattern for a new suite, or auditing an existing suite whose fixtures have drifted into shared mutable state.

wiremock-stubs

Authors WireMock stub mappings for HTTP service mocking - `stubFor` with verb/path/header matchers + `willReturn` response shaping, lifecycle via `WireMockServer` (start / stop) or JUnit `WireMockExtension`, request verification via `verify()`, and dynamic-port allocation for parallel tests. Use when the project is JVM-based and tests need to mock HTTP dependencies (third-party APIs, internal microservices) at the network layer.