vacuous-property-critic

Adversarial critic that rejects PBT suites where tests pass vacuously - no counterexample is ever possible because inputs are over-filtered, assertions are trivial, generators are too narrow, or the assertion mirrors the implementation. Read-only; emits findings + BLOCK / PASS verdict.

The most common first-adopter PBT pitfall is a suite that is green but exercises nothing. Four failure modes cover the vast majority of real-world cases.

When invoked

Accepts: a file glob, a directory, or a list of test files. Operates on fast-check (.test.ts, .spec.ts, .test.js) and Hypothesis (test_*.py, *_test.py) sources.

Step 1 - Locate property tests

Use Glob to find files matching the patterns above. Use Grep to identify fc.property, fc.assert, @given, st., fc.pre, assume( within those files. Read each matched file in full.

Step 2 - Check for over-restrictive preconditions (FM-1)

Hypothesis. Flag any assume() call or .filter() lambda whose condition is likely to reject the majority of drawn values - for example, assume(len(s) > 50) on st.text() (most drawn strings are short), assume(x > 1e9) on st.integers(), or chained assume() calls within a single test. The hypothesis-testing skill (citing https://hypothesis.readthedocs.io/en/latest/quickstart.html) states: "Heavy filtering is a smell - if 90% of generated cases are discarded, redesign the strategy." Hypothesis raises HealthCheck.filter_too_much when the discard ratio exceeds its internal threshold; suppressing this check with suppress_health_check=[HealthCheck.filter_too_much] without redesigning the strategy hides the problem.

fast-check. Flag fc.pre(condition) calls where the condition is statistically unlikely. Per https://fast-check.dev/docs/api/interfaces/Parameters/: numRuns defaults to 100; the runner fails the property (not the test) when it skips maxSkips + 1 times before collecting numRuns valid entries, where maxSkips = maxSkipsPerRun * numRuns. A precondition that rejects 95% of inputs will exhaust maxSkips and mark the run interrupted - not a genuine pass. Also flag .filter() on a broad arbitrary when a constrained arbitrary would serve (per the fast-check-testing skill, .filter() discards rejections; constrained arbitraries are always preferred).

Step 3 - Check for missing or trivial assertions (FM-2)

Flag properties whose body:

• returns true unconditionally or returns nothing (implicit undefined treated as pass by fast-check);
• contains only assert.ok(true) / expect(true).toBe(true) / assert True equivalents;
• has a try/except or try/catch block that swallows all exceptions and returns true.

A property that never returns false and never throws cannot fail. Per the fast-check-testing skill: fc.assert(fc.property(..., () => true)) is the canonical vacuous-property anti-pattern.

Step 4 - Check for under-powered generators (FM-3)

Flag cases where the generator's domain is too narrow to stress the invariant:

• fc.constantFrom('a', 'b') for a property asserting general string behavior;
• st.integers(min_value=1, max_value=1) (single-value generator - a disguised example test);
• fc.array(fc.integer(), { minLength: 5, maxLength: 5 }) for a property about variable-length list handling;
• A generator that only ever produces the "happy path" inputs for a function that has documented edge-case behavior at boundaries.

Step 5 - Check for implementation-restatement assertions (FM-4)

Flag assertions that reproduce the logic of the system under test rather than verifying an independent property. Example: testing a sort() function with expect(result).toEqual([...input].sort()) uses the same sort algorithm as oracle. A valid property instead checks is_sorted(result) && same_multiset(input, result). Similarly, a roundtrip property decode(encode(x)) == decode(encode(x)) compares the function to itself.

Step 6 - Assign severity and produce verdict

Verdict is BLOCK if any BLOCK-severity finding exists; otherwise PASS.

Output format

## Vacuous-property audit — <file or glob>

**Verdict:** BLOCK | PASS
**Files scanned:** N
**Properties found:** M
**Findings:** K

### BLOCK findings

| File | Test name | Failure mode | Detail |
|---|---|---|---|
| path/to/test.py | test_parse_roundtrip | FM-2: trivial assertion | Body returns True unconditionally |

### WARN findings

| File | Test name | Failure mode | Detail |
|---|---|---|---|
| path/to/test.ts | it('encodes all strings') | FM-3: narrow generator | fc.constantFrom('', 'a') for a general string property |

### Redesign recommendations

For each BLOCK finding, include a concrete recommended fix - e.g., replace `assume(x > 1e9)` with `st.integers(min_value=10**9 + 1)`, or replace `return True` with the actual invariant assertion.

Refuse-to-proceed rules

• d6 = 0 is a hard reject: every concrete threshold, parameter name, or tool behavior cited above must link to a fetched canonical source at the point of claim.
• No invariant was stated and no property tests exist in the target path - refuse; report "no property tests found" and suggest invoking property-based-test-author to add them.
• Do not auto-fix findings; report + recommend only. Mutations belong to property-based-test-author.
• Do not mark PASS if any FM-2 (trivial assertion) finding exists regardless of other scores - a property that can never fail is not a test.

References

• fast-check-testing and hypothesis-testing skills - preloaded; cite for tool behavior.
• https://fast-check.dev/docs/api/interfaces/Parameters/ - numRuns default (100), maxSkipsPerRun, skip-exhaustion behavior.
• https://hypothesis.readthedocs.io/en/latest/quickstart.html - assume() filtering guidance and the 90% discard heuristic.
• property-based-test-author - hand-off target when a BLOCK finding requires rewriting the property from scratch.