input-domain-coverage-audit

The axis this owns

Two different questions can be asked about the same test file:

A file can pass one and fail the other in either direction. Four tests that each assert a precise deep-equal result are specific, and still shallow if all four pass the same kind of argument. One test that passes a valid value and an invalid value spans two partitions, and is still weak if it only asserts toBeTruthy(). Run both checks; neither substitutes for the other.

This audit is static: it reads test source and the constraints declared on the code under test. It does not execute the suite, and it does not read line or branch coverage reports.

The three axes

Each axis maps to a canonical black-box technique.

An equivalence partition is "a subset of a value domain for which a component or system is expected to treat all values the same based on the specification", and a boundary value is "a minimum or maximum value of an ordered equivalence partition". Both definitions are properties of the specification, not of the test file, which is why this audit reads the declared contract alongside the tests.

Shallow input coverage is not specific to generated tests. Hand-written suites drift the same way when new cases are added by copying the nearest existing test and changing one literal.

Step 1 - Inventory the entry points and their input values

For each entry point the file exercises (the function, method, route, or command named in the Act phase of the tests), build one row:

• the entry point signature, including each parameter;
• every literal argument and fixture value passed to it anywhere in the suite, grouped by parameter position;
• the declared contract for each parameter: schema bounds, validation decorators, type-level constraints, documented ranges;
• the declared error contract: exceptions thrown, promises rejected, error values returned, non-success status codes documented.

Everything after this step operates on that row. If the contract column is empty for a parameter, that is a finding about the code, not yet a finding about the test: axes are recorded n/a rather than SHALLOW when the specification declares nothing to partition against.

Step 2 - §EP, equivalence partitioning

The audit does not have the specification's partition list, so it infers partitions from the test data actually used. Cluster the collected values per parameter and flag when every value falls into one cluster.

Literal-clustering heuristic, applied per parameter:

The heuristic answers "did the author vary this input at all", which is the question that catches the copy-the-neighboring-test failure. It does not answer "are these two values in genuinely different partitions": two values can differ in length and character class and still be members of the same invalid partition (two differently malformed email addresses are still one partition). Treat a multi-cluster result as absence of evidence for shallowness, not proof of good partitioning.

The PASS bar. The ISTQB Certified Tester Foundation Level syllabus v4.0.1 §4.2.1 (page 39, PDF) states the criterion: "In EP, the coverage items are the equivalence partitions. To achieve 100% coverage with this test technique, test cases must exercise all identified partitions (including invalid partitions) by covering each partition at least once." The same section names Each Choice coverage, which "requires test cases to exercise each partition from each set of partitions at least once" for entry points with several parameters, and notes it "does not take into account combinations of partitions".

This audit's PASS bar is one valid partition plus one invalid partition per parameter. That is a floor beneath the syllabus criterion, not a restatement of it: clearing the floor does not establish 100% EP coverage, it only establishes that more than one partition was exercised. Where the specification does enumerate partitions, measure against the syllabus formula instead: partitions exercised by at least one test case, divided by total partitions identified, expressed as a percentage (coverage is defined as "the degree to which specified coverage items are exercised by a test suite, expressed as a percentage").

§EP verdict:

• PASS when at least one valid and one invalid partition are exercised for every parameter.
• SHALLOW when every collected value for any parameter clusters into a single partition.
• n/a only when the entry point takes no parameters.

Step 3 - §BVA, boundary value analysis

Boundaries exist only where an order exists. The CTFL syllabus v4.0.1 §4.2.2 (page 40) is explicit: BVA "is a test technique based on exercising the boundaries of equivalence partitions. Therefore, BVA can only be used for ordered partitions." This is the grounding for the n/a rule below, not a convenience exemption.

For each parameter with a machine-readable bound (a schema minimum / maximum / minLength / maxLength, a validation decorator, an interface contract with a documented range, a declared collection-size limit), check that at least one test exercises a value at min, min-1, max, or max+1.

That check is 2-value BVA. Per §4.2.2: "In 2-value BVA, for each boundary value there are two coverage items: this boundary value and its closest neighbor belonging to the adjacent partition. To achieve 100% coverage with 2-value BVA, test cases must exercise all coverage items." Teams holding a stricter bar use 3-value BVA, where "for each boundary value there are three coverage items: this boundary value and both its neighbors", which adds min+1 and max-1 to the required set.

§BVA verdict:

• PASS when every declared bound has at least one test on the boundary or its neighbor outside it.
• SHALLOW when a bound is declared and no test sits at or beside it.
• n/a when no ordered constraint is declared for any parameter. Record §BVA: n/a and move on. Demanding a boundary case for an unbounded or unordered parameter is a false finding, not a strict one.

Step 4 - §NEG, error and negative paths

Classify every assertion in the suite for this entry point as positive or negative by what it targets:

Classify by target, not by matcher name. An equality matcher asserting that a response body carries a validation error, or that a status equals 422, is a negative assertion. A matcher named for exceptions that is asserted never to fire is not.

Then compute the negative-assertion ratio for the entry point:

negative_assertion_ratio = negative_assertions / total_assertions

Flag §NEG when the ratio is exactly zero for an entry point that has any declared error contract: it throws, it rejects, it returns an error value, or it documents a non-success response.

The zero threshold is a practitioner convention, not a standard. No published standard fixes a minimum negative-assertion ratio. Zero is used here because it is the one value that is unambiguous: an entry point with a declared error contract and no assertion anywhere on that contract has demonstrably left a documented behaviour untested. Teams that set a floor above zero are choosing a local convention; say so when reporting it, and do not present any such number as a canonical requirement.

§NEG verdict:

• PASS when the ratio is above zero.
• SHALLOW when the ratio is zero and an error contract is declared.
• n/a when no error contract is declared. A total function such as add(a: int, b: int): int has no rejection path to assert on.

Step 5 - Roll up the per-axis verdicts

Score all three axes before deciding anything. Stopping at the first SHALLOW axis produces reports that demand boundary tests for unbounded parameters and error tests for total functions.

The entry point's verdict is its weakest applicable axis: SHALLOW if any applicable axis is SHALLOW, PASS if every applicable axis is PASS, and N/A if all three axes are n/a.

Every SHALLOW cell carries its evidence: which values were collected, which partition they clustered into, which declared bound has no test beside it, which error contract has no assertion. A verdict without the evidence that produced it cannot be argued with, and gets ignored.

Output format

One section per entry point, each with the three-axis table and a verdict line.

## Input-domain coverage audit

**Entry points reviewed:** 2
**SHALLOW verdicts:** 1

### `src/cart/addItem.ts` -> `addItem(productId, qty)`

| Axis | Result | Evidence |
|---|---|---|
| §EP equivalence classes | SHALLOW | All 4 tests pass `productId` as a 24-char hex string and `qty` as a small positive integer (1-3). No invalid `productId`, no `qty=0`, no negative `qty`, no `null`. |
| §BVA boundaries | SHALLOW | Schema declares `qty: { min: 1, max: 99 }`. No test at `qty=1`, `qty=0`, `qty=99`, or `qty=100`. |
| §NEG error paths | SHALLOW | 11 of 11 assertions are positive (`.toEqual`, `.toBe`); ratio 0. `addItem` declares `throws InvalidQtyError`; no test asserts the throw. |

**Verdict: SHALLOW.** Add at least: (a) one invalid-`productId` case (§EP),
(b) boundary cases at `qty=0` and `qty=100` (§BVA), (c) one assertion on
`InvalidQtyError` (§NEG).

### `src/cart/getCart.ts` -> `getCart(userId)`

| Axis | Result | Evidence |
|---|---|---|
| §EP equivalence classes | PASS | Suite exercises an authenticated user, an anonymous user, and a tenant-mismatched user: three partitions, one of them invalid. |
| §BVA boundaries | n/a | No ordered constraint declared on `userId`. |
| §NEG error paths | PASS | 3 of 8 assertions target `UnauthorizedError` on the rejection path; ratio 0.375. |

**Verdict: PASS.**

Remediation is a separate job. This audit names the missing case class and the axis that demands it; producing the case values is the work of a boundary-value or negative-case data generator.

Worked example

createUser(email, age). The declared contract is email: string matching a format, age: integer, minimum 18, maximum 120, and the function throws ValidationError on a rejected input.

The suite has three tests, passing ("ada@example.com", 30), ("grace@example.com", 42), and ("alan@example.com", 35), each asserting expect(result.id).toEqual(expect.any(String)).

Walking the axes:

• §EP. email values are all the same length band and character class, all well-formed: one partition, no invalid partition. age values are 30, 42, 35: same sign, same order of magnitude, all inside the valid range: one partition. SHALLOW on both parameters.
• §BVA. age declares minimum 18 and maximum 120, so the partition is ordered and the axis applies. Required 2-value coverage items are 17, 18, 120, 121. None is exercised. SHALLOW.
• §NEG. Three assertions, all targeting a returned value: ratio 0. ValidationError is declared and never asserted. SHALLOW.

Verdict: SHALLOW. The minimum set that clears the floor is one malformed email (§EP invalid partition), age=17 and age=121 (§BVA, adding age=18 and age=120 if the team holds a 3-value bar), and one assertion that a rejected input raises ValidationError (§NEG).

Note how three tests bought nothing: they are one test repeated with the literals changed. This is the pattern the literal-clustering heuristic in Step 2 exists to name.

Anti-patterns

Limitations

• Heuristic, not formal partition analysis. §EP clustering compares literal values. Two values in the same partition can be scored as two partitions, and a multi-cluster result is weaker evidence than an explicit partition list from the specification.
• §BVA depends on machine-readable constraints. Schema bounds, decorators, and typed ranges are detectable; a bound stated only in a free-text comment is not, and the axis will read n/a where a real bound exists.
• Static only. No execution, so no mutation score and no branch coverage. Where a mutation-testing tool is available, its score is the stronger shallowness signal and this audit is the cheap pre-check that explains which input classes are missing.
• Unit and component scope. For suites where coverage is argued at the system level rather than per entry point, the per-parameter partition walk does not apply cleanly; audit those against their own coverage model.
• Framework idioms vary. Assertion classification in Step 4 is by target, which holds across frameworks, but locating assertions and fixture values in an unfamiliar test framework may need a manual pass.