test-case-review-rubric
Scores an already-written test case against six per-case quality axes (objective specificity, precondition executability, step granularity, step abstraction level, expected-result observability, traceability validity) and six set-level axes (partition coverage, boundary coverage, duplication, orphan and uncovered requirements, tier shape, identifier consistency). Derives a per-case PASS / WEAK / FAIL verdict and a set verdict from it without averaging, and marks every threshold as either standard-backed (ISTQB glossary, ISTQB CTFL v4.0, ISO/IEC/IEEE 29119-3:2021) or practitioner convention (step-count ceiling, tier bands, provenance threshold). Assumes the case field list and field cardinality are already defined by a test-case anatomy reference and judges content quality only. Use when reviewing a batch of hand-written test cases before promoting them to a release suite or handing them to an automation engineer.
Install with skills.sh (any agent)
npx skills add testland/qa --skill test-case-review-rubrictest-case-review-rubric
What this rubric judges, and what it does not
This rubric answers one question: given a test case whose fields are already populated, is the content good enough to hand to someone else?
It does not define what a test case is made of. The canonical field list (identifier, objective, preconditions, inputs, steps, expected results, postconditions, environment, traceability) and each field's cardinality (one-to-one vs one-to-many) belong to a test-case anatomy reference and are deliberately not restated here. That is the differentiation axis: anatomy answers "which fields", this rubric answers "is the content in them any good".
Two things follow from that split:
Anchor definition, used throughout: a test case is "a set of preconditions, inputs, actions (where applicable), expected results and postconditions, developed based on test conditions" (ISTQB glossary, test case, V4.7.2). Every axis below tests one clause of that sentence.
Gate 0: is the case scorable at all?
Run this before scoring. A case that fails Gate 0 gets no axis verdicts: it is reported as UNSCORABLE with the missing field named, and returned to its author.
| Gate 0 check | Result if absent |
|---|---|
| Identifier present and unique | UNSCORABLE |
| Objective (title) present | UNSCORABLE |
| Preconditions present | UNSCORABLE |
| At least one step present | UNSCORABLE |
| At least one expected result present | UNSCORABLE |
| Environment stated | Score normally, raise a warning |
| Priority, severity, type populated | Score normally, raise a warning |
| Traceability reference present | Score normally, scored by axis A6 |
Scoring an empty-field case wastes the reviewer's time and produces findings the author already knows about. Separating presence from quality also keeps a templated stub (a case created but never filled in) from being reported as six independent quality defects.
Per-case axes
| ID | Axis | PASS bar | FAIL trigger | Basis |
|---|---|---|---|---|
| A1 | Objective specificity | The title names a concrete observable behavior and a single verification. "Applies a 10 percent coupon to the subtotal before tax". | Title names only the feature or the activity: "Test checkout", "Checkout works", "Login". Or it joins two unrelated verifications with and. | ISO/IEC/IEEE 29119-3:2021 objective field; specificity bar is convention. |
| A2 | Precondition executability | Each precondition names a state a second tester can reach and confirm: named account, named fixture, named build, named data row. | Precondition is a mood rather than a state: "system is ready", "user has data". Or it is stated but not reachable ("prod DB in Tuesday's shape"). | ISTQB precondition: "the required state of a test item and its test environment prior to test execution" (glossary). |
| A3 | Step granularity | Each step is one interaction and carries its own expected result. | A step combines two interactions ("log in and add to cart"), or a step has no paired expected result, or the step's leading verb is Test / Verify / Check instead of naming an interaction. | ISTQB test step: "a single interaction between an actor and a test object consisting of an input, an action, and an expected result" (glossary). |
| A4 | Step abstraction match | Step phrasing sits at the layer the objective claims to verify. | A business-rule case is written in DOM mechanics (click #btn-checkout-submit), or a UI-mechanics case is written so abstractly the mechanic under test disappears ("the user navigates the form"). | Convention, informed by Cucumber, Writing better Gherkin: declarative phrasing "helps you focus on the value that the customer is getting, rather than the keystrokes they will use". |
| A5 | Expected-result observability | Every expected result names something a tester can observe and compare without judgment: a value, a state, a message, a status code. | Expected result asserts a quality rather than an observation: "works correctly", "performs well", "looks right". Or it needs a judgment call with no documented bar. | ISTQB expected result: "the observable predicted behavior of a test item under specified conditions based on its test basis" (glossary). Where a bar is needed, name the oracle: "a source to determine an expected result" (glossary). |
| A6 | Traceability validity | The reference resolves to a live requirement, acceptance criterion, or a named exploratory charter. | The reference is present but stale: it points at a requirement that no longer exists. A stale reference is worse than none, because it reports coverage that does not exist. | ISTQB traceability: "the ability to establish explicit relationships between related work products or items within work products" (glossary); ISTQB CTFL v4.0 section 1.4.4: "traceability of test cases to requirements can verify that the requirements are covered by test cases" (syllabus PDF). |
A6 note on the empty case: a missing reference is WEAK, not FAIL. A case with no link is still runnable and still finds defects; a case with a wrong link actively misreports coverage. An intentionally unlinked case (an exploratory charter, a smoke check that covers no single requirement) passes A6 when the absence is stated in the case rather than left blank.
Set-level axes
Run these once over the whole set, never per case. They diagnose the shape of the suite, not the craft of any one case.
| ID | Axis | PASS bar | FAIL trigger | Basis |
|---|---|---|---|---|
| S1 | Equivalence-partition coverage | For every parameter the set exercises, each identified partition (valid and invalid) is covered by at least one case. | Only valid partitions covered, or several cases pile into one partition while others are untouched. | ISTQB CTFL v4.0 section 4.2.1: "to achieve 100% coverage with this technique, test cases must exercise all identified partitions (including invalid partitions) by covering each partition at least once" (syllabus PDF). |
| S2 | Boundary coverage | For every ordered partition with a declared bound, the boundary values are exercised. State which BVA variant the set claims: 2-value covers each boundary and its closest neighbor in the adjacent partition; 3-value covers the boundary and both neighbors. | Bounded parameters have cases only at partition midpoints. | ISTQB CTFL v4.0 section 4.2.2, which defines both 2-value and 3-value BVA and their coverage items (syllabus PDF). |
| S3 | Duplication | No two cases assert the same postcondition under the same precondition with only cosmetic variation. | Near-identical cases that differ only in title wording or in an input value inside the same partition. Both cost maintenance; only one adds coverage. | Follows from S1: two cases in one partition are one coverage item. |
| S4 | Orphans and uncovered requirements | Every requirement in scope has at least one case; every case either links to a requirement or declares why it does not. | Requirements in scope with zero linked cases. Report both directions: cases without requirements, and requirements without cases. | ISTQB CTFL v4.0 section 1.4.4: "accurate traceability supports coverage evaluation" (syllabus PDF). |
| S5 | Tier shape | The set has a defensible mix of smoke, regression, negative, and edge cases, and the mix matches what the team says the suite is for. | All-smoke sets, or zero negative cases in a set that covers a validated input. | Convention, not a standard. See "Conventions" below. |
| S6 | Identifier consistency | One identifier scheme across the set. | CART-142-TC-01 mixed with cart-tc-2 mixed with Test Case 03. Inconsistent IDs break defect links and coverage rollups. | Convention. |
Scoring and verdict derivation
Per case, take the worst axis verdict:
| Condition | Case verdict | Meaning |
|---|---|---|
| Any axis FAILs | FAIL | Not runnable by a second person as written. Rewrite before execution. |
| No FAIL, one or more axes WEAK | WEAK | Runnable, but will cost the next reader time. Fix within the sprint. |
| All axes PASS | PASS | Hand it over. |
Per set, derive mechanically from the per-case verdicts plus S1 to S6:
| Condition | Set verdict |
|---|---|
Any case is FAIL or UNSCORABLE, or any set axis FAILs | BLOCK |
No FAIL anywhere, one or more WEAK | PASS WITH CAVEATS |
| Everything PASSes | PASS |
Two rules keep the aggregate honest:
Conventions, stated plainly
Three thresholds in common use have no standard behind them. They are useful tripwires that start a conversation. They are not pass or fail lines, and a team that has calibrated its own numbers should use those instead.
| Convention | Common value | What it is really detecting |
|---|---|---|
| Step-count ceiling | Roughly 15 steps | Not length. A long case is usually a case with more than one objective. Check A1 before invoking the ceiling: if the title still names a single behavior, a 20-step case may be correct. Flag as WEAK, never FAIL. |
| Tier distribution bands | Smoke 10 to 20 percent, regression 50 to 70 percent, negative 15 to 25 percent, edge 5 to 15 percent | Whether anyone designed the suite. The failure the bands catch is the all-smoke or zero-negative set, not a set that sits five points outside a band. |
| Unresolved-provenance threshold | More than 30 percent of cases with blank or placeholder references | An upstream authoring problem, not a case-review problem. Above the line, stop reviewing cases and go fix how they are being written. |
Nothing in the ISTQB glossary, the ISTQB CTFL v4.0 syllabus, or ISO/IEC/IEEE 29119-3:2021 sets any of these numbers. Presenting them as standards is the most common way a review loses the author's trust.
Worked example: a weak case, scored
The case as written:
ID: TC-07
Title: Test checkout
Preconditions: System is ready
Steps:
1. Log in and go to the cart
2. Enter a coupon
3. Click #btn-checkout-submit
4. Verify the order
Expected: Checkout works correctly
Refs: (blank)Gate 0: passes. Every required field has content, so the case is scorable.
| Axis | Verdict | Evidence |
|---|---|---|
| A1 Objective specificity | FAIL | "Test checkout" names a feature, not a behavior. A reader cannot tell which of the dozen things checkout does is under test, so they cannot tell whether the case passed for the right reason. |
| A2 Precondition executability | FAIL | "System is ready" is not a state. Ready with which account, which cart contents, which build, which coupon configured? Two testers will set up two different systems and get two different results. |
| A3 Step granularity | FAIL | Step 1 combines two interactions ("log in" and "go to the cart"). No step carries a paired expected result, so a failure at step 2 is indistinguishable from a failure at step 4. Step 4 leads with Verify, which names no interaction. |
| A4 Step abstraction match | WEAK | The objective is a business flow, but step 3 is a DOM selector. The case will break on a markup change that does not affect checkout behavior. |
| A5 Expected-result observability | FAIL | "Checkout works correctly" is not observable. There is no value, state, or message to compare against, so two testers can disagree about whether the case passed. |
| A6 Traceability validity | WEAK | Reference is blank and no exploratory justification is given. WEAK rather than FAIL: the case is still runnable, it just does not report coverage. |
Case verdict: FAIL. Four axes fail. This is not a case that needs polishing; it is a case that needs writing.
Worked example: the rewrite
ID: CHECKOUT-TC-07
Title: Applies a 10 percent coupon to the order subtotal before tax
Preconditions:
- Account checkout-qa@example.com exists, is verified, and has an empty cart
- Coupon SAVE10 is active: 10 percent off subtotal, no minimum spend,
expires 2027-01-01
- Build web 4.12.0 on staging, Chrome 138, locale en-US, tax rate 8 percent
Inputs: SKU DEMO-001 at 100.00 USD, quantity 1, coupon SAVE10,
test card 4242 4242 4242 4242
Steps:
1. Sign in as checkout-qa@example.com
-> Account menu shows checkout-qa@example.com
2. Add SKU DEMO-001, quantity 1, to the cart
-> Cart badge shows 1; cart subtotal shows 100.00 USD
3. Open the cart and apply coupon SAVE10
-> Discount line shows 10.00 USD off; subtotal shows 90.00 USD
4. Complete payment with card 4242 4242 4242 4242
-> Confirmation shows order total 97.20 USD
(90.00 discounted subtotal plus 7.20 tax at 8 percent)
Postconditions: Order exists in status paid; the account's cart is empty
Refs: REQ-CHECKOUT-114 (percentage coupons apply before tax)Re-scored:
| Axis | Verdict | What changed |
|---|---|---|
| A1 | PASS | The title names one observable behavior (coupon applies before tax) and one verification. The ordering claim is the whole point of the case and is now visible in the title. |
| A2 | PASS | Account, coupon configuration, build, browser, locale, and tax rate are all named. A second tester reaches the same starting state. |
| A3 | PASS | Four steps, one interaction each, each with a paired expected result. A failure now localizes to a step. |
| A4 | PASS | Steps are phrased at the business layer the objective claims, with concrete inputs. No DOM selectors, because no DOM mechanic is under test. |
| A5 | PASS | Every expected result is a number or a string on screen. The arithmetic is shown, so the case doubles as its own oracle: a reviewer can check 90.00 plus 7.20 without opening the requirement. |
| A6 | PASS | Resolves to REQ-CHECKOUT-114, and the parenthetical says which clause of it. |
Case verdict: PASS.
One thing the rewrite cannot fix on its own: S1 and S2 are set-level. This case covers exactly one partition (a valid percentage coupon, no minimum spend). The review should therefore emit a follow-up list rather than declare coverage done: expired coupon, coupon below a minimum spend, coupon at exactly the minimum spend and one cent under it, and a coupon that would drive the subtotal below zero.
Judgment calls reviewers commonly split on
These five are where two competent reviewers reach opposite verdicts on the same case. Each has a ruling so a review stays consistent across reviewers.
1. Imperative steps: always a defect? No. Declarative phrasing ages better because it survives UI change, which is why Cucumber's guidance prefers it, but a step is defined as a single interaction between an actor and a test object (ISTQB), and some interactions genuinely are keystrokes. Judge abstraction against the case's own objective (axis A4): a keyboard-traversal case should say Tab and Shift+Tab; a pricing case should not say #btn-checkout-submit. Flag the mismatch, not the style.
2. Is Verify a banned word? Only in steps. A step that begins "Verify the order" names no interaction and no input, so it fails A3. A title that begins "Verifies the order total excludes shipping" names a concrete observable behavior and passes A1. Banning the verb everywhere produces title churn with no gain.
3. A missing requirement link: FAIL or WEAK? WEAK. A stale link is FAIL. The asymmetry is deliberate: an unlinked case under-reports coverage, which is visible in the S4 rollup; a case pointing at a deleted requirement over-reports it, which is invisible and is exactly what coverage reporting is supposed to prevent.
4. Does one failing case fail the set? The set verdict goes to BLOCK, and the report names the three failing identifiers out of two hundred. Both halves matter. Dropping the block lets broken cases ship; dropping the per-case detail turns the review into an unactionable grade.
5. Is a 22-step case too long? Not automatically. Step count is a proxy for "more than one objective", and A1 measures that directly. If the title still names one behavior and every step earns its place, note the length and move on.
Anti-patterns in applying this rubric
| Anti-pattern | Why it fails | Instead |
|---|---|---|
| Scoring a stub with empty fields across all six axes | Produces six findings for one problem and buries the real signal. | Gate 0 first: report UNSCORABLE with the missing field named. |
| Reporting a single percentage for the set | A number lets everyone feel measured while nothing gets fixed, and it hides which cases are broken. | Per-case verdicts, then a routing verdict derived from them. |
| Quoting the step ceiling or tier bands as standards | They are conventions with no standard behind them; a reviewer who overstates them loses the author's trust on the axes that are standard-backed. | Say "our convention" and give the reason (one objective per case; a designed mix). |
| Demanding S1 and S2 verdicts on flow-only cases | Not every case has parameters. Fabricating partition findings for a navigation case is noise. | Emit n/a, no parameterized inputs for S1 and S2 on those cases. |
| Failing every case that permits two outcomes | A first-run probe of undocumented behavior may legitimately say "either the client blocks at 32 characters or the server returns 422". | Score it WEAK with a note to collapse to one expected result after the first run, then re-review. |
| Rewriting cases silently during review | The author loses the context of what was wrong, and the same defect returns in the next batch. | Score, cite the axis, propose the rewrite; leave the commit to the author. |
| Running the rubric once | Case sets drift as fast as code. A one-off audit is a snapshot. | Re-run on every batch promoted to a release suite. |