confirmation-testing-workflow

The evidence procedure that justifies moving a defect out of Fixed: prove the build under test contains the fix, re-run the reproduction that failed before the fix, and classify the result.

What this owns, and what it does not

The axis is verification procedure vs lifecycle vocabulary. A defect lifecycle catalog owns the state names and which transitions between them are legal (Fixed can go to Verified or Reopened, and Fixed to Closed without Verified is illegitimate). This procedure owns the evidence that justifies picking one of those transitions. It answers "may I claim Verified?", not "what states exist?".

Also out of scope, deliberately:

• Triage of the original defect. Whether the report is valid, a duplicate, or reproducible in the first place is decided before a fix is attempted.
• Severity and priority. Those are separate axes from state, and nothing here changes them.
• The broader regression suite. See the next section: confirmation testing and regression testing are different activities.

Confirmation testing is not regression testing

Both are change-related testing, and conflating them is a common error.

The ISTQB glossary entry for confirmation testing lists retesting as its synonym (same entry, linked above; there is no separate glossary page under that name). The Foundation Level syllabus v4.0 §2.2.3 pairs the two activities in one section, which is where the confusion starts (syllabus distributed from istqb.org/certifications/certified-tester-foundation-level).

Practical consequence: a green full regression suite is not confirmation that the defect is fixed, and a passing confirmation test is not evidence that nothing else broke. Run and report them separately.

Input this procedure assumes

A fix commit SHA that is already merged into the target branch, plus the name of the environment under verification. A pull request number, a release note, or a Jira comment saying "fixed" is not a SHA. If no merged commit exists yet, there is nothing to confirm: confirmation testing is performed "after fixing a defect" by the definition above.

Step 1 - Prove the build under test contains the fix

This is the step teams skip, and skipping it produces both false NOT FIXED verdicts (the fix was never deployed) and false VERIFIED verdicts (a different change masked the symptom).

Do not trust a version label. A tag, a release name, a package.json version, a "deployed 14:02" timestamp, or a green deploy pipeline are all claims about a build, not properties of it. Version strings get bumped in a separate commit from the fix, deploys get rolled back, and hotfix branches get cut from a base that predates the fix.

Trust an ancestry check instead. git merge-base --is-ancestor <A> <B> checks whether the first commit is an ancestor of the second and exits 0 if true, 1 if not; errors are signalled by "a non-zero status that is not 1" (per git-scm.com/docs/git-merge-base).

git merge-base --is-ancestor "$FIX_SHA" "$BUILD_SHA"
case $? in
  0) echo "CONTAINS FIX" ;;
  1) echo "MISSING FIX" ;;
  *) echo "CHECK ERRORED - treat as unknown, not as MISSING" ;;
esac

Read the three outcomes as three different things:

Two arguments are needed, so the environment must expose the commit SHA it was built from, not just a version number. If it exposes only a version label, the containment question cannot be answered and the verdict is BLOCKED until build metadata is added. That is a real finding worth reporting, not a reason to guess.

Two common traps this catches:

• A shallow or partial clone can make an ancestry check error out or answer from truncated history. Verify against a repository with the full history reachable from both SHAs.
• The branch moved. Check ancestry against the exact SHA the environment is running, not against the branch tip, which may have advanced past it.

When the check exits 1, stop and name the gap concretely, for example: staging is on 3f1d0aa, and fix 9c4e7b2 is not an ancestor of it. Re-running the reproduction there would only re-confirm the original failure and would produce a NOT FIXED verdict that blames the developer for a deployment problem.

Step 2 - Choose the reproduction, in priority order

Re-run the narrowest artifact that failed before the fix. Prefer the highest available source:

1. The committed automated reproduction test. A test written against this defect that was red before the fix. Strongest evidence: its pre-fix red state is recorded in history, so its post-fix green state is a real before/after comparison rather than an assertion that has never failed.
2. Written reproduction steps from the defect report. The commit plus command plus observed-versus-expected block. Weaker, because a human executes it and the pre-fix failure is attested rather than recorded. Convert them into a numbered re-verification script (precondition, steps 1..N, expected result per step) and record the human's outcome verbatim.
3. A recorded artifact: a replayable session, a HAR capture, a video, a crash dump with the triggering input. Weakest of the three, because replay fidelity varies, but still an artifact rather than a memory.

If none of the three exists, the verdict is BLOCKED. Verification without a reproduction is an opinion. The route out is to build a reproduction first, which is separate work with its own output.

Two disqualifiers apply regardless of tier:

• A quarantined or known-flaky test cannot confirm anything. Its pass rate is uncorrelated with the fix, so a green run is not evidence. Stabilise it first, then re-run.
• A reproduction that never failed before the fix proves nothing. If nobody can show the artifact red on a pre-fix commit, it is not a reproduction of this defect. Confirm the red state on the fix commit's parent before trusting the green state on the fix commit.

Run exactly that artifact, nothing wider. A broader suite dilutes the signal and drags unrelated failures into the verdict.

Step 3 - Verdict

The governing rule: uncertainty resolves to BLOCKED, never to a guess. BLOCKED is not a failure of the procedure, it is the correct output whenever the evidence does not support a clean classification. Specifically:

• A partially passing run is BLOCKED, not VERIFIED.
• A run whose output cannot be tied to the defect's original symptom is BLOCKED, not VERIFIED.
• A pass on a build whose containment could not be proven is BLOCKED, not VERIFIED.
• A failure on a build proven not to contain the fix is BLOCKED, not NOT FIXED. Blaming the fix for a deployment gap costs a real reopen cycle.

VERIFIED is the only verdict that licenses a Fixed to Verified transition. NOT FIXED licenses a reopen with the failing output attached. BLOCKED licenses neither: record the blocker and leave the state where it is, because a BLOCKED defect that is silently closed is exactly the failure mode the verification gate exists to prevent.

Output shape

## Confirmation test result - <defect-id>

**Verdict:** VERIFIED | NOT FIXED | BLOCKED
**Fix commit:** `<sha>` on `<branch>`
**Environment:** <env>, build `<sha>`
**Containment check:** `git merge-base --is-ancestor <fix-sha> <build-sha>` exit <0|1|other>
**Reproduction:** <tier 1 test path | tier 2 manual script | tier 3 recording>
**Pre-fix state of that reproduction:** <red at <parent-sha> | attested by reporter>

### Re-run output

<verbatim runner output, or the recorded manual execution>

### Conclusion

<one sentence tying the output to the defect's original symptom>

Worked example

Defect: duplicate invoice rows are created when the same invoice is submitted concurrently. Fix commit 9c4e7b2. Environment: staging, reporting build SHA 9c4e7b2 from its build-metadata endpoint.

1. Containment. git merge-base --is-ancestor 9c4e7b2 9c4e7b2 exits 0 (a commit is its own ancestor). Staging contains the fix. Note that the check ran against the build SHA staging reported, not against the release tag v2.14.0, which was cut in a later commit.

2. Reproduction. A tier 1 artifact exists: tests/invoices/concurrent-submit.spec.ts, committed red before the fix and confirmed red on 9c4e7b2^.

3. Re-run. Exactly that spec, nothing wider:

   Running 1 test using 1 worker
     ok  tests/invoices/concurrent-submit.spec.ts:18:5 > creates exactly one
         invoice row for 5 concurrent submits (2.9s)
   1 passed (4.1s)
   

4. Verdict: VERIFIED. The artifact that produced the duplicate rows before the fix produces exactly one row now, in a build proven to contain the fix.

Two variants of the same run:

• Output reads 2 rows found, expected 1 -> NOT FIXED. Same report shape, failing output attached, defect reopened.
• Output reads 1 passed but the containment check exited 1 because staging is on 3f1d0aa -> BLOCKED, not VERIFIED. The pass came from a build without the fix, so it says nothing about the fix. Report the deployment gap.

Anti-patterns

Limitations

• Ancestry proves the source, not the artifact. A commit being an ancestor of the build SHA does not prove the deployed binary was built from that source, that the migration ran, or that a feature flag gating the fix is on. For environments with an independent build pipeline, treat containment as necessary but not sufficient, and note any flag state in the report.
• Merged is not deployed. An ancestry check against the target branch shows the fix is in the branch. It says nothing about any environment until it is re-run against the SHA that environment actually reports.
• Rebases and squashes change the SHA. After a squash merge the original commit SHA is not an ancestor of anything. Use the SHA that landed on the target branch, and expect the pre-merge SHA to answer 1.
• Confirmation of one defect only. A VERIFIED verdict covers this defect's reproduction and nothing else. Pair it with a separate regression run before concluding the change is safe to release.