Tool-selection decision record format

Deep reference for the framework-choice-advisor SKILL.md. The output contract for writing down a chosen developer tool as a portable decision record: the observed project signal, exactly one primary recommendation, rationale that names the rejected alternative, what to read next, and a mandatory list of the conditions that would flip the choice.

The differentiation axis is output contract versus analysis: the SKILL.md spine (and the vendor evaluation in vendor-evaluation.md (opens in new window)) compare candidate tools on their merits; this format governs only the shape of what gets committed once the weighing is finished. Use both together: the analysis chooses, this record preserves the choice.

Overview

A tool gets chosen once and questioned for years. The choice survives in a chat thread, the reasoning does not, and the next person either re-litigates it from scratch or inherits it as folklore.

This format is a narrowed Architecture Decision Record. It keeps the ADR skeleton and tightens three things generic ADR templates leave open - how many tools may be recommended (Rule 1), what counts as admissible evidence for the context (Rule 2), and whether the reversal conditions are optional (Rule 3). It is tool-agnostic: the same record shape works for a test framework, a package manager, a linter, a migration tool, a CI runner, or a logging library.

What this record is and is not

ADR provenance and field mapping

An ADR is "a document that captures an important architecture decision made along with its context and consequences" (joelparkerhenderson/architecture-decision-record (opens in new window)), and the canonical five-part shape is Title, Context, Decision, Status, Consequences (Nygard, Documenting Architecture Decisions (opens in new window)). A tool choice is a decision of exactly that kind: it "addresses a functional or non-functional requirement that is architecturally significant" (adr.github.io (opens in new window)).

The split of Consequences into two fields is deliberate. Nygard requires that "all consequences should be listed here, not just the 'positive' ones" (source (opens in new window)), and in practice tool records collapse to a list of benefits unless the negative half has its own heading with its own required content.

The record fields

Eight fields. Seven are always required; the eighth appears only under the co-equal rule below.

Rule 1: exactly one primary recommendation

A record names one tool in the Decision field. Two tools in that field is not a decision, it is the reader's decision deferred.

This follows the ADR specificity convention: a well-written record "should be about one AD, not multiple ADs" (joelparkerhenderson/architecture-decision-record (opens in new window)).

The one exception is a genuine co-equal: two candidates that the available signal cannot separate, where the tiebreak depends on a constraint nobody has stated yet (team size, an unwritten hiring plan, a language the team may or may not adopt). Then, and only then:

1. Put the stronger-by-default candidate in Decision.
2. Put the other in Secondary fallback.
3. State the constraint that flips between them, in one line.

A Secondary fallback entry without that constraint line is a tiebreak handed back to the reader. Delete it or complete it.

Never list a third option. If three candidates look co-equal, the signal was too weak to write a record at all, and Rule 2 applies instead.

Rule 2: every record rests on an observed signal

A recommendation is never inferred from a README, a project description, a folder name, or what the team says the stack is. The Signal field must quote one of:

Inadmissible on its own: a README paragraph, a wiki page, a folder named after a framework with nothing in it, a compiled artefact, or a verbal claim about the stack. Each of these describes intent rather than state, and intent and state diverge constantly.

This is the ADR Context field taken literally. Context "describes the forces at play" in neutral language (Nygard (opens in new window)); a force that cannot be quoted from the repository is not yet a force, it is a plan. Plans belong in Flip conditions, not in Signal.

When no admissible signal exists, do not write a record. Say so, name the file that would resolve it, and stop. A record built on a guess is worse than no record: it launders the guess into a citable decision.

Rule 3: the flip conditions field is mandatory

Every record declares the observations that would reopen it. This is what keeps a decision log from silently rotting into a set of unchallengeable defaults.

Good flip conditions are observable and specific:

• The project adds a second target platform.
• The service drops the ORM the chosen tool is built around.
• The suite passes 20 minutes of wall-clock time in CI.
• A non-JVM team takes ownership of this module.

Bad flip conditions are unfalsifiable: "if requirements change", "if the tool stops meeting our needs", "on annual review".

If the author genuinely cannot foresee a reversal condition, the field says so explicitly and gives the reason. An empty field reads as an oversight; a stated "no foreseeable condition, because the choice is forced by the runtime" reads as a claim a later reader can attack.

Flip conditions are also the confirmation hook. MADR carries an optional Confirmation section describing how compliance with the decision gets verified (MADR template (opens in new window)); in this format the flip conditions are what a periodic review actually checks.

Rule 4: supersede, never edit

When a flip condition fires, the old record is not rewritten. Nygard: "If a decision is reversed, we will keep the old one around, but mark it as superseded" (source (opens in new window)). The same immutability guidance appears in the widely used ADR collection: when an earlier decision becomes invalid, "a new ADR should be created" rather than modifying the original (source (opens in new window)).

So: write a new record, set the old one's Status to superseded with a pointer, and let the pair show the reader both the original reasoning and what changed. The set of records accumulated this way is the project's decision log (adr.github.io (opens in new window)).

Template

# <NNNN> - <tool category> for <component>

**Status:** <proposed | accepted YYYY-MM-DD | superseded by NNNN>

**Signal:**
<file path>:<line>
> <quoted excerpt that drove the detection>

**Decision:** We will use <exactly one tool>.

**Rationale:**
- Fits because: <one line tying the tool to the quoted signal>
- Not <strongest alternative>, because: <one line>

**Secondary fallback:** <tool | none>
<if present: the one constraint that would select it instead>

**Read next:** <the specific doc, guide, or config reference to open first>

**Flip conditions:**
- <observable event that reopens this decision>
- <observable event that reopens this decision>

Keep it to one screen. A tool record that runs to three pages has absorbed the comparison it was supposed to summarize.

Worked example

A billing service needs a database migration tool. The analysis has already happened; this is the record it produces.

# 0007 - Database migration tool for the billing service

**Status:** accepted 2026-07-19

**Signal:**
services/billing/pyproject.toml:14
> django = "^5.2"

services/billing/billing/migrations/0001_initial.py exists and is tracked.

**Decision:** We will use Django's built-in migrations.

**Rationale:**
- Fits because: the service's schema is already defined as Django ORM models,
  and Django ships a migrations system that propagates model changes into the
  database schema via `makemigrations` and `migrate`, storing migration files
  in a `migrations` directory inside each app
  (https://docs.djangoproject.com/en/5.2/topics/migrations/). The tracked
  `0001_initial.py` shows that history has already started here.
- Not Alembic, because: Alembic is "a lightweight database migration tool for
  usage with the SQLAlchemy Database Toolkit for Python"
  (https://alembic.sqlalchemy.org/en/latest/). Adopting it would mean
  maintaining a second schema definition alongside the Django models, and
  abandoning the migration history already committed.

**Secondary fallback:** none. The two candidates are not co-equal: the ORM in
the signal selects one of them outright.

**Read next:** the Django migrations topic guide
(https://docs.djangoproject.com/en/5.2/topics/migrations/), specifically the
sections on dependencies between app migrations and on squashing.

**Flip conditions:**
- The billing service moves off the Django ORM to SQLAlchemy.
- A second, non-Django service needs to share this migration history.
- The team adopts a database engine with no supported Django backend.

Note what the record does not contain: no scoring table, no third candidate, no paragraph about either tool's community size. Everything in it is either quoted from the project or cited to a document the reader can open.

Review checklist

Reject a draft record that fails any line:

• [ ] Signal quotes a manifest, lockfile, config file, or a non-empty test or source directory, with a path.
• [ ] Decision names exactly one tool.
• [ ] Rationale contains an explicit why-not clause naming a real alternative.
• [ ] Every factual claim in Rationale is traceable to the signal or to a linked document.
• [ ] Read next points at one specific document, not a reading list.
• [ ] Flip conditions lists at least one observable event, or states why none is foreseeable.
• [ ] Secondary fallback, if present, carries the constraint that selects it.
• [ ] Status is set, and any reversal is a new record rather than an edit.
• [ ] The record fits on one screen.

Anti-patterns

Limitations

• This format records a decision. It does not evaluate candidates, score them, or price them. The SKILL.md spine and vendor-evaluation.md (opens in new window) are the comparison side.
• The signal rule cannot see intent that exists only in people's heads. A planned migration off the current stack is invisible to a manifest, so it belongs in Flip conditions and must be volunteered by the team.
• Status vocabularies are not standardized. Proposed, accepted, and superseded are the minimum from the original ADR format; longer lifecycles such as initiating, researching, evaluating, implementing, maintaining, and sunsetting are also in use, and teams customize status values to fit their workflow (source (opens in new window)). Pick one vocabulary per decision log and keep it stable.
• A record is only as durable as the log it lives in. Records scattered across ticket comments do not form a decision log (adr.github.io (opens in new window)); keep them in one directory in the repository they describe.

Reference directory layouts

Deep reference for the framework-choice-advisor SKILL.md, Step 4. After the team has chosen a stack, these are the canonical directory layouts the per-framework skill assumes. Layouts are conventions, not mandates - every project has reasons to deviate, but the canonical layout is the starting point a newcomer can read.

Playwright + Jest (TypeScript) - the 2026 default for web E2E

tests/
├── e2e/
│   ├── auth/
│   │   ├── login.spec.ts
│   │   └── login.fixture.ts
│   ├── cart/
│   │   ├── add-item.spec.ts
│   │   └── checkout.spec.ts
│   └── pages/                  # Page Objects (per Martin Fowler's pattern)
│       ├── LoginPage.ts
│       ├── CartPage.ts
│       └── CheckoutPage.ts
├── helpers/
│   ├── api-client.ts            # HTTP client for setup / teardown
│   ├── test-data.ts             # Fixtures and seeds
│   └── selectors.ts             # Shared accessibility-first locators
├── fixtures/                    # Static test data
├── playwright.config.ts
├── tsconfig.json
└── package.json

Conventions:

• One *.spec.ts per feature flow; one Page Object per page or major component.
• Fixtures scoped to describe blocks; global fixtures are an anti-pattern (see test-code-conventions §6).
• Page Objects follow Martin Fowler's pattern (opens in new window): they wrap a page with an application-specific API and do not make assertions.

Cypress + Mocha (TypeScript)

cypress/
├── e2e/
│   ├── auth/login.cy.ts
│   └── cart/checkout.cy.ts
├── support/
│   ├── commands.ts              # Custom Cypress commands
│   ├── pages/                   # Page Objects (Cypress idiom: command-based, not class-based)
│   └── e2e.ts
├── fixtures/
├── cypress.config.ts
└── package.json

Cypress idiom prefers custom commands over class-based POMs; the directory layout reflects that.

Selenium / WebdriverIO (TypeScript or Java)

test/
├── specs/
│   ├── auth/login.spec.ts
│   └── cart/checkout.spec.ts
├── pageobjects/
│   ├── login.page.ts
│   └── cart.page.ts
├── helpers/
├── wdio.conf.ts
└── package.json

WDIO's runner ergonomics improve on raw Selenium; the layout is conventional.

Commercial vendor evaluation

Deep reference for the framework-choice-advisor SKILL.md. The commercial-procurement counterpart of the open-source framework decision: a side-by-side vendor evaluation matrix for QA tools - test-management platforms (TestRail / Qase / Xray / Zephyr / TestCollab), no-code platforms (mabl / Testim / Functionize / TestSigma / Reflect), visual regression services (Applitools / Percy / Chromatic), and commercial AI copilots - scoring each on capability fit, cost model, integration depth, vendor lock-in risk, exit cost, contractual posture, and customer-reference data.

Scoped to commercial procurement - contract, lock-in, and exit-cost axes - not to choosing an open-source code-first framework on architectural fit (that is the SKILL.md spine). The evaluation refuses to recommend a winner; the team owns the procurement choice.

Overview

QA managers procure commercial tools every 12 - 24 months: a new test-management platform, a no-code automation vendor, a visual-regression service, an AI copilot tier. The Capgemini World Quality Report 2025-26 (opens in new window) identifies integration friction (37% of teams) as the dominant blocker for AI-in-testing adoption - the proximate failure mode is teams that adopted a vendor without scoring integration cost in advance. This workflow produces the structured side-by-side comparison the manager carries into the procurement decision, with every score citing its source.

This is decision-support, not recommendation. The output is the evidence pack: capability matrix, cost-model breakdown, integration / lock-in / exit-cost analysis, and contractual posture per vendor. The manager (or a procurement committee) makes the call against the team's NFR priorities.

When to run one

• A 12 - 24 month procurement cycle is approaching for a commercial QA tool.
• The team is evaluating a switch from one vendor to another (mid-contract renewal, post-acquisition consolidation).
• A new tool category is being adopted (the team has no incumbent - e.g., first visual-regression service).
• A vendor is being added to a multi-vendor stack and the team wants to ensure no overlap with incumbents.
• An RFP / RFI / procurement-committee process needs a structured evaluation artifact.

Do not use this workflow when:

• The decision is between open-source code-first frameworks (Playwright vs Cypress vs Selenium) - use the SKILL.md spine. Different axis entirely (architecture, not procurement; no contract, lock-in, or exit-cost dimensions).
• Only one vendor is being evaluated - comparison requires ≥2 candidates. (For a single-vendor go/no-go, use the team's standard procurement checklist; this workflow needs comparison anchors.)
• The team has no defined NFR priorities - the matrix cannot score "capability fit" without knowing what the team needs the tool to do.

Step 1 - Capture the inputs

Required:

Halt with INSUFFICIENT_INPUT if any required input is missing.

Step 2 - Score on the seven procurement axes

Score each vendor on each axis, with every score citing its source.

Axis A1 - Capability fit

How well does the vendor's feature set match the team's documented NFR priorities?

Axis A2 - Cost model

How is the vendor priced and what does it cost at the team's scale?

Axis A3 - Integration depth with existing stack

Per the Capgemini WQR 2025-26 (opens in new window) finding (37% blocked by integration friction), this axis is often under-weighted in procurement. Weight it explicitly.

For each integration, score: native (1.0) / API-buildable (0.7) / community-plugin (0.5) / not available (0.0). The per-vendor result-sync baselines (test-management-sync, test-management-sync, test-management-sync, currents-integration in the qa-test-reporting plugin) feed this axis.

Axis A4 - Vendor lock-in risk

The cost of being unable to leave.

Axis A5 - Exit cost

The team has decided to leave in 24 months - what does it cost?

Axis A6 - Contractual posture

Procurement / legal / security review needs structured data.

Axis A7 - Customer-reference data

Independent (not vendor-published) signal.

Step 3 - Emit the comparison matrix

Emit a single markdown document: the vendors compared, the team profile, the ordered NFR priorities, a per-axis matrix (A1-A7), a weighted-score table using the NFR order as weights, an explicit "What this workflow did NOT do" disclaimer (does not pick the winner, negotiate, validate vendor claims, or replace a reference call), and an evidence appendix tracing every cell to a source.

Worked example (values illustrative)

# Vendor evaluation - `<category>` - `<team>` - 2026-07

## Vendors compared

| Code | Vendor | Pricing page | Cited integration doc |
|---|---|---|---|
| V1 | TestRail (Gurock / Idera) | https://www.testrail.com/pricing/ | https://support.testrail.com/hc/en-us/articles/7077873061908 |
| V2 | Qase | https://www.qase.io/pricing/ | https://docs.qase.io/en/articles/6417206-github |
| V3 | Xray (Xpand IT, for Jira) | https://marketplace.atlassian.com/apps/1211769/xray-test-management-for-jira | https://docs.getxray.app/display/XRAYCLOUD/REST+API |

## Team profile

- Size: 12 QA engineers
- Stack: Playwright + Jest, GitHub Actions, Linear (tracker), Datadog (observability)
- Geography: distributed US + EU; data-residency: EU required
- Regulated-industry: no
- Time horizon: 24 months

## NFR priorities (manager-supplied, ordered)

1. Integration with Linear + GitHub Actions
2. Cost at year-2 (team will grow to 18 engineers)
3. Data residency (EU)
4. Test-history portability (exit-cost matters; 24-month horizon)
5. SSO (SAML / OIDC)
6. Capability fit
7. Customer-reference depth

## Per-axis matrix

### A1 - Capability fit

| Vendor | Score (0-1.0) | Strengths | Gaps |
|---|---|---|---|
| TestRail | 0.85 | Mature test-case management, custom fields, bulk import / export | API rate limits documented at 180/min - may bind at scale |
| Qase | 0.80 | Modern UI, AI-assisted case authoring | Smaller plugin ecosystem |
| Xray | 0.95 | Deep Jira integration, BDD-native | Heavyweight Jira dependency the team doesn't have |

### A2 - Cost model

| Vendor | Year-1 (12 eng) | Year-2 (18 eng) | Hidden costs |
|---|---|---|---|
| TestRail | $5,328 (12 × $37/seat/mo Professional × 12) | $7,992 | SSO, automated backups, priority support are Enterprise-tier only |
| Qase | $4,320 (12 × $30/seat/mo Business × 12) | $6,480 | None at this tier; SSO included from Business plan |
| Xray | Quote from the Marketplace listing - Xray licenses by total Jira user tier, not by tester seat | Same tier rule at 18 engineers; re-quote if the Jira tier changes | Requires Jira Software seats if not already licensed |

### A3 - Integration depth

| Vendor | CI (GitHub Actions) | Tracker (Linear) | Observability (Datadog) | Test-framework (Playwright) | SSO |
|---|---|---|---|---|---|
| TestRail | Native (1.0) | API-buildable (0.7) | Community plugin (0.5) | API + `testrail-cli` (0.9) | SAML / OIDC (1.0) |
| Qase | Native action (1.0) | Native (1.0) | Webhook (0.7) | Native @qase/playwright (1.0) | SAML (Business+) (0.8) |
| Xray | API only (0.7) | API-buildable (0.7) | None native (0.0) | xray-junit-extensions (0.9) | SAML / OIDC (1.0) |

### A4 - Vendor lock-in risk

| Vendor | Format | Export | Lock-in score |
|---|---|---|---|
| TestRail | Proprietary case format; bulk CSV export | Documented CSV / XML export, JSON via API | Moderate (0.6) - export possible, but tests need re-authoring on migration |
| Qase | YAML / JSON case format; native import / export | First-class export to JSON / YAML | Low (0.85) - portable artifacts |
| Xray | BDD-native (Gherkin), JUnit / Cucumber export | Tied to Jira issue model; export possible but harder to disentangle | Moderate-high (0.5) - Jira coupling is the lock-in axis |

### A5 - Exit cost (24-month migration scenario)

| Vendor | Test re-authoring | History portability | Total exit cost (hand-wave) |
|---|---|---|---|
| TestRail | Cases portable as CSV; ~30% needs re-authoring for new tool | History exportable via API | ~3 person-months |
| Qase | YAML / Gherkin cases mostly portable; ~10% re-authoring | Native export | ~1 person-month |
| Xray | Gherkin scenarios portable; Jira-issue history harder to extract | API export; needs custom tooling | ~4 person-months |

### A6 - Contractual posture

| Vendor | SLA | Support | Security | EU residency |
|---|---|---|---|---|
| TestRail | 99.9% (Cloud), no SLA for self-hosted | Email; phone on Enterprise | SOC 2 Type II + ISO 27001 (cite vendor security page) | EU AWS region available on Enterprise |
| Qase | 99.9% on Business+ | Email + chat; CSM on Enterprise | SOC 2 Type II (cite) | EU region available on Business |
| Xray | Bound to Jira's SLA | Email + chat | SOC 2 Type II inherited from Xpand IT | Tied to Jira region |

### A7 - Customer-reference data

| Vendor | Gartner Peer Insights | G2 (recency / density) | Practitioner-signal |
|---|---|---|---|
| TestRail | 4.4/5 (382 reviews, mostly 2023-25) | 4.2/5, 250+ reviews | Cited in Lisa Crispin's *Agile Testing Condensed* |
| Qase | 4.6/5 (180 reviews, 2024-26) | 4.7/5, 120+ reviews | Featured in TestBash 2025 case studies |
| Xray | 4.4/5 (290 reviews) | 4.4/5, 200+ reviews | Heavy enterprise adoption signal; lighter mid-market |

## Weighted score per NFR priorities

| Axis | Weight (per team NFR order) | TestRail | Qase | Xray |
|---|---|---|---|---|
| A3 Integration | 0.25 | 0.78 | 0.92 | 0.62 |
| A2 Cost | 0.20 | 0.65 | 0.95 | 0.70 |
| A6 EU residency | 0.15 | 0.80 | 0.90 | 0.70 |
| A5 Exit cost | 0.15 | 0.60 | 0.90 | 0.50 |
| A6 SSO | 0.10 | 1.00 | 0.80 | 1.00 |
| A1 Capability fit | 0.10 | 0.85 | 0.80 | 0.95 |
| A7 Customer-reference | 0.05 | 0.90 | 0.85 | 0.85 |
| **Total** | **1.00** | **0.76** | **0.89** | **0.69** |

## What this workflow did NOT do

- Pick the winner. The matrix and weighted scores are the input to the procurement decision; the team owns the choice. The team may legitimately pick the lower-scored vendor for reasons outside the matrix (existing relationship, hiring-pool, founder preference).
- Negotiate the contract. Once a vendor is picked, contract terms (discount, multi-year commit, SLA tier) are a separate procurement conversation.
- Validate vendor claims. Where the matrix cites vendor-data (pricing, feature lists, case studies), the data is vendor-published and should be re-verified in a sales call before commitment.
- Replace a reference call. Customer references should be called directly, not just scored from public-review averages.

## Evidence appendix

Every cell in the matrix above traces to a source URL or cited document. The full appendix lists every source (per axis × per vendor) so the team can spot-check.

Step 4 - Hand off to procurement / decision committee

The matrix is the input to the decision, not the decision itself. Downstream:

1. Vendor sales call to re-verify pricing, SLA, and roadmap claims.
2. Customer reference call to 2 - 3 named customers per finalist.
3. Security review of SOC 2 / ISO 27001 reports.
4. Pilot / POC on the top-2 candidates (typically 30 - 60 day evaluation).
5. Procurement-committee decision with the matrix as the structured evidence pack.

Anti-patterns

Limitations

• Vendor-data dominates the matrix. Pricing pages, feature lists, and case studies are vendor-published. Flag these explicitly; the team verifies in sales calls.
• Customer-reference scoring is shallow. Public-review averages (Gartner, G2) are noisy. Real customer calls are the high-signal version; the matrix supplements but doesn't replace.
• Categories drift fast. A 2026 evaluation is stale by 2028; product features, pricing, and vendor stability all change. Re-run before each renewal cycle.
• No competitive-dynamics axis. The matrix doesn't account for vendor M&A risk (TestRail's parent Idera, mabl's growth stage, etc.). Add as an A8 if the team's horizon is >24 months.
• Local market variation. Pricing and contract terms differ by geography; the matrix uses US-list pricing unless the team specifies otherwise.
• No regulatory-specific guidance. Regulated-industry teams (healthcare, finance, automotive) have additional contractual axes (BAA, HIPAA, FDA Class II evidence) not enumerated per industry. Layer those onto A6.
• Honest about being a draft. The matrix is the manager's starting point; the procurement committee customises weights, adds axes, and replaces data after the sales / reference / security calls.

References

• Capgemini World Quality Report 2025-26 - 37% of teams cite integration friction as the dominant AI-in-testing blocker; load-bearing for axis A3: https://www.capgemini.com/insights/research-library/world-quality-report-2025-26/
• Gartner Peer Insights - AI-augmented software testing category: https://www.gartner.com/reviews/market/ai-augmented-software-testing-tools
• G2 / Capterra - methodology disclosure for review-density and recency scoring (general SaaS evaluation context; not QA-specific): https://www.g2.com/about
• ISTQB glossary - test automation framework (the open-source / commercial boundary): https://glossary.istqb.org/en_US/term/test-automation-framework
• ISO/IEC 25010 - quality model for non-functional requirements (used in A1 capability scoring): https://en.wikipedia.org/wiki/ISO/IEC_25010
• test-management-sync, test-management-sync, test-management-sync, currents-integration (qa-test-reporting plugin) - per-vendor integration baselines that feed A3.