Overview
Decision support for picking a test stack (framework, runner, reporter, CI shape): score the project's needs, compare per layer, then record one decision or an explicit deferral. It does not generate boilerplate; after the choice, the per-framework skill (playwright-testing, cypress-testing, ...) configures it.
When to use
Do not use this skill when:
Step 1 - Frame the decision against the project's NFRs
Six NFR axes drive framework choice. Score each 1 - 5 for the project; rank them by priority. The framework matrix in Step 2 uses these scores.
| NFR axis | Question |
|---|---|
| Cross-browser scope | Is multi-browser execution required? (Chromium-only? + Firefox + WebKit? + IE/Edge legacy?) |
| Mobile scope | Real device + emulator? Hybrid app webview? Native-only? Or web-mobile-viewport only? |
| Team language | What languages do the engineers already know? (Avoiding the framework-language mismatch is the #1 maintenance cost.) |
| Execution speed | Parallel-shard target - minutes for the full suite. CI-cost-driven? |
| Ecosystem maturity | Third-party integrations the team needs (visual regression, accessibility, perf, contract). |
| Hire-ability | Can the team hire engineers familiar with the framework? Smaller frameworks → smaller talent pool. |
Checkpoint before Step 2. Write the six scores down, each with the evidence behind it (a manifest line, a CI wall-clock figure, the team's language list, a roadmap item with a date), and get the team to confirm the ranking. If a top-ranked axis cannot be scored from evidence today (language not decided, no stable product surface), stop and go to Step 6 (defer) instead of reading the matrix. A roadmap line ("Android in H2") is a condition that would reopen the decision, not a score.
Step 2 - Framework tradeoff matrix (web E2E)
| Framework | Cross-browser | Mobile | Language | Speed (parallel) | Ecosystem | Hire-ability | Notes |
|---|---|---|---|---|---|---|---|
| Playwright | Chromium / Firefox / WebKit native, all in one runtime | Mobile-viewport emulation; experimental Android support (Chrome for Android, WebView) (Android API (opens in new window)) | TS / JS / Python / .NET / Java | Excellent (auto-parallel, sharding built-in) | Strong (trace viewer, visual snapshots, fixtures, MCP integration) | High | A common default for greenfield web E2E; its built-in auto-waiting removes the manual-wait flakiness that dominates Selenium suites (Playwright actionability (opens in new window)). |
| Cypress | Chromium-family + Firefox; WebKit only behind experimentalWebKitSupport (experiments (opens in new window)) | Mobile viewport only; no real-device | JS / TS only | Good (parallel via Cypress Cloud; CLI-parallel limited) | Strong (huge plugin ecosystem) | High | Strong DX for component testing; runs inside-browser limits cross-origin and iframe scenarios. |
| Selenium / WebdriverIO | All browsers via WebDriver protocol | Real device via Appium | All major languages (Java / Python / C# / JS / Ruby) | Moderate (Selenium Grid; WDIO improves on Selenium's runner) | Mature (oldest ecosystem) | Highest (historical talent pool) | Mature but more flake-prone than Playwright: async-wait is the single largest flake category at 45% (Luo et al. 2014 (opens in new window)), and Selenium leaves that synchronization manual. Migration target, not greenfield default. |
| TestCafe | All browsers; proxy-based (no WebDriver) | Mobile via emulators | JS / TS | Moderate | Smaller ecosystem | Lower | Niche; integrated runner. |
| Puppeteer | Chrome and Firefox (stable Firefox from v23, supported browsers (opens in new window)) | Limited | JS / TS | Good | Smaller than Playwright | Lower | Browser automation library rather than a test runner; pair it with Jest or similar. |
Decision tree for greenfield web E2E:
Step 3 - Framework tradeoff matrix (other test layers)
The per-layer matrices (mobile native, API / contract, performance) are in references/other-layers.md. The short rules:
Step 4 - Reference directory layouts
After the team has chosen a stack, adopt the canonical directory layout the per-framework skill assumes - a convention, not a mandate, but the starting point a newcomer can read. The layouts for Playwright + Jest, Cypress + Mocha, and Selenium / WebdriverIO are in references/directory-layouts.md.
Step 5 - CI integration patterns
Universal across frameworks:
| Concern | Convention |
|---|---|
| Parallelisation | Shard by file (Playwright --shard=X/Y, Cypress Cloud, WDIO maxInstances). Aim for 5 - 10 minute wall-clock for the full suite per shard. |
| Retries | Retry once on first failure; never retry locally (only CI). Tests retried >1× are flake candidates - triage them. |
| Trace / video | Capture on-first-retry (off for green runs to save storage). Playwright trace: 'on-first-retry' is the default; Cypress + cypress-video-trim similar. |
| Reporting | JUnit XML output for the CI's test-result panel; Allure for human reporting; both via plugin. |
| Secrets | Load from CI secret store (GitHub Actions Secrets, GitLab CI Variables); never commit. |
| Environment matrix | One job per (framework, browser, environment) cell; do not mix in one job. |
Step 6 - When to defer the decision
The skill recommends deferring framework choice when:
In these cases, the right output is an explicit deferral note: "no decision today; revisit when (a)/(b)/(c) resolves."
Step 7 - Commercial vendors (procurement axis)
When the decision is a commercial tool - a test-management platform (TestRail / Qase / Xray / Zephyr), a no-code automation vendor (mabl / Testim / Functionize), a visual-regression service (Applitools / Percy / Chromatic), or an AI copilot tier - the axes change: contract, lock-in, and exit cost matter as much as capability. Run the seven-axis vendor evaluation (capability fit, cost model, integration depth, lock-in risk, exit cost, contractual posture, customer-reference data) in references/vendor-evaluation.md. It emits an evidence pack with a weighted score per the team's NFR order and deliberately refuses to pick the winner - the team owns the procurement choice.
Step 8 - Record the decision
Whichever path produced the choice, write it down as a decision record so a later reader can see the signal, the rejected alternative, and what would reverse it. The ADR-based format - observed project signal, exactly one primary recommendation, a why-not clause, one read-next link, and mandatory flip conditions - is in references/decision-record-format.md.
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Picking the framework before the NFRs are scored | Choice driven by hype, not fit; high migration cost when the wrong framework can't deliver. | Step 1 - score the NFRs first. |
| Standardising on one framework across every test layer | Different layers need different tools (Playwright for E2E ≠ k6 for perf ≠ Pact for contract). | Pick per layer; the stack is multiple frameworks. |
| Picking Selenium for greenfield in 2026 | Manual async-wait is the dominant flake category at 45% (Luo et al. 2014 (opens in new window)), and Selenium does not auto-wait. | Use Playwright for greenfield; reserve Selenium for legacy maintenance. |
| Cross-language teams picking a single-language framework | Engineers can't contribute; suite becomes one person's domain. | Either pick a multi-language framework (Playwright / Selenium) or commit to retraining. |
| Adopting a framework because a contractor used it | Contractor leaves; team can't maintain. | Hire-ability is an NFR. |
| Skipping the directory-layout convention | Every newcomer authoring tests in a different shape; review burden grows. | Step 4 - pick a canonical layout up front, even if you deviate later. |
| Treating this skill as "framework recommender" rather than "decision support" | The skill recommends; the team decides. Automating the decision strips accountability. | The output of this skill is a documented choice, not an automatic install. |
Limitations
Hand-off targets
References
Tool-selection decision record format
View source (opens in a new tab) (opens in new window)Deep reference for the framework-choice-advisor SKILL.md. The output contract for writing down a chosen developer tool as a portable decision record: the observed project signal, exactly one primary recommendation, rationale that names the rejected alternative, what to read next, and a mandatory list of the conditions that would flip the choice.
The differentiation axis is output contract versus analysis: the SKILL.md spine (and the vendor evaluation in vendor-evaluation.md (opens in new window)) compare candidate tools on their merits; this format governs only the shape of what gets committed once the weighing is finished. Use both together: the analysis chooses, this record preserves the choice.
Overview
A tool gets chosen once and questioned for years. The choice survives in a chat thread, the reasoning does not, and the next person either re-litigates it from scratch or inherits it as folklore.
This format is a narrowed Architecture Decision Record. It keeps the ADR skeleton and tightens three things generic ADR templates leave open - how many tools may be recommended (Rule 1), what counts as admissible evidence for the context (Rule 2), and whether the reversal conditions are optional (Rule 3). It is tool-agnostic: the same record shape works for a test framework, a package manager, a linter, a migration tool, a CI runner, or a logging library.
What this record is and is not
| This record | Not this record |
|---|---|
| The written artefact produced after a tool is picked | The comparison that picks it |
| One decision, one tool, one file | A matrix scoring every candidate on every axis |
| Evidence taken from the target project | Evidence taken from vendor feature pages |
| Expires when its own stated conditions occur | Expires when someone happens to notice it is stale |
ADR provenance and field mapping
An ADR is "a document that captures an important architecture decision made along with its context and consequences" (joelparkerhenderson/architecture-decision-record (opens in new window)), and the canonical five-part shape is Title, Context, Decision, Status, Consequences (Nygard, Documenting Architecture Decisions (opens in new window)). A tool choice is a decision of exactly that kind: it "addresses a functional or non-functional requirement that is architecturally significant" (adr.github.io (opens in new window)).
| ADR field (Nygard (opens in new window)) | Field here |
|---|---|
| Title | Title |
| Status | Status |
| Context ("the forces at play") | Signal, restricted to observed project evidence |
| Decision | Decision, restricted to one tool |
| Consequences ("all consequences ... not just the positive ones") | Rationale (the why-not clause) plus Flip conditions |
The split of Consequences into two fields is deliberate. Nygard requires that "all consequences should be listed here, not just the 'positive' ones" (source (opens in new window)), and in practice tool records collapse to a list of benefits unless the negative half has its own heading with its own required content.
The record fields
Eight fields. Seven are always required; the eighth appears only under the co-equal rule below.
| Field | Required | Contents |
|---|---|---|
| Title | yes | Short noun phrase naming the decision, prefixed with a sequence number. Nygard specifies "short noun phrases" for ADR titles (source (opens in new window)). |
| Status | yes | One of proposed, accepted, or superseded by a named later record. Per Nygard, "a decision may be 'proposed' if the project stakeholders haven't agreed with it yet, or 'accepted' once it is agreed" (source (opens in new window)). |
| Signal | yes | The observed project evidence, quoted: file path plus the line or block that drove the detection. This is the ADR Context field, restricted to observable forces. |
| Decision | yes | Exactly one tool, written in active voice. Nygard's Decision field uses "full sentences, with active voice. 'We will ...'" (source (opens in new window)). |
| Rationale | yes | Two clauses minimum: why the chosen tool fits the signal, and why not the strongest alternative that was considered. |
| Read next | yes | The one document the implementer opens first: the tool's setup guide, config reference, or migration guide. One link, not a reading list. |
| Flip conditions | yes | The specific future observations that would reopen this decision. |
| Secondary fallback | only for genuine co-equals | One alternative, with the constraint that would select it instead. |
Rule 1: exactly one primary recommendation
A record names one tool in the Decision field. Two tools in that field is not a decision, it is the reader's decision deferred.
This follows the ADR specificity convention: a well-written record "should be about one AD, not multiple ADs" (joelparkerhenderson/architecture-decision-record (opens in new window)).
The one exception is a genuine co-equal: two candidates that the available signal cannot separate, where the tiebreak depends on a constraint nobody has stated yet (team size, an unwritten hiring plan, a language the team may or may not adopt). Then, and only then:
A Secondary fallback entry without that constraint line is a tiebreak handed back to the reader. Delete it or complete it.
Never list a third option. If three candidates look co-equal, the signal was too weak to write a record at all, and Rule 2 applies instead.
Rule 2: every record rests on an observed signal
A recommendation is never inferred from a README, a project description, a folder name, or what the team says the stack is. The Signal field must quote one of:
| Admissible signal | Examples |
|---|---|
| A manifest | package.json, pom.xml, pyproject.toml, *.csproj, Cargo.toml, go.mod |
| A lockfile | package-lock.json, pnpm-lock.yaml, poetry.lock, Gemfile.lock |
| A tool config file | playwright.config.ts, .eslintrc.json, tsconfig.json, Dockerfile, a CI workflow file |
| An existing test or source directory with real contents | tests/e2e/, src/androidTest/, migrations/ |
Inadmissible on its own: a README paragraph, a wiki page, a folder named after a framework with nothing in it, a compiled artefact, or a verbal claim about the stack. Each of these describes intent rather than state, and intent and state diverge constantly.
This is the ADR Context field taken literally. Context "describes the forces at play" in neutral language (Nygard (opens in new window)); a force that cannot be quoted from the repository is not yet a force, it is a plan. Plans belong in Flip conditions, not in Signal.
When no admissible signal exists, do not write a record. Say so, name the file that would resolve it, and stop. A record built on a guess is worse than no record: it launders the guess into a citable decision.
Rule 3: the flip conditions field is mandatory
Every record declares the observations that would reopen it. This is what keeps a decision log from silently rotting into a set of unchallengeable defaults.
Good flip conditions are observable and specific:
Bad flip conditions are unfalsifiable: "if requirements change", "if the tool stops meeting our needs", "on annual review".
If the author genuinely cannot foresee a reversal condition, the field says so explicitly and gives the reason. An empty field reads as an oversight; a stated "no foreseeable condition, because the choice is forced by the runtime" reads as a claim a later reader can attack.
Flip conditions are also the confirmation hook. MADR carries an optional Confirmation section describing how compliance with the decision gets verified (MADR template (opens in new window)); in this format the flip conditions are what a periodic review actually checks.
Rule 4: supersede, never edit
When a flip condition fires, the old record is not rewritten. Nygard: "If a decision is reversed, we will keep the old one around, but mark it as superseded" (source (opens in new window)). The same immutability guidance appears in the widely used ADR collection: when an earlier decision becomes invalid, "a new ADR should be created" rather than modifying the original (source (opens in new window)).
So: write a new record, set the old one's Status to superseded with a pointer, and let the pair show the reader both the original reasoning and what changed. The set of records accumulated this way is the project's decision log (adr.github.io (opens in new window)).
Template
# <NNNN> - <tool category> for <component>
**Status:** <proposed | accepted YYYY-MM-DD | superseded by NNNN>
**Signal:**
<file path>:<line>
> <quoted excerpt that drove the detection>
**Decision:** We will use <exactly one tool>.
**Rationale:**
- Fits because: <one line tying the tool to the quoted signal>
- Not <strongest alternative>, because: <one line>
**Secondary fallback:** <tool | none>
<if present: the one constraint that would select it instead>
**Read next:** <the specific doc, guide, or config reference to open first>
**Flip conditions:**
- <observable event that reopens this decision>
- <observable event that reopens this decision>Keep it to one screen. A tool record that runs to three pages has absorbed the comparison it was supposed to summarize.
Worked example
A billing service needs a database migration tool. The analysis has already happened; this is the record it produces.
# 0007 - Database migration tool for the billing service
**Status:** accepted 2026-07-19
**Signal:**
services/billing/pyproject.toml:14
> django = "^5.2"
services/billing/billing/migrations/0001_initial.py exists and is tracked.
**Decision:** We will use Django's built-in migrations.
**Rationale:**
- Fits because: the service's schema is already defined as Django ORM models,
and Django ships a migrations system that propagates model changes into the
database schema via `makemigrations` and `migrate`, storing migration files
in a `migrations` directory inside each app
(https://docs.djangoproject.com/en/5.2/topics/migrations/). The tracked
`0001_initial.py` shows that history has already started here.
- Not Alembic, because: Alembic is "a lightweight database migration tool for
usage with the SQLAlchemy Database Toolkit for Python"
(https://alembic.sqlalchemy.org/en/latest/). Adopting it would mean
maintaining a second schema definition alongside the Django models, and
abandoning the migration history already committed.
**Secondary fallback:** none. The two candidates are not co-equal: the ORM in
the signal selects one of them outright.
**Read next:** the Django migrations topic guide
(https://docs.djangoproject.com/en/5.2/topics/migrations/), specifically the
sections on dependencies between app migrations and on squashing.
**Flip conditions:**
- The billing service moves off the Django ORM to SQLAlchemy.
- A second, non-Django service needs to share this migration history.
- The team adopts a database engine with no supported Django backend.Note what the record does not contain: no scoring table, no third candidate, no paragraph about either tool's community size. Everything in it is either quoted from the project or cited to a document the reader can open.
Review checklist
Reject a draft record that fails any line:
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Two tools in the Decision field | The reader still has to choose, so nothing was decided; it also breaks the one-decision-per-record convention (source (opens in new window)) | Promote one, demote the other to Secondary fallback with its selecting constraint |
| Signal reads "this is a Node project" | A folder shape is a description, not evidence, and it goes stale silently | Quote the manifest or lockfile line that proves it |
| Rationale lists only advantages of the winner | Consequences must include the non-positive ones (source (opens in new window)) | Add the why-not clause and the cost the team is accepting |
| Flip conditions omitted or written as "revisit annually" | The record can never expire on evidence, so a stale choice outlives its reasons | Name observable events tied to the signal |
| The record is edited in place when the tool changes | The original reasoning is destroyed, so nobody can tell whether the reversal was justified | New record, old one marked superseded (source (opens in new window)) |
| Rationale paraphrases the tool's marketing page | Unfalsifiable, and it will read as true forever regardless of the project | Tie each clause to the quoted signal or to a linked technical document |
Limitations
Deep reference for the framework-choice-advisor SKILL.md, Step 4. After the team has chosen a stack, these are the canonical directory layouts the per-framework skill assumes. Layouts are conventions, not mandates - every project has reasons to deviate, but the canonical layout is the starting point a newcomer can read.
Playwright + Jest (TypeScript) - the 2026 default for web E2E
tests/
├── e2e/
│ ├── auth/
│ │ ├── login.spec.ts
│ │ └── login.fixture.ts
│ ├── cart/
│ │ ├── add-item.spec.ts
│ │ └── checkout.spec.ts
│ └── pages/ # Page Objects (per Martin Fowler's pattern)
│ ├── LoginPage.ts
│ ├── CartPage.ts
│ └── CheckoutPage.ts
├── helpers/
│ ├── api-client.ts # HTTP client for setup / teardown
│ ├── test-data.ts # Fixtures and seeds
│ └── selectors.ts # Shared accessibility-first locators
├── fixtures/ # Static test data
├── playwright.config.ts
├── tsconfig.json
└── package.jsonConventions:
Cypress + Mocha (TypeScript)
cypress/
├── e2e/
│ ├── auth/login.cy.ts
│ └── cart/checkout.cy.ts
├── support/
│ ├── commands.ts # Custom Cypress commands
│ ├── pages/ # Page Objects (Cypress idiom: command-based, not class-based)
│ └── e2e.ts
├── fixtures/
├── cypress.config.ts
└── package.jsonCypress idiom prefers custom commands over class-based POMs; the directory layout reflects that.
Selenium / WebdriverIO (TypeScript or Java)
test/
├── specs/
│ ├── auth/login.spec.ts
│ └── cart/checkout.spec.ts
├── pageobjects/
│ ├── login.page.ts
│ └── cart.page.ts
├── helpers/
├── wdio.conf.ts
└── package.jsonWDIO's runner ergonomics improve on raw Selenium; the layout is conventional.
Tradeoff matrices for other test layers
View source (opens in a new tab) (opens in new window)Deep reference for framework-choice-advisor Step 3. Score the Step 1 axes first; these tables only compare options inside one layer. Pick per layer: the stack is several tools, not one.
Mobile native
| Framework | Platform | Language | Notes |
|---|---|---|---|
| Espresso | Android native | Kotlin / Java | Google's first-party. In-process, fast, deterministic. |
| XCUITest | iOS native | Swift / Obj-C | Apple's first-party; UI tests drive the app from a separate test-runner process. |
| Appium | iOS + Android (and others) | All major | Cross-platform unifier; trades depth for breadth. WebDriver-based - same flake patterns as Selenium. |
| Detox | React Native | JS / TS | RN-specialist; grey-box testing. |
Decision: if the team is single-platform native (iOS only or Android only), use the first-party framework. Cross-platform → Appium, accept the WebDriver flake-tax. React-Native specifically → Detox.
API / contract
| Framework | Scope | Language | Notes |
|---|---|---|---|
| RestAssured | REST API integration tests | Java / Kotlin | The JVM-default; mature, deeply integrated with JUnit / TestNG. |
| Karate | REST + SOAP + GraphQL + gRPC | Karate DSL (Cucumber-like) | DSL-first; lowers barrier for non-Java testers. |
| schemathesis | OpenAPI / GraphQL property-based fuzzing | Python | Generative; complements example-based tests. |
| Pact | Consumer-driven contract tests | JS / JVM / Python / Go / Ruby / .NET | Different category - contract, not integration. See pact-contract-testing. |
| Postman / Newman | Collection-driven API tests | Postman DSL | UI-driven authoring; not code-first. Often used by non-engineers. |
Performance
| Framework | Scope | Language | Notes |
|---|---|---|---|
| k6 | Load + perf, code-first | JS (with TS support) | Grafana's; lowest barrier for engineers, excellent CI integration. |
| Locust | Load + perf, code-first | Python | Open-source; user-class-based modelling. |
| JMeter | Load + perf, GUI-first | XML config | Mature, ecosystem-heavy; GUI-driven authoring is the trade-off. |
| Gatling | Load + perf, code-first | Scala / Java / Kotlin | High-throughput; JVM stack. |
Sources
Deep reference for the framework-choice-advisor SKILL.md. The commercial-procurement counterpart of the open-source framework decision: a side-by-side vendor evaluation matrix for QA tools - test-management platforms (TestRail / Qase / Xray / Zephyr / TestCollab), no-code platforms (mabl / Testim / Functionize / TestSigma / Reflect), visual regression services (Applitools / Percy / Chromatic), and commercial AI copilots - scoring each on capability fit, cost model, integration depth, vendor lock-in risk, exit cost, contractual posture, and customer-reference data.
Scoped to commercial procurement - contract, lock-in, and exit-cost axes - not to choosing an open-source code-first framework on architectural fit (that is the SKILL.md spine). The evaluation refuses to recommend a winner; the team owns the procurement choice.
Overview
QA managers procure commercial tools every 12 - 24 months: a new test-management platform, a no-code automation vendor, a visual-regression service, an AI copilot tier. The Capgemini World Quality Report 2025-26 (opens in new window) identifies integration friction (37% of teams) as the dominant blocker for AI-in-testing adoption - the proximate failure mode is teams that adopted a vendor without scoring integration cost in advance. This workflow produces the structured side-by-side comparison the manager carries into the procurement decision, with every score citing its source.
This is decision-support, not recommendation. The output is the evidence pack: capability matrix, cost-model breakdown, integration / lock-in / exit-cost analysis, and contractual posture per vendor. The manager (or a procurement committee) makes the call against the team's NFR priorities.
When to run one
Do not use this workflow when:
Step 1 - Capture the inputs
Required:
| Input | Notes |
|---|---|
| ≥2 vendor candidates | The vendors being compared. Halts on 1; recommends ≥3 for a healthy comparison (avoids the two-choice false-binary). |
| Team profile | Team size, existing stack (CI, test framework, observability, tracker), seat / volume profile, geography, regulated-industry flag (if any). |
| NFR priorities | Ordered list of what the team needs most: capability fit / cost / integration / lock-in / contract / support / data-residency. The order matters - the matrix's weighted score depends on it. |
| Time horizon | 12 / 24 / 36 month decision window. Drives the lock-in and exit-cost scoring (longer horizon = more weight on lock-in). |
| Per-vendor data | Pricing page, feature page, integration docs, customer-reference reviews (Gartner Peer Insights, G2, Capterra), vendor-published case studies (tagged as vendor-data). |
Halt with INSUFFICIENT_INPUT if any required input is missing.
Step 2 - Score on the seven procurement axes
Score each vendor on each axis, with every score citing its source.
Axis A1 - Capability fit
How well does the vendor's feature set match the team's documented NFR priorities?
| Sub-axis | Scoring rubric (per-vendor) |
|---|---|
| Core feature coverage | % of team's required features present (cite the team's requirement list) |
| Advanced / aspirational features | Features the team doesn't need today but might in 2 years |
| Documented limits | Per-account / per-test / per-user caps that may bind |
Axis A2 - Cost model
How is the vendor priced and what does it cost at the team's scale?
| Sub-axis | Scoring rubric |
|---|---|
| Pricing model | Per-seat / per-test / per-execution / flat licence / hybrid |
| Cost at current team size | Year-1 cost, cited to vendor pricing page (vendor-data) |
| Cost at projected team size | Year-2 and year-3 projections; the team's growth plan drives this |
| Hidden costs | Add-ons (parallel execution, premium support, SSO, audit log, on-prem option) |
| Volume-discount commitments | Annual commits, multi-year discounts (cite to vendor sales channel) |
Axis A3 - Integration depth with existing stack
Per the Capgemini WQR 2025-26 (opens in new window) finding (37% blocked by integration friction), this axis is often under-weighted in procurement. Weight it explicitly.
| Sub-axis | Scoring rubric |
|---|---|
| CI integration | Native plugin? REST API? Webhook? CLI? Cite the integration doc URL per vendor |
| Tracker integration | Jira / Linear / GitHub Issues / Azure DevOps |
| Observability integration | Datadog / Grafana / New Relic / Sentry |
| Test-framework binding | Playwright / Cypress / Selenium / per-language |
| SSO / IAM | SAML / OIDC / SCIM provisioning |
| Reverse data flow | Can the team export raw test results? In what format? |
For each integration, score: native (1.0) / API-buildable (0.7) / community-plugin (0.5) / not available (0.0). The per-vendor result-sync baselines (test-management-sync, test-management-sync, test-management-sync, currents-integration in the qa-test-reporting plugin) feed this axis.
Axis A4 - Vendor lock-in risk
The cost of being unable to leave.
| Sub-axis | Scoring rubric |
|---|---|
| Proprietary data formats | Are tests in a portable format (Gherkin / standard JSON / open spec) or vendor-proprietary DSL? |
| Test artifact portability | Can the team export tests, results, history? In what format? Vendor-published export tools count toward portability. |
| Data residency | Where is data stored? Can the team request export and deletion? |
| Migration path | Are there documented or community-tested migration paths off this vendor (mabl -> Playwright, Testim -> Cypress)? |
Axis A5 - Exit cost
The team has decided to leave in 24 months - what does it cost?
| Sub-axis | Scoring rubric |
|---|---|
| Test re-authoring effort | If tests are vendor-DSL-bound, the migration is "rewrite from scratch." If tests are portable (Gherkin, standard fixtures), migration is mostly mechanical. |
| History portability | Can the team take its test-result history? Defect-history correlation depends on this. |
| Re-training | How long to retrain the team on the new vendor? |
| Data egress fees | Some vendors charge for bulk data export. Cite the contract clause if applicable. |
| Contract early-termination cost | Multi-year commits often have early-termination fees. |
Axis A6 - Contractual posture
Procurement / legal / security review needs structured data.
| Sub-axis | Scoring rubric |
|---|---|
| SLA tier | Uptime guarantee, support response time, escalation paths |
| Support tier | Email-only / chat / phone / dedicated CSM |
| Security audit availability | SOC 2 Type II report? ISO 27001? Penetration-test results? |
| On-prem / private-cloud option | For regulated industries; cite the vendor's deployment options page |
| Data-processing agreement | GDPR-compliant DPA available? HIPAA BAA? |
| Sub-processor disclosure | Vendor's sub-processor list (the regulated-industry view) |
Axis A7 - Customer-reference data
Independent (not vendor-published) signal.
| Sub-axis | Scoring rubric |
|---|---|
| Gartner Peer Insights | Rating, review density, recency. Cite the category report URL. |
| G2 / Capterra | Rating, review density. Flag if reviews are sparse or stale (<10 reviews in last 12 months). |
| Practitioner blog / conference signal | Has the vendor been written about by recognised practitioners (Lisa Crispin, James Whittaker, etc.)? Cite the source. |
| Reddit / r/QualityAssurance / Hacker News | Anecdotal community signal - tag as such. Don't weight equally with surveyed data. |
Step 3 - Emit the comparison matrix
Emit a single markdown document: the vendors compared, the team profile, the ordered NFR priorities, a per-axis matrix (A1-A7), a weighted-score table using the NFR order as weights, an explicit "What this workflow did NOT do" disclaimer (does not pick the winner, negotiate, validate vendor claims, or replace a reference call), and an evidence appendix tracing every cell to a source.
Worked example (values illustrative)
# Vendor evaluation - `<category>` - `<team>` - 2026-07
## Vendors compared
| Code | Vendor | Pricing page | Cited integration doc |
|---|---|---|---|
| V1 | TestRail (Gurock / Idera) | https://www.testrail.com/pricing/ | https://support.testrail.com/hc/en-us/articles/7077873061908 |
| V2 | Qase | https://www.qase.io/pricing/ | https://docs.qase.io/en/articles/6417206-github |
| V3 | Xray (Xpand IT, for Jira) | https://marketplace.atlassian.com/apps/1211769/xray-test-management-for-jira | https://docs.getxray.app/display/XRAYCLOUD/REST+API |
## Team profile
- Size: 12 QA engineers
- Stack: Playwright + Jest, GitHub Actions, Linear (tracker), Datadog (observability)
- Geography: distributed US + EU; data-residency: EU required
- Regulated-industry: no
- Time horizon: 24 months
## NFR priorities (manager-supplied, ordered)
1. Integration with Linear + GitHub Actions
2. Cost at year-2 (team will grow to 18 engineers)
3. Data residency (EU)
4. Test-history portability (exit-cost matters; 24-month horizon)
5. SSO (SAML / OIDC)
6. Capability fit
7. Customer-reference depth
## Per-axis matrix
### A1 - Capability fit
| Vendor | Score (0-1.0) | Strengths | Gaps |
|---|---|---|---|
| TestRail | 0.85 | Mature test-case management, custom fields, bulk import / export | API rate limits documented at 180/min - may bind at scale |
| Qase | 0.80 | Modern UI, AI-assisted case authoring | Smaller plugin ecosystem |
| Xray | 0.95 | Deep Jira integration, BDD-native | Heavyweight Jira dependency the team doesn't have |
### A2 - Cost model
| Vendor | Year-1 (12 eng) | Year-2 (18 eng) | Hidden costs |
|---|---|---|---|
| TestRail | $5,328 (12 × $37/seat/mo Professional × 12) | $7,992 | SSO, automated backups, priority support are Enterprise-tier only |
| Qase | $4,320 (12 × $30/seat/mo Business × 12) | $6,480 | None at this tier; SSO included from Business plan |
| Xray | Quote from the Marketplace listing - Xray licenses by total Jira user tier, not by tester seat | Same tier rule at 18 engineers; re-quote if the Jira tier changes | Requires Jira Software seats if not already licensed |
### A3 - Integration depth
| Vendor | CI (GitHub Actions) | Tracker (Linear) | Observability (Datadog) | Test-framework (Playwright) | SSO |
|---|---|---|---|---|---|
| TestRail | Native (1.0) | API-buildable (0.7) | Community plugin (0.5) | API + `testrail-cli` (0.9) | SAML / OIDC (1.0) |
| Qase | Native action (1.0) | Native (1.0) | Webhook (0.7) | Native @qase/playwright (1.0) | SAML (Business+) (0.8) |
| Xray | API only (0.7) | API-buildable (0.7) | None native (0.0) | xray-junit-extensions (0.9) | SAML / OIDC (1.0) |
### A4 - Vendor lock-in risk
| Vendor | Format | Export | Lock-in score |
|---|---|---|---|
| TestRail | Proprietary case format; bulk CSV export | Documented CSV / XML export, JSON via API | Moderate (0.6) - export possible, but tests need re-authoring on migration |
| Qase | YAML / JSON case format; native import / export | First-class export to JSON / YAML | Low (0.85) - portable artifacts |
| Xray | BDD-native (Gherkin), JUnit / Cucumber export | Tied to Jira issue model; export possible but harder to disentangle | Moderate-high (0.5) - Jira coupling is the lock-in axis |
### A5 - Exit cost (24-month migration scenario)
| Vendor | Test re-authoring | History portability | Total exit cost (hand-wave) |
|---|---|---|---|
| TestRail | Cases portable as CSV; ~30% needs re-authoring for new tool | History exportable via API | ~3 person-months |
| Qase | YAML / Gherkin cases mostly portable; ~10% re-authoring | Native export | ~1 person-month |
| Xray | Gherkin scenarios portable; Jira-issue history harder to extract | API export; needs custom tooling | ~4 person-months |
### A6 - Contractual posture
| Vendor | SLA | Support | Security | EU residency |
|---|---|---|---|---|
| TestRail | 99.9% (Cloud), no SLA for self-hosted | Email; phone on Enterprise | SOC 2 Type II + ISO 27001 (cite vendor security page) | EU AWS region available on Enterprise |
| Qase | 99.9% on Business+ | Email + chat; CSM on Enterprise | SOC 2 Type II (cite) | EU region available on Business |
| Xray | Bound to Jira's SLA | Email + chat | SOC 2 Type II inherited from Xpand IT | Tied to Jira region |
### A7 - Customer-reference data
| Vendor | Gartner Peer Insights | G2 (recency / density) | Practitioner-signal |
|---|---|---|---|
| TestRail | 4.4/5 (382 reviews, mostly 2023-25) | 4.2/5, 250+ reviews | Cited in Lisa Crispin's *Agile Testing Condensed* |
| Qase | 4.6/5 (180 reviews, 2024-26) | 4.7/5, 120+ reviews | Featured in TestBash 2025 case studies |
| Xray | 4.4/5 (290 reviews) | 4.4/5, 200+ reviews | Heavy enterprise adoption signal; lighter mid-market |
## Weighted score per NFR priorities
| Axis | Weight (per team NFR order) | TestRail | Qase | Xray |
|---|---|---|---|---|
| A3 Integration | 0.25 | 0.78 | 0.92 | 0.62 |
| A2 Cost | 0.20 | 0.65 | 0.95 | 0.70 |
| A6 EU residency | 0.15 | 0.80 | 0.90 | 0.70 |
| A5 Exit cost | 0.15 | 0.60 | 0.90 | 0.50 |
| A6 SSO | 0.10 | 1.00 | 0.80 | 1.00 |
| A1 Capability fit | 0.10 | 0.85 | 0.80 | 0.95 |
| A7 Customer-reference | 0.05 | 0.90 | 0.85 | 0.85 |
| **Total** | **1.00** | **0.76** | **0.89** | **0.69** |
## What this workflow did NOT do
- Pick the winner. The matrix and weighted scores are the input to the procurement decision; the team owns the choice. The team may legitimately pick the lower-scored vendor for reasons outside the matrix (existing relationship, hiring-pool, founder preference).
- Negotiate the contract. Once a vendor is picked, contract terms (discount, multi-year commit, SLA tier) are a separate procurement conversation.
- Validate vendor claims. Where the matrix cites vendor-data (pricing, feature lists, case studies), the data is vendor-published and should be re-verified in a sales call before commitment.
- Replace a reference call. Customer references should be called directly, not just scored from public-review averages.
## Evidence appendix
Every cell in the matrix above traces to a source URL or cited document. The full appendix lists every source (per axis × per vendor) so the team can spot-check.Step 4 - Hand off to procurement / decision committee
The matrix is the input to the decision, not the decision itself. Downstream:
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Scoring "capability fit" without the team's NFR priorities | The matrix favours feature-richest vendor regardless of fit | Step 1 NFR priorities are mandatory inputs |
| Equal-weight matrix | Treats integration, cost, and capability as equally important - almost never true | Step 3 weighted score per team's NFR order |
| Picking the vendor by gut from the matrix | Per the Capgemini WQR (opens in new window), integration cost is under-weighted; gut decisions favour capability over integration | The weighted-score column is the discipline; the team must justify deviations |
| Skipping A4 / A5 (lock-in / exit cost) because "we'll figure it out later" | The dominant cost surfaces at year-2+; skipping these axes optimises for year-1 happiness | These axes are mandatory in the matrix |
| Treating vendor-data and Gartner / G2 data identically | Vendor-data is marketing; reviewed data is signal | A7 explicitly separates them |
| Using the matrix as the procurement decision | Procurement requires sales / reference / security calls beyond the matrix | Step 4 hand-off lists the required downstream actions |
| Comparing only 2 vendors | Two-choice procurement is a false binary; the team often missed a third option | Step 1 recommends ≥3 candidates |
| Skipping the weighted score because "it feels mechanical" | Without weighting, the matrix is decoration | Step 3 weighted score is required |
| Auto-recommending the highest-scored vendor | The team's context (existing relationship, hire-ability, contract leverage) is outside the matrix; auto-recommend strips that context | The "What this workflow did NOT do" block explicitly disclaims the recommendation |
Limitations
References
- Both 2023 tooling records are stale now the migration has landed
- Tooling records for checkout and pricing before four contractors land
- Job ad goes out Monday and it needs to name our test stack
- Committee meets Thursday and I need one vendor name on the slide
- One tool for every layer by Q1 - I need the name and the bill