framework-choice-advisor

Use when picking a test automation framework or QA tool before adopting it: Playwright vs Cypress vs Selenium, Appium vs Espresso/XCUITest, RestAssured vs Karate, k6 vs Locust. Compares tradeoffs, matches project needs to a choice, covers commercial vendor evaluation and records the decision.

/plugin install qa-process@testland-qa

First time in Claude Code? Add the marketplace with /plugin marketplace add testland/qa

View source (opens in a new tab)

Overview

Decision support for picking a test stack (framework, runner, reporter, CI shape): score the project's needs, compare per layer, then record one decision or an explicit deferral. It does not generate boilerplate; after the choice, the per-framework skill (playwright-testing, cypress-testing, ...) configures it.

When to use

  • Greenfield: starting a new test automation suite from zero.
  • Migration: the team is moving off a legacy framework (for example Selenium to Playwright).
  • Multi-stack consolidation: the team has three frameworks across product areas and is deciding which to standardise on.
  • Hiring-driven re-evaluation: the team's skills mix shifted (e.g., from Java to TypeScript) and the framework choice should follow.

Do not use this skill when:

  • The framework is already chosen and the team needs configuration / API help - use the per-framework skill (playwright-testing, cypress-testing, etc.).
  • The team's question is "should we add E2E vs unit vs contract tests" - that's test-pyramid-balancer.
  • Mobile-native vs hybrid app selection - use the qa-mobile plugin's documentation directly.

Step 1 - Frame the decision against the project's NFRs

Six NFR axes drive framework choice. Score each 1 - 5 for the project; rank them by priority. The framework matrix in Step 2 uses these scores.

NFR axisQuestion
Cross-browser scopeIs multi-browser execution required? (Chromium-only? + Firefox + WebKit? + IE/Edge legacy?)
Mobile scopeReal device + emulator? Hybrid app webview? Native-only? Or web-mobile-viewport only?
Team languageWhat languages do the engineers already know? (Avoiding the framework-language mismatch is the #1 maintenance cost.)
Execution speedParallel-shard target - minutes for the full suite. CI-cost-driven?
Ecosystem maturityThird-party integrations the team needs (visual regression, accessibility, perf, contract).
Hire-abilityCan the team hire engineers familiar with the framework? Smaller frameworks → smaller talent pool.

Checkpoint before Step 2. Write the six scores down, each with the evidence behind it (a manifest line, a CI wall-clock figure, the team's language list, a roadmap item with a date), and get the team to confirm the ranking. If a top-ranked axis cannot be scored from evidence today (language not decided, no stable product surface), stop and go to Step 6 (defer) instead of reading the matrix. A roadmap line ("Android in H2") is a condition that would reopen the decision, not a score.

Step 2 - Framework tradeoff matrix (web E2E)

FrameworkCross-browserMobileLanguageSpeed (parallel)EcosystemHire-abilityNotes
PlaywrightChromium / Firefox / WebKit native, all in one runtimeMobile-viewport emulation; experimental Android support (Chrome for Android, WebView) (Android API (opens in new window))TS / JS / Python / .NET / JavaExcellent (auto-parallel, sharding built-in)Strong (trace viewer, visual snapshots, fixtures, MCP integration)HighA common default for greenfield web E2E; its built-in auto-waiting removes the manual-wait flakiness that dominates Selenium suites (Playwright actionability (opens in new window)).
CypressChromium-family + Firefox; WebKit only behind experimentalWebKitSupport (experiments (opens in new window))Mobile viewport only; no real-deviceJS / TS onlyGood (parallel via Cypress Cloud; CLI-parallel limited)Strong (huge plugin ecosystem)HighStrong DX for component testing; runs inside-browser limits cross-origin and iframe scenarios.
Selenium / WebdriverIOAll browsers via WebDriver protocolReal device via AppiumAll major languages (Java / Python / C# / JS / Ruby)Moderate (Selenium Grid; WDIO improves on Selenium's runner)Mature (oldest ecosystem)Highest (historical talent pool)Mature but more flake-prone than Playwright: async-wait is the single largest flake category at 45% (Luo et al. 2014 (opens in new window)), and Selenium leaves that synchronization manual. Migration target, not greenfield default.
TestCafeAll browsers; proxy-based (no WebDriver)Mobile via emulatorsJS / TSModerateSmaller ecosystemLowerNiche; integrated runner.
PuppeteerChrome and Firefox (stable Firefox from v23, supported browsers (opens in new window))LimitedJS / TSGoodSmaller than PlaywrightLowerBrowser automation library rather than a test runner; pair it with Jest or similar.

Decision tree for greenfield web E2E:

  • Multi-browser required + multi-language team → Playwright.
  • Single-browser (Chromium) + JS-only team → Playwright or Cypress; Cypress has stronger component-testing DX.
  • Legacy / migration off existing Selenium suite → either incremental migration to Playwright (preferred) or modernise the Selenium suite via WebdriverIO's runner ergonomics.

Step 3 - Framework tradeoff matrix (other test layers)

The per-layer matrices (mobile native, API / contract, performance) are in references/other-layers.md. The short rules:

  • Mobile: single-platform native app → that platform's first-party framework (Espresso, XCUITest); both platforms with one suite → Appium; React Native → Detox.
  • API / contract: JVM team → RestAssured; non-programmer authors → Karate or Postman / Newman; consumer-driven contracts are a separate decision (pact-contract-testing).
  • Performance: pick the tool in the team's language (k6 for JS, Locust for Python, Gatling for JVM); JMeter when a GUI-built plan is already in place.

Step 4 - Reference directory layouts

After the team has chosen a stack, adopt the canonical directory layout the per-framework skill assumes - a convention, not a mandate, but the starting point a newcomer can read. The layouts for Playwright + Jest, Cypress + Mocha, and Selenium / WebdriverIO are in references/directory-layouts.md.

Step 5 - CI integration patterns

Universal across frameworks:

ConcernConvention
ParallelisationShard by file (Playwright --shard=X/Y, Cypress Cloud, WDIO maxInstances). Aim for 5 - 10 minute wall-clock for the full suite per shard.
RetriesRetry once on first failure; never retry locally (only CI). Tests retried >1× are flake candidates - triage them.
Trace / videoCapture on-first-retry (off for green runs to save storage). Playwright trace: 'on-first-retry' is the default; Cypress + cypress-video-trim similar.
ReportingJUnit XML output for the CI's test-result panel; Allure for human reporting; both via plugin.
SecretsLoad from CI secret store (GitHub Actions Secrets, GitLab CI Variables); never commit.
Environment matrixOne job per (framework, browser, environment) cell; do not mix in one job.

Step 6 - When to defer the decision

The skill recommends deferring framework choice when:

  • The product surface is too new - no stable URL / API to test against.
  • The team's required languages aren't decided yet (hiring in progress).
  • The team is debating monolith vs micro-frontend; the framework choice depends on the product architecture.

In these cases, the right output is an explicit deferral note: "no decision today; revisit when (a)/(b)/(c) resolves."

Step 7 - Commercial vendors (procurement axis)

When the decision is a commercial tool - a test-management platform (TestRail / Qase / Xray / Zephyr), a no-code automation vendor (mabl / Testim / Functionize), a visual-regression service (Applitools / Percy / Chromatic), or an AI copilot tier - the axes change: contract, lock-in, and exit cost matter as much as capability. Run the seven-axis vendor evaluation (capability fit, cost model, integration depth, lock-in risk, exit cost, contractual posture, customer-reference data) in references/vendor-evaluation.md. It emits an evidence pack with a weighted score per the team's NFR order and deliberately refuses to pick the winner - the team owns the procurement choice.

Step 8 - Record the decision

Whichever path produced the choice, write it down as a decision record so a later reader can see the signal, the rejected alternative, and what would reverse it. The ADR-based format - observed project signal, exactly one primary recommendation, a why-not clause, one read-next link, and mandatory flip conditions - is in references/decision-record-format.md.

Anti-patterns

Anti-patternWhy it failsFix
Picking the framework before the NFRs are scoredChoice driven by hype, not fit; high migration cost when the wrong framework can't deliver.Step 1 - score the NFRs first.
Standardising on one framework across every test layerDifferent layers need different tools (Playwright for E2E ≠ k6 for perf ≠ Pact for contract).Pick per layer; the stack is multiple frameworks.
Picking Selenium for greenfield in 2026Manual async-wait is the dominant flake category at 45% (Luo et al. 2014 (opens in new window)), and Selenium does not auto-wait.Use Playwright for greenfield; reserve Selenium for legacy maintenance.
Cross-language teams picking a single-language frameworkEngineers can't contribute; suite becomes one person's domain.Either pick a multi-language framework (Playwright / Selenium) or commit to retraining.
Adopting a framework because a contractor used itContractor leaves; team can't maintain.Hire-ability is an NFR.
Skipping the directory-layout conventionEvery newcomer authoring tests in a different shape; review burden grows.Step 4 - pick a canonical layout up front, even if you deviate later.
Treating this skill as "framework recommender" rather than "decision support"The skill recommends; the team decides. Automating the decision strips accountability.The output of this skill is a documented choice, not an automatic install.

Limitations

  • Coverage is the web / mobile / API / perf canonical set. Specialised layers (desktop apps via Spectron / Tauri, embedded devices, hardware-in-loop) are out of scope.
  • Tradeoff matrix is point-in-time (checked October 2026). Frameworks change quickly; re-check each row against the framework's docs before a greenfield decision.
  • Tradeoffs are illustrative. A specific product can have constraints that flip the recommendation - e.g., a Salesforce internal app where the legacy stack is JVM-only would override the "Playwright by default" guidance.
  • Hire-ability is geography-dependent. "Hire-ability" assumes US / EU markets in the matrix; for other markets, the team's local pool is the authority.
  • No automated framework scaffolding. Intentional: the output is a recorded decision. Use the per-framework skill after it.
  • Closed-platform comparison is a different axis. Mabl / Testim / Functionize / TestSigma are no-code platforms; they compete with this entire category, not with one framework. The "code vs no-code" strategic call stays with the team; once the team is comparing commercial vendors, use the procurement workflow in references/vendor-evaluation.md.

Hand-off targets

  • After the framework is chosen, configure it → per-framework skill (playwright-testing, cypress-testing, selenium-testing, webdriverio-testing).
  • Record the choice → references/decision-record-format.md.
  • Commercial-vendor procurement → references/vendor-evaluation.md.
  • Tune the layer balance after the framework is in place → test-pyramid-balancer.
  • Convert Selenium suite → Playwright → migration is a project, not an agent; this skill provides the decision input, the per-framework skills provide the target configuration.

References

  • Playwright official documentation - locator hierarchy, trace viewer, parallel sharding: https://playwright.dev/
  • Cypress documentation: https://docs.cypress.io/
  • WebdriverIO documentation: https://webdriver.io/
  • Martin Fowler - Page Object pattern (canonical definition): https://martinfowler.com/bliki/PageObject.html
  • Luo et al., "An Empirical Analysis of Flaky Tests" (FSE 2014) - async-wait is the largest flake category (45%), the class Selenium leaves to manual synchronization: https://mir.cs.illinois.edu/marinov/publications/LuoETAL14FlakyTestsAnalysis.pdf
  • Playwright actionability docs - built-in auto-waiting and auto-retrying assertions that remove a major Selenium flake source: https://playwright.dev/docs/actionability
  • Playwright Android (experimental): https://playwright.dev/docs/api/class-android
  • Cypress experiments (experimentalWebKitSupport): https://docs.cypress.io/app/references/experiments
  • Puppeteer supported browsers: https://pptr.dev/supported-browsers
  • references/other-layers.md - mobile, API / contract, and performance matrices with their sources.
  • ISTQB glossary - test automation framework: https://glossary.istqb.org/en_US/term/test-automation-framework
  • ISTQB glossary - keyword-driven testing (relevant to Karate / Postman DSL choice): https://glossary.istqb.org/en_US/term/keyword-driven-testing
  • ISO/IEC 25010 - quality characteristics (used in Step 1 NFR scoring): https://en.wikipedia.org/wiki/ISO/IEC_25010
  • playwright-testing, cypress-testing, selenium-testing, webdriverio-testing - downstream per-framework configuration skills.
  • test-pyramid-balancer - layer-mix tuning after the framework is in place.
Reference

Tool-selection decision record format

View source (opens in a new tab) (opens in new window)

Deep reference for the framework-choice-advisor SKILL.md. The output contract for writing down a chosen developer tool as a portable decision record: the observed project signal, exactly one primary recommendation, rationale that names the rejected alternative, what to read next, and a mandatory list of the conditions that would flip the choice.

The differentiation axis is output contract versus analysis: the SKILL.md spine (and the vendor evaluation in vendor-evaluation.md (opens in new window)) compare candidate tools on their merits; this format governs only the shape of what gets committed once the weighing is finished. Use both together: the analysis chooses, this record preserves the choice.

Overview

A tool gets chosen once and questioned for years. The choice survives in a chat thread, the reasoning does not, and the next person either re-litigates it from scratch or inherits it as folklore.

This format is a narrowed Architecture Decision Record. It keeps the ADR skeleton and tightens three things generic ADR templates leave open - how many tools may be recommended (Rule 1), what counts as admissible evidence for the context (Rule 2), and whether the reversal conditions are optional (Rule 3). It is tool-agnostic: the same record shape works for a test framework, a package manager, a linter, a migration tool, a CI runner, or a logging library.

What this record is and is not

This recordNot this record
The written artefact produced after a tool is pickedThe comparison that picks it
One decision, one tool, one fileA matrix scoring every candidate on every axis
Evidence taken from the target projectEvidence taken from vendor feature pages
Expires when its own stated conditions occurExpires when someone happens to notice it is stale

ADR provenance and field mapping

An ADR is "a document that captures an important architecture decision made along with its context and consequences" (joelparkerhenderson/architecture-decision-record (opens in new window)), and the canonical five-part shape is Title, Context, Decision, Status, Consequences (Nygard, Documenting Architecture Decisions (opens in new window)). A tool choice is a decision of exactly that kind: it "addresses a functional or non-functional requirement that is architecturally significant" (adr.github.io (opens in new window)).

ADR field (Nygard (opens in new window))Field here
TitleTitle
StatusStatus
Context ("the forces at play")Signal, restricted to observed project evidence
DecisionDecision, restricted to one tool
Consequences ("all consequences ... not just the positive ones")Rationale (the why-not clause) plus Flip conditions

The split of Consequences into two fields is deliberate. Nygard requires that "all consequences should be listed here, not just the 'positive' ones" (source (opens in new window)), and in practice tool records collapse to a list of benefits unless the negative half has its own heading with its own required content.

The record fields

Eight fields. Seven are always required; the eighth appears only under the co-equal rule below.

FieldRequiredContents
TitleyesShort noun phrase naming the decision, prefixed with a sequence number. Nygard specifies "short noun phrases" for ADR titles (source (opens in new window)).
StatusyesOne of proposed, accepted, or superseded by a named later record. Per Nygard, "a decision may be 'proposed' if the project stakeholders haven't agreed with it yet, or 'accepted' once it is agreed" (source (opens in new window)).
SignalyesThe observed project evidence, quoted: file path plus the line or block that drove the detection. This is the ADR Context field, restricted to observable forces.
DecisionyesExactly one tool, written in active voice. Nygard's Decision field uses "full sentences, with active voice. 'We will ...'" (source (opens in new window)).
RationaleyesTwo clauses minimum: why the chosen tool fits the signal, and why not the strongest alternative that was considered.
Read nextyesThe one document the implementer opens first: the tool's setup guide, config reference, or migration guide. One link, not a reading list.
Flip conditionsyesThe specific future observations that would reopen this decision.
Secondary fallbackonly for genuine co-equalsOne alternative, with the constraint that would select it instead.

Rule 1: exactly one primary recommendation

A record names one tool in the Decision field. Two tools in that field is not a decision, it is the reader's decision deferred.

This follows the ADR specificity convention: a well-written record "should be about one AD, not multiple ADs" (joelparkerhenderson/architecture-decision-record (opens in new window)).

The one exception is a genuine co-equal: two candidates that the available signal cannot separate, where the tiebreak depends on a constraint nobody has stated yet (team size, an unwritten hiring plan, a language the team may or may not adopt). Then, and only then:

  1. Put the stronger-by-default candidate in Decision.
  2. Put the other in Secondary fallback.
  3. State the constraint that flips between them, in one line.

A Secondary fallback entry without that constraint line is a tiebreak handed back to the reader. Delete it or complete it.

Never list a third option. If three candidates look co-equal, the signal was too weak to write a record at all, and Rule 2 applies instead.

Rule 2: every record rests on an observed signal

A recommendation is never inferred from a README, a project description, a folder name, or what the team says the stack is. The Signal field must quote one of:

Admissible signalExamples
A manifestpackage.json, pom.xml, pyproject.toml, *.csproj, Cargo.toml, go.mod
A lockfilepackage-lock.json, pnpm-lock.yaml, poetry.lock, Gemfile.lock
A tool config fileplaywright.config.ts, .eslintrc.json, tsconfig.json, Dockerfile, a CI workflow file
An existing test or source directory with real contentstests/e2e/, src/androidTest/, migrations/

Inadmissible on its own: a README paragraph, a wiki page, a folder named after a framework with nothing in it, a compiled artefact, or a verbal claim about the stack. Each of these describes intent rather than state, and intent and state diverge constantly.

This is the ADR Context field taken literally. Context "describes the forces at play" in neutral language (Nygard (opens in new window)); a force that cannot be quoted from the repository is not yet a force, it is a plan. Plans belong in Flip conditions, not in Signal.

When no admissible signal exists, do not write a record. Say so, name the file that would resolve it, and stop. A record built on a guess is worse than no record: it launders the guess into a citable decision.

Rule 3: the flip conditions field is mandatory

Every record declares the observations that would reopen it. This is what keeps a decision log from silently rotting into a set of unchallengeable defaults.

Good flip conditions are observable and specific:

  • The project adds a second target platform.
  • The service drops the ORM the chosen tool is built around.
  • The suite passes 20 minutes of wall-clock time in CI.
  • A non-JVM team takes ownership of this module.

Bad flip conditions are unfalsifiable: "if requirements change", "if the tool stops meeting our needs", "on annual review".

If the author genuinely cannot foresee a reversal condition, the field says so explicitly and gives the reason. An empty field reads as an oversight; a stated "no foreseeable condition, because the choice is forced by the runtime" reads as a claim a later reader can attack.

Flip conditions are also the confirmation hook. MADR carries an optional Confirmation section describing how compliance with the decision gets verified (MADR template (opens in new window)); in this format the flip conditions are what a periodic review actually checks.

Rule 4: supersede, never edit

When a flip condition fires, the old record is not rewritten. Nygard: "If a decision is reversed, we will keep the old one around, but mark it as superseded" (source (opens in new window)). The same immutability guidance appears in the widely used ADR collection: when an earlier decision becomes invalid, "a new ADR should be created" rather than modifying the original (source (opens in new window)).

So: write a new record, set the old one's Status to superseded with a pointer, and let the pair show the reader both the original reasoning and what changed. The set of records accumulated this way is the project's decision log (adr.github.io (opens in new window)).

Template

# <NNNN> - <tool category> for <component>

**Status:** <proposed | accepted YYYY-MM-DD | superseded by NNNN>

**Signal:**
<file path>:<line>
> <quoted excerpt that drove the detection>

**Decision:** We will use <exactly one tool>.

**Rationale:**
- Fits because: <one line tying the tool to the quoted signal>
- Not <strongest alternative>, because: <one line>

**Secondary fallback:** <tool | none>
<if present: the one constraint that would select it instead>

**Read next:** <the specific doc, guide, or config reference to open first>

**Flip conditions:**
- <observable event that reopens this decision>
- <observable event that reopens this decision>

Keep it to one screen. A tool record that runs to three pages has absorbed the comparison it was supposed to summarize.

Worked example

A billing service needs a database migration tool. The analysis has already happened; this is the record it produces.

# 0007 - Database migration tool for the billing service

**Status:** accepted 2026-07-19

**Signal:**
services/billing/pyproject.toml:14
> django = "^5.2"

services/billing/billing/migrations/0001_initial.py exists and is tracked.

**Decision:** We will use Django's built-in migrations.

**Rationale:**
- Fits because: the service's schema is already defined as Django ORM models,
  and Django ships a migrations system that propagates model changes into the
  database schema via `makemigrations` and `migrate`, storing migration files
  in a `migrations` directory inside each app
  (https://docs.djangoproject.com/en/5.2/topics/migrations/). The tracked
  `0001_initial.py` shows that history has already started here.
- Not Alembic, because: Alembic is "a lightweight database migration tool for
  usage with the SQLAlchemy Database Toolkit for Python"
  (https://alembic.sqlalchemy.org/en/latest/). Adopting it would mean
  maintaining a second schema definition alongside the Django models, and
  abandoning the migration history already committed.

**Secondary fallback:** none. The two candidates are not co-equal: the ORM in
the signal selects one of them outright.

**Read next:** the Django migrations topic guide
(https://docs.djangoproject.com/en/5.2/topics/migrations/), specifically the
sections on dependencies between app migrations and on squashing.

**Flip conditions:**
- The billing service moves off the Django ORM to SQLAlchemy.
- A second, non-Django service needs to share this migration history.
- The team adopts a database engine with no supported Django backend.

Note what the record does not contain: no scoring table, no third candidate, no paragraph about either tool's community size. Everything in it is either quoted from the project or cited to a document the reader can open.

Review checklist

Reject a draft record that fails any line:

  • [ ] Signal quotes a manifest, lockfile, config file, or a non-empty test or source directory, with a path.
  • [ ] Decision names exactly one tool.
  • [ ] Rationale contains an explicit why-not clause naming a real alternative.
  • [ ] Every factual claim in Rationale is traceable to the signal or to a linked document.
  • [ ] Read next points at one specific document, not a reading list.
  • [ ] Flip conditions lists at least one observable event, or states why none is foreseeable.
  • [ ] Secondary fallback, if present, carries the constraint that selects it.
  • [ ] Status is set, and any reversal is a new record rather than an edit.
  • [ ] The record fits on one screen.

Anti-patterns

Anti-patternWhy it failsFix
Two tools in the Decision fieldThe reader still has to choose, so nothing was decided; it also breaks the one-decision-per-record convention (source (opens in new window))Promote one, demote the other to Secondary fallback with its selecting constraint
Signal reads "this is a Node project"A folder shape is a description, not evidence, and it goes stale silentlyQuote the manifest or lockfile line that proves it
Rationale lists only advantages of the winnerConsequences must include the non-positive ones (source (opens in new window))Add the why-not clause and the cost the team is accepting
Flip conditions omitted or written as "revisit annually"The record can never expire on evidence, so a stale choice outlives its reasonsName observable events tied to the signal
The record is edited in place when the tool changesThe original reasoning is destroyed, so nobody can tell whether the reversal was justifiedNew record, old one marked superseded (source (opens in new window))
Rationale paraphrases the tool's marketing pageUnfalsifiable, and it will read as true forever regardless of the projectTie each clause to the quoted signal or to a linked technical document

Limitations

  • This format records a decision. It does not evaluate candidates, score them, or price them. The SKILL.md spine and vendor-evaluation.md (opens in new window) are the comparison side.
  • The signal rule cannot see intent that exists only in people's heads. A planned migration off the current stack is invisible to a manifest, so it belongs in Flip conditions and must be volunteered by the team.
  • Status vocabularies are not standardized. Proposed, accepted, and superseded are the minimum from the original ADR format; longer lifecycles such as initiating, researching, evaluating, implementing, maintaining, and sunsetting are also in use, and teams customize status values to fit their workflow (source (opens in new window)). Pick one vocabulary per decision log and keep it stable.
  • A record is only as durable as the log it lives in. Records scattered across ticket comments do not form a decision log (adr.github.io (opens in new window)); keep them in one directory in the repository they describe.

Deep reference for the framework-choice-advisor SKILL.md, Step 4. After the team has chosen a stack, these are the canonical directory layouts the per-framework skill assumes. Layouts are conventions, not mandates - every project has reasons to deviate, but the canonical layout is the starting point a newcomer can read.

Playwright + Jest (TypeScript) - the 2026 default for web E2E

tests/
├── e2e/
│   ├── auth/
│   │   ├── login.spec.ts
│   │   └── login.fixture.ts
│   ├── cart/
│   │   ├── add-item.spec.ts
│   │   └── checkout.spec.ts
│   └── pages/                  # Page Objects (per Martin Fowler's pattern)
│       ├── LoginPage.ts
│       ├── CartPage.ts
│       └── CheckoutPage.ts
├── helpers/
│   ├── api-client.ts            # HTTP client for setup / teardown
│   ├── test-data.ts             # Fixtures and seeds
│   └── selectors.ts             # Shared accessibility-first locators
├── fixtures/                    # Static test data
├── playwright.config.ts
├── tsconfig.json
└── package.json

Conventions:

  • One *.spec.ts per feature flow; one Page Object per page or major component.
  • Fixtures scoped to describe blocks; global fixtures are an anti-pattern (see test-code-conventions §6).
  • Page Objects follow Martin Fowler's pattern (opens in new window): they wrap a page with an application-specific API and do not make assertions.

Cypress + Mocha (TypeScript)

cypress/
├── e2e/
│   ├── auth/login.cy.ts
│   └── cart/checkout.cy.ts
├── support/
│   ├── commands.ts              # Custom Cypress commands
│   ├── pages/                   # Page Objects (Cypress idiom: command-based, not class-based)
│   └── e2e.ts
├── fixtures/
├── cypress.config.ts
└── package.json

Cypress idiom prefers custom commands over class-based POMs; the directory layout reflects that.

Selenium / WebdriverIO (TypeScript or Java)

test/
├── specs/
│   ├── auth/login.spec.ts
│   └── cart/checkout.spec.ts
├── pageobjects/
│   ├── login.page.ts
│   └── cart.page.ts
├── helpers/
├── wdio.conf.ts
└── package.json

WDIO's runner ergonomics improve on raw Selenium; the layout is conventional.

Reference

Tradeoff matrices for other test layers

View source (opens in a new tab) (opens in new window)

Deep reference for framework-choice-advisor Step 3. Score the Step 1 axes first; these tables only compare options inside one layer. Pick per layer: the stack is several tools, not one.

Mobile native

FrameworkPlatformLanguageNotes
EspressoAndroid nativeKotlin / JavaGoogle's first-party. In-process, fast, deterministic.
XCUITestiOS nativeSwift / Obj-CApple's first-party; UI tests drive the app from a separate test-runner process.
AppiumiOS + Android (and others)All majorCross-platform unifier; trades depth for breadth. WebDriver-based - same flake patterns as Selenium.
DetoxReact NativeJS / TSRN-specialist; grey-box testing.

Decision: if the team is single-platform native (iOS only or Android only), use the first-party framework. Cross-platform → Appium, accept the WebDriver flake-tax. React-Native specifically → Detox.

API / contract

FrameworkScopeLanguageNotes
RestAssuredREST API integration testsJava / KotlinThe JVM-default; mature, deeply integrated with JUnit / TestNG.
KarateREST + SOAP + GraphQL + gRPCKarate DSL (Cucumber-like)DSL-first; lowers barrier for non-Java testers.
schemathesisOpenAPI / GraphQL property-based fuzzingPythonGenerative; complements example-based tests.
PactConsumer-driven contract testsJS / JVM / Python / Go / Ruby / .NETDifferent category - contract, not integration. See pact-contract-testing.
Postman / NewmanCollection-driven API testsPostman DSLUI-driven authoring; not code-first. Often used by non-engineers.

Performance

FrameworkScopeLanguageNotes
k6Load + perf, code-firstJS (with TS support)Grafana's; lowest barrier for engineers, excellent CI integration.
LocustLoad + perf, code-firstPythonOpen-source; user-class-based modelling.
JMeterLoad + perf, GUI-firstXML configMature, ecosystem-heavy; GUI-driven authoring is the trade-off.
GatlingLoad + perf, code-firstScala / Java / KotlinHigh-throughput; JVM stack.

Sources

  • Espresso: https://developer.android.com/training/testing/espresso
  • XCUITest (XCTest UI testing): https://developer.apple.com/documentation/xctest/user_interface_tests
  • Appium: https://appium.io/docs/en/latest/
  • Detox: https://wix.github.io/Detox/
  • REST Assured: https://rest-assured.io/
  • Karate: https://github.com/karatelabs/karate
  • Schemathesis: https://schemathesis.readthedocs.io/
  • Pact: https://docs.pact.io/
  • Newman: https://github.com/postmanlabs/newman
  • k6: https://grafana.com/docs/k6/latest/
  • Locust: https://docs.locust.io/
  • Apache JMeter: https://jmeter.apache.org/
  • Gatling: https://docs.gatling.io/

Deep reference for the framework-choice-advisor SKILL.md. The commercial-procurement counterpart of the open-source framework decision: a side-by-side vendor evaluation matrix for QA tools - test-management platforms (TestRail / Qase / Xray / Zephyr / TestCollab), no-code platforms (mabl / Testim / Functionize / TestSigma / Reflect), visual regression services (Applitools / Percy / Chromatic), and commercial AI copilots - scoring each on capability fit, cost model, integration depth, vendor lock-in risk, exit cost, contractual posture, and customer-reference data.

Scoped to commercial procurement - contract, lock-in, and exit-cost axes - not to choosing an open-source code-first framework on architectural fit (that is the SKILL.md spine). The evaluation refuses to recommend a winner; the team owns the procurement choice.

Overview

QA managers procure commercial tools every 12 - 24 months: a new test-management platform, a no-code automation vendor, a visual-regression service, an AI copilot tier. The Capgemini World Quality Report 2025-26 (opens in new window) identifies integration friction (37% of teams) as the dominant blocker for AI-in-testing adoption - the proximate failure mode is teams that adopted a vendor without scoring integration cost in advance. This workflow produces the structured side-by-side comparison the manager carries into the procurement decision, with every score citing its source.

This is decision-support, not recommendation. The output is the evidence pack: capability matrix, cost-model breakdown, integration / lock-in / exit-cost analysis, and contractual posture per vendor. The manager (or a procurement committee) makes the call against the team's NFR priorities.

When to run one

  • A 12 - 24 month procurement cycle is approaching for a commercial QA tool.
  • The team is evaluating a switch from one vendor to another (mid-contract renewal, post-acquisition consolidation).
  • A new tool category is being adopted (the team has no incumbent - e.g., first visual-regression service).
  • A vendor is being added to a multi-vendor stack and the team wants to ensure no overlap with incumbents.
  • An RFP / RFI / procurement-committee process needs a structured evaluation artifact.

Do not use this workflow when:

  • The decision is between open-source code-first frameworks (Playwright vs Cypress vs Selenium) - use the SKILL.md spine. Different axis entirely (architecture, not procurement; no contract, lock-in, or exit-cost dimensions).
  • Only one vendor is being evaluated - comparison requires ≥2 candidates. (For a single-vendor go/no-go, use the team's standard procurement checklist; this workflow needs comparison anchors.)
  • The team has no defined NFR priorities - the matrix cannot score "capability fit" without knowing what the team needs the tool to do.

Step 1 - Capture the inputs

Required:

InputNotes
≥2 vendor candidatesThe vendors being compared. Halts on 1; recommends ≥3 for a healthy comparison (avoids the two-choice false-binary).
Team profileTeam size, existing stack (CI, test framework, observability, tracker), seat / volume profile, geography, regulated-industry flag (if any).
NFR prioritiesOrdered list of what the team needs most: capability fit / cost / integration / lock-in / contract / support / data-residency. The order matters - the matrix's weighted score depends on it.
Time horizon12 / 24 / 36 month decision window. Drives the lock-in and exit-cost scoring (longer horizon = more weight on lock-in).
Per-vendor dataPricing page, feature page, integration docs, customer-reference reviews (Gartner Peer Insights, G2, Capterra), vendor-published case studies (tagged as vendor-data).

Halt with INSUFFICIENT_INPUT if any required input is missing.

Step 2 - Score on the seven procurement axes

Score each vendor on each axis, with every score citing its source.

Axis A1 - Capability fit

How well does the vendor's feature set match the team's documented NFR priorities?

Sub-axisScoring rubric (per-vendor)
Core feature coverage% of team's required features present (cite the team's requirement list)
Advanced / aspirational featuresFeatures the team doesn't need today but might in 2 years
Documented limitsPer-account / per-test / per-user caps that may bind

Axis A2 - Cost model

How is the vendor priced and what does it cost at the team's scale?

Sub-axisScoring rubric
Pricing modelPer-seat / per-test / per-execution / flat licence / hybrid
Cost at current team sizeYear-1 cost, cited to vendor pricing page (vendor-data)
Cost at projected team sizeYear-2 and year-3 projections; the team's growth plan drives this
Hidden costsAdd-ons (parallel execution, premium support, SSO, audit log, on-prem option)
Volume-discount commitmentsAnnual commits, multi-year discounts (cite to vendor sales channel)

Axis A3 - Integration depth with existing stack

Per the Capgemini WQR 2025-26 (opens in new window) finding (37% blocked by integration friction), this axis is often under-weighted in procurement. Weight it explicitly.

Sub-axisScoring rubric
CI integrationNative plugin? REST API? Webhook? CLI? Cite the integration doc URL per vendor
Tracker integrationJira / Linear / GitHub Issues / Azure DevOps
Observability integrationDatadog / Grafana / New Relic / Sentry
Test-framework bindingPlaywright / Cypress / Selenium / per-language
SSO / IAMSAML / OIDC / SCIM provisioning
Reverse data flowCan the team export raw test results? In what format?

For each integration, score: native (1.0) / API-buildable (0.7) / community-plugin (0.5) / not available (0.0). The per-vendor result-sync baselines (test-management-sync, test-management-sync, test-management-sync, currents-integration in the qa-test-reporting plugin) feed this axis.

Axis A4 - Vendor lock-in risk

The cost of being unable to leave.

Sub-axisScoring rubric
Proprietary data formatsAre tests in a portable format (Gherkin / standard JSON / open spec) or vendor-proprietary DSL?
Test artifact portabilityCan the team export tests, results, history? In what format? Vendor-published export tools count toward portability.
Data residencyWhere is data stored? Can the team request export and deletion?
Migration pathAre there documented or community-tested migration paths off this vendor (mabl -> Playwright, Testim -> Cypress)?

Axis A5 - Exit cost

The team has decided to leave in 24 months - what does it cost?

Sub-axisScoring rubric
Test re-authoring effortIf tests are vendor-DSL-bound, the migration is "rewrite from scratch." If tests are portable (Gherkin, standard fixtures), migration is mostly mechanical.
History portabilityCan the team take its test-result history? Defect-history correlation depends on this.
Re-trainingHow long to retrain the team on the new vendor?
Data egress feesSome vendors charge for bulk data export. Cite the contract clause if applicable.
Contract early-termination costMulti-year commits often have early-termination fees.

Axis A6 - Contractual posture

Procurement / legal / security review needs structured data.

Sub-axisScoring rubric
SLA tierUptime guarantee, support response time, escalation paths
Support tierEmail-only / chat / phone / dedicated CSM
Security audit availabilitySOC 2 Type II report? ISO 27001? Penetration-test results?
On-prem / private-cloud optionFor regulated industries; cite the vendor's deployment options page
Data-processing agreementGDPR-compliant DPA available? HIPAA BAA?
Sub-processor disclosureVendor's sub-processor list (the regulated-industry view)

Axis A7 - Customer-reference data

Independent (not vendor-published) signal.

Sub-axisScoring rubric
Gartner Peer InsightsRating, review density, recency. Cite the category report URL.
G2 / CapterraRating, review density. Flag if reviews are sparse or stale (<10 reviews in last 12 months).
Practitioner blog / conference signalHas the vendor been written about by recognised practitioners (Lisa Crispin, James Whittaker, etc.)? Cite the source.
Reddit / r/QualityAssurance / Hacker NewsAnecdotal community signal - tag as such. Don't weight equally with surveyed data.

Step 3 - Emit the comparison matrix

Emit a single markdown document: the vendors compared, the team profile, the ordered NFR priorities, a per-axis matrix (A1-A7), a weighted-score table using the NFR order as weights, an explicit "What this workflow did NOT do" disclaimer (does not pick the winner, negotiate, validate vendor claims, or replace a reference call), and an evidence appendix tracing every cell to a source.

Worked example (values illustrative)

# Vendor evaluation - `<category>` - `<team>` - 2026-07

## Vendors compared

| Code | Vendor | Pricing page | Cited integration doc |
|---|---|---|---|
| V1 | TestRail (Gurock / Idera) | https://www.testrail.com/pricing/ | https://support.testrail.com/hc/en-us/articles/7077873061908 |
| V2 | Qase | https://www.qase.io/pricing/ | https://docs.qase.io/en/articles/6417206-github |
| V3 | Xray (Xpand IT, for Jira) | https://marketplace.atlassian.com/apps/1211769/xray-test-management-for-jira | https://docs.getxray.app/display/XRAYCLOUD/REST+API |

## Team profile

- Size: 12 QA engineers
- Stack: Playwright + Jest, GitHub Actions, Linear (tracker), Datadog (observability)
- Geography: distributed US + EU; data-residency: EU required
- Regulated-industry: no
- Time horizon: 24 months

## NFR priorities (manager-supplied, ordered)

1. Integration with Linear + GitHub Actions
2. Cost at year-2 (team will grow to 18 engineers)
3. Data residency (EU)
4. Test-history portability (exit-cost matters; 24-month horizon)
5. SSO (SAML / OIDC)
6. Capability fit
7. Customer-reference depth

## Per-axis matrix

### A1 - Capability fit

| Vendor | Score (0-1.0) | Strengths | Gaps |
|---|---|---|---|
| TestRail | 0.85 | Mature test-case management, custom fields, bulk import / export | API rate limits documented at 180/min - may bind at scale |
| Qase | 0.80 | Modern UI, AI-assisted case authoring | Smaller plugin ecosystem |
| Xray | 0.95 | Deep Jira integration, BDD-native | Heavyweight Jira dependency the team doesn't have |

### A2 - Cost model

| Vendor | Year-1 (12 eng) | Year-2 (18 eng) | Hidden costs |
|---|---|---|---|
| TestRail | $5,328 (12 × $37/seat/mo Professional × 12) | $7,992 | SSO, automated backups, priority support are Enterprise-tier only |
| Qase | $4,320 (12 × $30/seat/mo Business × 12) | $6,480 | None at this tier; SSO included from Business plan |
| Xray | Quote from the Marketplace listing - Xray licenses by total Jira user tier, not by tester seat | Same tier rule at 18 engineers; re-quote if the Jira tier changes | Requires Jira Software seats if not already licensed |

### A3 - Integration depth

| Vendor | CI (GitHub Actions) | Tracker (Linear) | Observability (Datadog) | Test-framework (Playwright) | SSO |
|---|---|---|---|---|---|
| TestRail | Native (1.0) | API-buildable (0.7) | Community plugin (0.5) | API + `testrail-cli` (0.9) | SAML / OIDC (1.0) |
| Qase | Native action (1.0) | Native (1.0) | Webhook (0.7) | Native @qase/playwright (1.0) | SAML (Business+) (0.8) |
| Xray | API only (0.7) | API-buildable (0.7) | None native (0.0) | xray-junit-extensions (0.9) | SAML / OIDC (1.0) |

### A4 - Vendor lock-in risk

| Vendor | Format | Export | Lock-in score |
|---|---|---|---|
| TestRail | Proprietary case format; bulk CSV export | Documented CSV / XML export, JSON via API | Moderate (0.6) - export possible, but tests need re-authoring on migration |
| Qase | YAML / JSON case format; native import / export | First-class export to JSON / YAML | Low (0.85) - portable artifacts |
| Xray | BDD-native (Gherkin), JUnit / Cucumber export | Tied to Jira issue model; export possible but harder to disentangle | Moderate-high (0.5) - Jira coupling is the lock-in axis |

### A5 - Exit cost (24-month migration scenario)

| Vendor | Test re-authoring | History portability | Total exit cost (hand-wave) |
|---|---|---|---|
| TestRail | Cases portable as CSV; ~30% needs re-authoring for new tool | History exportable via API | ~3 person-months |
| Qase | YAML / Gherkin cases mostly portable; ~10% re-authoring | Native export | ~1 person-month |
| Xray | Gherkin scenarios portable; Jira-issue history harder to extract | API export; needs custom tooling | ~4 person-months |

### A6 - Contractual posture

| Vendor | SLA | Support | Security | EU residency |
|---|---|---|---|---|
| TestRail | 99.9% (Cloud), no SLA for self-hosted | Email; phone on Enterprise | SOC 2 Type II + ISO 27001 (cite vendor security page) | EU AWS region available on Enterprise |
| Qase | 99.9% on Business+ | Email + chat; CSM on Enterprise | SOC 2 Type II (cite) | EU region available on Business |
| Xray | Bound to Jira's SLA | Email + chat | SOC 2 Type II inherited from Xpand IT | Tied to Jira region |

### A7 - Customer-reference data

| Vendor | Gartner Peer Insights | G2 (recency / density) | Practitioner-signal |
|---|---|---|---|
| TestRail | 4.4/5 (382 reviews, mostly 2023-25) | 4.2/5, 250+ reviews | Cited in Lisa Crispin's *Agile Testing Condensed* |
| Qase | 4.6/5 (180 reviews, 2024-26) | 4.7/5, 120+ reviews | Featured in TestBash 2025 case studies |
| Xray | 4.4/5 (290 reviews) | 4.4/5, 200+ reviews | Heavy enterprise adoption signal; lighter mid-market |

## Weighted score per NFR priorities

| Axis | Weight (per team NFR order) | TestRail | Qase | Xray |
|---|---|---|---|---|
| A3 Integration | 0.25 | 0.78 | 0.92 | 0.62 |
| A2 Cost | 0.20 | 0.65 | 0.95 | 0.70 |
| A6 EU residency | 0.15 | 0.80 | 0.90 | 0.70 |
| A5 Exit cost | 0.15 | 0.60 | 0.90 | 0.50 |
| A6 SSO | 0.10 | 1.00 | 0.80 | 1.00 |
| A1 Capability fit | 0.10 | 0.85 | 0.80 | 0.95 |
| A7 Customer-reference | 0.05 | 0.90 | 0.85 | 0.85 |
| **Total** | **1.00** | **0.76** | **0.89** | **0.69** |

## What this workflow did NOT do

- Pick the winner. The matrix and weighted scores are the input to the procurement decision; the team owns the choice. The team may legitimately pick the lower-scored vendor for reasons outside the matrix (existing relationship, hiring-pool, founder preference).
- Negotiate the contract. Once a vendor is picked, contract terms (discount, multi-year commit, SLA tier) are a separate procurement conversation.
- Validate vendor claims. Where the matrix cites vendor-data (pricing, feature lists, case studies), the data is vendor-published and should be re-verified in a sales call before commitment.
- Replace a reference call. Customer references should be called directly, not just scored from public-review averages.

## Evidence appendix

Every cell in the matrix above traces to a source URL or cited document. The full appendix lists every source (per axis × per vendor) so the team can spot-check.

Step 4 - Hand off to procurement / decision committee

The matrix is the input to the decision, not the decision itself. Downstream:

  1. Vendor sales call to re-verify pricing, SLA, and roadmap claims.
  2. Customer reference call to 2 - 3 named customers per finalist.
  3. Security review of SOC 2 / ISO 27001 reports.
  4. Pilot / POC on the top-2 candidates (typically 30 - 60 day evaluation).
  5. Procurement-committee decision with the matrix as the structured evidence pack.

Anti-patterns

Anti-patternWhy it failsFix
Scoring "capability fit" without the team's NFR prioritiesThe matrix favours feature-richest vendor regardless of fitStep 1 NFR priorities are mandatory inputs
Equal-weight matrixTreats integration, cost, and capability as equally important - almost never trueStep 3 weighted score per team's NFR order
Picking the vendor by gut from the matrixPer the Capgemini WQR (opens in new window), integration cost is under-weighted; gut decisions favour capability over integrationThe weighted-score column is the discipline; the team must justify deviations
Skipping A4 / A5 (lock-in / exit cost) because "we'll figure it out later"The dominant cost surfaces at year-2+; skipping these axes optimises for year-1 happinessThese axes are mandatory in the matrix
Treating vendor-data and Gartner / G2 data identicallyVendor-data is marketing; reviewed data is signalA7 explicitly separates them
Using the matrix as the procurement decisionProcurement requires sales / reference / security calls beyond the matrixStep 4 hand-off lists the required downstream actions
Comparing only 2 vendorsTwo-choice procurement is a false binary; the team often missed a third optionStep 1 recommends ≥3 candidates
Skipping the weighted score because "it feels mechanical"Without weighting, the matrix is decorationStep 3 weighted score is required
Auto-recommending the highest-scored vendorThe team's context (existing relationship, hire-ability, contract leverage) is outside the matrix; auto-recommend strips that contextThe "What this workflow did NOT do" block explicitly disclaims the recommendation

Limitations

  • Vendor-data dominates the matrix. Pricing pages, feature lists, and case studies are vendor-published. Flag these explicitly; the team verifies in sales calls.
  • Customer-reference scoring is shallow. Public-review averages (Gartner, G2) are noisy. Real customer calls are the high-signal version; the matrix supplements but doesn't replace.
  • Categories drift fast. A 2026 evaluation is stale by 2028; product features, pricing, and vendor stability all change. Re-run before each renewal cycle.
  • No competitive-dynamics axis. The matrix doesn't account for vendor M&A risk (TestRail's parent Idera, mabl's growth stage, etc.). Add as an A8 if the team's horizon is >24 months.
  • Local market variation. Pricing and contract terms differ by geography; the matrix uses US-list pricing unless the team specifies otherwise.
  • No regulatory-specific guidance. Regulated-industry teams (healthcare, finance, automotive) have additional contractual axes (BAA, HIPAA, FDA Class II evidence) not enumerated per industry. Layer those onto A6.
  • Honest about being a draft. The matrix is the manager's starting point; the procurement committee customises weights, adds axes, and replaces data after the sales / reference / security calls.

References

  • Capgemini World Quality Report 2025-26 - 37% of teams cite integration friction as the dominant AI-in-testing blocker; load-bearing for axis A3: https://www.capgemini.com/insights/research-library/world-quality-report-2025-26/
  • Gartner Peer Insights - AI-augmented software testing category: https://www.gartner.com/reviews/market/ai-augmented-software-testing-tools
  • G2 / Capterra - methodology disclosure for review-density and recency scoring (general SaaS evaluation context; not QA-specific): https://www.g2.com/about
  • ISTQB glossary - test automation framework (the open-source / commercial boundary): https://glossary.istqb.org/en_US/term/test-automation-framework
  • ISO/IEC 25010 - quality model for non-functional requirements (used in A1 capability scoring): https://en.wikipedia.org/wiki/ISO/IEC_25010
  • test-management-sync, test-management-sync, test-management-sync, currents-integration (qa-test-reporting plugin) - per-vendor integration baselines that feed A3.
  • Both 2023 tooling records are stale now the migration has landed
  • Tooling records for checkout and pricing before four contractors land
  • Job ad goes out Monday and it needs to name our test stack
  • Committee meets Thursday and I need one vendor name on the slide
  • One tool for every layer by Q1 - I need the name and the bill
Blog

Related reading