browser-matrix-strategy-reference
Pure-reference for designing and reviewing a browser / OS / device test matrix from traffic data - the T1/T2/T3 tier-membership heuristics (T1 >=5% traffic, T2 1-5% or statutory, T3 <1% with customer demand), the traffic-share sources (own analytics, StatCounter, MDN browser-compat-data), a worked matrix template with tier-change log, the matrix review checklist (staleness, T1 oversize, below-threshold T1 entries, missing real-device coverage), how to justify dropping a legacy browser (IE11, old iOS Safari), and the compatibility budget (tier caps, CI cost formula, published support statement) in references/compatibility-budget.md. Use when designing an initial matrix, capping or publishing a support policy, running a quarterly re-tier review, or making the case to drop a browser. This is the WHAT-to-test strategy reference - to execute the matrix use playwright-testing browser projects (bundled engines), selenium-grid-4-runner (self-hosted), or cloud-grid-e2e (managed grids).
Install with skills.sh (any agent)
npx skills add testland/qa --skill browser-matrix-strategy-referencebrowser-matrix-strategy-reference
Overview
A real browser matrix tiers browsers by traffic share, regulatory requirement, and team budget, then runs each tier at its own cadence - not "test on everything".
This skill is a pure reference consumed by the sibling execution skills: cloud-grid-e2e (BrowserStack / Sauce Labs / LambdaTest), the self-hosted selenium-grid-4-runner, and playwright-testing browser projects (bundled engines; references/browser-matrix.md). Capping the matrix and publishing the support statement is covered in references/compatibility-budget.md.
When to use
How to use
The three-tier model
Cadence per tier; exact traffic thresholds are defined once in the Tier-membership heuristic section below.
| Tier | Cadence | Coverage criterion | Example |
|---|---|---|---|
| T1 - Must pass every PR | Per-PR + main | Highest-traffic, must-not-break combos | Chrome current + Chrome-1 |
| T2 - Pre-release / nightly | Nightly + pre-release | Mid-traffic OR statutory (regulated industries) | Firefox / Safari / Edge current |
| T3 - Quarterly / on-demand | Quarterly | Long-tail with customer demand | Safari iOS 16 / IE11 / niche Android |
Data sources for traffic share
| Source | Use for |
|---|---|
| Your product's own analytics (GA / Plausible / etc.) | Definitive ground truth for your users |
| StatCounter (gs.statcounter.com (opens in new window)) | Global / regional baseline when own data thin |
| MDN browser-compat-data (github.com/mdn/browser-compat-data (opens in new window)) | Per-feature compatibility map - which versions support a given Web API |
| caniuse.com | Same purpose as MDN, easier UI |
| web-platform.org browser support (webstatus.dev (opens in new window)) | Web Platform features cross-browser readiness |
| GitHub Actions browser-default versions (github.com/actions/runner-images (opens in new window)) | What CI runners ship with by default |
Use your own analytics for traffic-share decisions; supplement with StatCounter when slicing geographies you don't have own data for.
Worked example
A hypothetical SaaS B2B web app builds its first matrix from its own analytics:
Result: a 3-combo T1, 5-combo T2, 4-combo T3 matrix with a documented reason per row. The filled-in artifact is references/matrix-template.md.
Tier-membership heuristic
The single authoritative traffic thresholds, referenced by the three-tier model and worked example above. A browser belongs in:
Cost-tier mapping
Where to test each tier:
| Tier | Recommended infrastructure | Reason |
|---|---|---|
| T1 | Bundled engines (Playwright + Chromium / Firefox / WebKit) on CI runner | Free; fast; sufficient for engine-level coverage |
| T2 | Cloud grid (BrowserStack / Sauce / LambdaTest) | Real devices + real OS; manageable cost at nightly cadence |
| T3 | Cloud grid on-demand | Real device + low-frequency; budget-friendly |
| Internal apps | Self-hosted Selenium Grid 4 + tunnels | Data residency or cost-control |
For very high-volume + cost-sensitive teams, self-hosted Selenium Grid 4 for T1 + cloud grid for T2 / T3 is the optimum mix.
Engine vs runner distinction
A T1 entry "Chrome latest" can mean:
For T1 the bundled-engine path is usually sufficient; for T2 / T3, real-browser via cloud grid is the choice when the difference matters (some bugs are real-Chrome-only).
Capping the matrix: the compatibility budget
The tiering above decides WHICH combos to test; the budget decides HOW MANY the team can afford to commit to, and publishes that commitment. references/compatibility-budget.md carries the four-tier commitment model (per-PR / nightly / pre-release / unsupported), the CI cost formula, example budgets per product type, the external "what we support" statement template, and the telemetry-to-tier mapping.
Matrix review checklist
When auditing a committed matrix (quarterly review, or before relying on it for a release), check four failure modes:
| # | Check | Severity | Rule |
|---|---|---|---|
| 1 | Staleness | BLOCK | The matrix must carry a Reviewed: date under 90 days old and a Next review: date, per the quarterly-review convention. Absent or older = STALE. |
| 2 | T1 oversize | WARN | T1 runs on every PR and dominates CI cost (budget reference §3). More than 6 T1 combos with no written cost rationale = review trigger. |
| 3 | Below-threshold T1 entries | BLOCK | T1 requires >=5% own traffic (heuristic above). Any T1 entry below 1% in both own analytics and StatCounter, with no contractual obligation, belongs in T3 or unsupported. Same-engine duplicates (Brave beside Chrome) count as one. |
| 4 | Missing real-device coverage | WARN | If the audience includes mobile and iOS Safari / Chrome on Android sits in T1/T2, a bundled-engine-only CI config is insufficient - bundled WebKit is not Safari; those rows need a cloud-grid or real-device leg. |
Refuse to treat a matrix as PASS while any BLOCK finding stands, and flag own-analytics data older than 90 days as stale telemetry.
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| "Test on all browsers" without tiering | Wastes resources; nothing actually gates the release | Tier explicitly |
| Tier membership never reviewed | Drift: T3 browsers stay in T3 long after traffic dies | Quarterly review |
| Same tests across all tiers | Some tests are environment-specific; running all everywhere is wasteful | Smoke at T1 / T2 / T3; full regression at T1 only |
| Counting Chromium-engine browsers separately (Chrome, Brave, Vivaldi, Opera) | Same engine; one test covers them | Group by engine; test the dominant browser only |
| IE11 in 2026 | Trivial traffic in nearly all contexts | Audit own analytics before committing |
| Mobile and desktop in same tier | Different breakage surfaces | Treat mobile as its own dimension |
| No tier-change log | "Why is Safari 14 in T2?" - nobody remembers | Always log tier changes |
Limitations
References
Compatibility budget - tier definitions and example budgets
View source (opens in new window)Compatibility budget - tier definitions and example budgets
Detailed tier tables for compatibility-budget.md (opens in new window). The tier model below defines the four commitment levels; the per-product-type tables are starting templates to copy and adjust against your own telemetry.
Tier model
| Tier | Definition | CI cadence |
|---|---|---|
| Tier 1 | Must work; failure blocks releases. | Per-PR smoke. |
| Tier 2 | Must work; failure blocks releases on detection. | Nightly full suite. |
| Tier 3 | Should work; broken-here is a known issue. | Pre-release manual / weekly. |
| Unsupported | Explicitly out of scope; bugs closed as "not supported." | None. |
The tier signals engineering investment, not user importance - a configuration with low traffic but contractual obligation may be Tier 1.
Example budget per product type
Modern web app
| Configuration | Tier |
|---|---|
| Chrome (current + 1 prior) | 1 |
| Edge (current) | 1 |
| Safari (current + 1 prior) | 1 |
| Firefox (current) | 2 |
| iOS Safari (current + 1 prior) | 1 |
| Chrome on Android (current) | 1 |
| Firefox Android | 3 |
| Samsung Internet | 3 |
| Internet Explorer | unsupported |
| < Chrome 100 | unsupported |
Internal SaaS (controlled audience)
| Configuration | Tier |
|---|---|
| Chrome (latest stable) | 1 |
| Chrome (current - 1 stable) | 1 |
| Edge (latest) | 2 |
| Firefox | 3 |
| Safari | 3 |
| All others | unsupported |
Open-source library
| Configuration | Tier |
|---|---|
| Node 18, 20, 22 on Linux | 1 |
| Node 18, 20, 22 on macOS | 2 |
| Node 18, 20, 22 on Windows | 2 |
| Bun (current) | 3 |
| Deno (current) | 3 |
| Older Node EOL versions | unsupported |
Mobile native app
| Configuration | Tier |
|---|---|
| iOS 17, 16 (current + 1 prior) | 1 |
| iOS 15 | 2 |
| iOS 14 | 3 |
| < iOS 14 | unsupported |
| Android 14, 13 | 1 |
| Android 12 | 2 |
| Android 11 | 3 |
| < Android 11 | unsupported |
The compatibility budget
View source (opens in new window)The compatibility budget
Budget reference for browser-matrix-strategy-reference: decide how large a compatibility matrix the team can afford and publish that commitment - tier-1 (must work; per-PR) vs tier-2 (must work; nightly) vs tier-3 (should work; pre-release) vs unsupported, with example budgets per product type, the matrix-size cost / coverage trade-off, and "what we support" external templates. The main skill's tier heuristics pick WHICH browsers belong in each tier; this reference caps HOW MANY and publishes the policy.
Overview
Without an explicit budget, "compatibility" creeps:
A compatibility budget is a deliberate decision: which configurations the team commits to support, at what tier, with what consequences for unsupported configurations.
When to use
How to use
§1 - Tier model
Four tiers signal engineering investment, not user importance - a low-traffic configuration under contractual obligation may still be Tier 1:
Full definitions and CI cadence: compatibility-budget-tiers.md (opens in new window).
§2 - Example budget per product type
Worked starting budgets for a modern web app, an internal SaaS, an open-source library, and a mobile native app are tabulated in compatibility-budget-tiers.md (opens in new window). Copy the closest template and adjust each combo's tier against your telemetry (§6).
§3 - Cost / coverage trade-off
The budget directly affects CI cost:
CI cost = N_tier1 × per_PR_cost + N_tier2 × nightly_cost + N_tier3 × manual_review_costFor a typical web product (5 Tier 1 configs + 2 Tier 2 + 4 Tier 3):
A team's CI budget caps the total; the budget shapes the matrix.
§4 - "What we support" template
Publish externally (docs, marketing, README):
# Browser / OS support
We officially support:
## Tier 1 (per-release tested)
- Chrome (current + 1 prior major version)
- Edge (current)
- Safari (current + 1 prior major version)
- iOS Safari (current + 1 prior major version)
- Chrome on Android (current)
## Tier 2 (nightly tested; bugs fixed within 1 release)
- Firefox (current)
## Tier 3 (best-effort; bugs may take longer to fix)
- Firefox on Android
- Samsung Internet
## Unsupported
- Internet Explorer 11 - last supported v1.4.0 (EOL 2025-12-31).
- Chrome < version 100 - security vulnerabilities; not supported.
- Older mobile OSes - see mobile platform support table.
If you experience an issue on a Tier 3 or unsupported configuration,
please open an issue but understand the priority is lower.The external statement sets user expectations; the internal tiers guide engineering.
§5 - Quarterly review
Browsers / OS / runtime versions evolve; the budget needs review:
| Trigger | Action |
|---|---|
| New major OS / browser release | Add to Tier 1; re-evaluate older. |
| Tier 3 configuration generates >5 issues | Promote to Tier 2 OR retire to unsupported. |
| Tier 1 configuration generates 0 issues / quarter | Consider demoting (controversial - get team consensus). |
| Vendor announces EOL | Move to unsupported on EOL date. |
§6 - User-agent telemetry → budget
If the team has analytics on browser / OS distribution, use it:
| Browser | % of users (last 30d) | Recommended tier |
|---|---|---|
| Chrome 130 | 42% | Tier 1 |
| Chrome 129 | 18% | Tier 1 |
| Safari 18 | 15% | Tier 1 |
| iOS Safari 18 | 10% | Tier 1 |
| Edge 130 | 5% | Tier 1 / 2 |
| Firefox 132 | 4% | Tier 2 |
| Chrome 128 and older | 3% | Tier 3 |
| Other (long tail) | 3% | Tier 3 |
The 80/20 rule: if a configuration has <1% usage, Tier 3 or unsupported. If <0.1%, unsupported.
Worked example
A mid-size web app team caps its browser scope:
Cost check (§3): 6 Tier 1 combos on per-PR smoke stay inside the CI budget while 4 Tier 2 combos run once nightly. The team publishes the §4 support statement listing the three tiers plus the unsupported set, then calendars the §5 quarterly review. Result: cross-browser scope is capped at 10 committed combos behind a documented, defensible support policy.
§7 - Compatibility statement vs accessibility commitment
These are different:
A user with a screen reader on Tier 1 Chrome should have Tier 1 accessibility experience. The two budgets compose.
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| No documented budget | Compatibility creeps; surprise costs. | Author per §2 + publish per §4. |
| Tier 1 = "everything we can test" | CI cost explodes. | Tier 1 should be the minimum-viable; promote consciously. |
| Unsupported = "we don't talk about it" | Users assume support exists; complaints surprise. | Explicit unsupported list (§4). |
| Quarterly review missed | Budget stale; supports end-of-life software. | Calendar invite (§5). |
| User-agent telemetry ignored | Tier 1 list doesn't match reality. | Use telemetry (§6). |
| Same budget across product variants | A B2B SaaS doesn't need the same browser support as a consumer site. | Per-product budget (§2 examples). |
Limitations
References
Worked matrix template
View source (opens in new window)Worked matrix template
A copy-paste template for a committed browser test matrix, filled in for a hypothetical SaaS B2B web app. Replace the traffic numbers with your own analytics, keep the per-row "Why" and the tier-change log, and set real owner / review dates.
# Browser test matrix - Q2 2026
**Owner:** QA lead
**Reviewed:** YYYY-MM-DD
**Next review:** Q3 2026
## Tier 1 - must pass every PR (3 combos)
| Browser | Version | OS | Why T1 | Where tested |
|---|---|---|---|---|
| Chrome | latest stable | Linux | 68% of traffic | `playwright-testing` on CI |
| Chrome | latest-1 (N-1) | Linux | Lag-behind users (Group Policy) | `playwright-testing` on CI |
| Firefox | latest stable | Linux | 12% of traffic | `playwright-testing` (bundled) |
## Tier 2 - nightly + pre-release (5 combos)
| Browser | Version | OS | Why T2 | Where tested |
|---|---|---|---|---|
| Safari | 17 | macOS Sonoma | 8% traffic; Apple-platform-only | BrowserStack Automate |
| Safari iOS | 17 | iOS 17 | Mobile Safari = 9% mobile traffic | BrowserStack Automate (real device) |
| Edge | latest | Windows 11 | 5% traffic | BrowserStack Automate |
| Chrome | latest | Android 14 | 5% mobile traffic | BrowserStack Automate (real device) |
| Firefox | ESR | Linux | Enterprise users on long-support track | `playwright-testing` |
## Tier 3 - quarterly / on-demand (4 combos)
| Browser | Version | OS | Why T3 | Where tested |
|---|---|---|---|---|
| Safari iOS | 16 | iOS 16 | 2% mobile traffic; declining | BrowserStack quarterly |
| Chrome | latest | Linux ARM | <1% but B2B request | BrowserStack quarterly |
| Samsung Internet | latest | Android | <1% but high in some regions | BrowserStack quarterly |
| Opera | latest | Linux | <1%; on-demand only when customer reports | LambdaTest on-demand |
## Not in matrix (out of scope)
- IE11: customer survey 0 users; statutory not required → out of
scope. Re-evaluate Q1 2027.
- Brave / Vivaldi: Chromium-based; covered by Chrome T1.
- Older Firefox versions (Firefox 100 and below): negligible traffic.
## Tier-change log
- 2026-Q1 → Q2: Safari iOS 15 retired (0.4% traffic, below
threshold); Safari iOS 17 added.
- 2026-Q1 → Q2: Edge promoted to T2 (was T3) - traffic grew 3 →
5%.Related skills
cloud-grid-e2e
Author and run E2E tests on a cloud browser grid - BrowserStack Automate, Sauce Labs, or LambdaTest. All three follow one pattern: username + access-key env vars, a W3C WebDriver hub URL, a vendor options dict inside the capabilities (bstack:options / sauce:options / LT:Options), a local tunnel binary for internal apps, session pass/fail reporting, and a CI matrix throttled to the plan's parallel-session limit. Worked example uses BrowserStack; per-vendor deltas live in references/. Use for cross-browser regression on real devices + browsers beyond the engines bundled on the local machine - distinct from a local matrix runner and from self-hosted Selenium Grid.
cypress-testing
Authors and improves Cypress E2E tests - installs Cypress, configures `cypress.config.ts`, authors `cy.*` command chains, refactors existing specs (`cy.wait(ms)` sleeps into assertions, repeated flows into `cy.session` custom commands), and debugs with the time-travel GUI; Cypress Cloud for parallel runs and recording. Use for both greenfield test authoring and improving hand-written specs already in the codebase. For automated refactor of raw Cypress Studio recordings specifically, use a dedicated codegen-review pass.
playwright-testing
Authors and remediates Playwright E2E tests across Chromium, Firefox, WebKit - `npm init playwright@latest` scaffolding, `playwright.config.ts` browser projects, accessibility-first locators (`getByRole`/`getByLabelText`) to replace brittle CSS selectors, web-first assertions to eliminate `waitForTimeout` flakiness, Page Object pattern, trace viewer debugging, sharded parallel execution with merged HTML reporting, mobile-web emulation via the `devices` catalog (viewport / DPR / touch per-device projects), the cross-browser matrix with branded channels (chrome / msedge) in references/browser-matrix.md, and GitHub Actions CI integration. Use for new test authoring, flakiness remediation, mobile-breakpoint regression, cross-browser matrix setup, and CI setup; for reviewing codegen output specifically, use a dedicated codegen-review pass.
selenium-grid-4-runner
Author and operate Selenium Grid 4 - self-hosted distributed WebDriver. Covers the six-component architecture (Router / Distributor / Session Map / Event Bus / New Session Queue / Node), standalone vs hub-and-node modes, the Docker-image stack (selenium/standalone-chrome, selenium/hub, selenium/node-chrome), node registration, session-queue tuning, and observability. Use for self-hosted cross-browser testing when data residency or cost-control require an on-prem grid. This is the self-hosted execution RUNNER - for the zero-infra alternative use playwright-testing browser projects (bundled engines); for managed cloud grids use cloud-grid-e2e (BrowserStack / Sauce Labs / LambdaTest); to decide WHICH browsers and tiers to run use browser-matrix-strategy-reference.
selenium-testing
Authors Selenium WebDriver tests in any of its 6+ supported languages (Java, Python, JavaScript, C#, Ruby, Kotlin, PHP) - picks the appropriate language binding, configures WebDriver per browser, uses `By.*` locators with the team's accessibility-first preference where supported, runs locally + via Selenium Grid for distributed execution, parses results to JUnit XML. Use for legacy Selenium-locked stacks; new projects pick Playwright or Cypress.
web-e2e-overview
Teaches web end-to-end testing from first principles: what browser-driven E2E covers and how it differs from unit and integration tests, a decision table for choosing between Playwright, Cypress, Selenium WebDriver, WebdriverIO, Puppeteer, TestCafe and the BrowserStack / Sauce Labs / LambdaTest cloud grids based on files already present in the repo, install and first-run commands for each, and the flakiness traps (fixed sleeps, CSS and XPath selectors, state shared between tests) that sink new suites. Use when a web application has no E2E coverage yet, when picking or replacing an E2E framework, or when a first browser test needs to go green end to end.
webdriverio-testing
Authors WebdriverIO E2E tests - `npm init wdio@latest` scaffolding, services architecture (sauce, browserstack, appium, devtools), reporters (spec, allure, junit), built-in Mocha/Jasmine/Cucumber framework integrations. WebdriverIO sits between Selenium (W3C protocol) and Playwright (modern API) - Selenium-protocol-compatible with rich plugin ecosystem. Use when the team needs WebDriver protocol + service-based device-farm integration.