browser-matrix-strategy-reference

Overview

"Test on every browser" is not a strategy. A real matrix tiers browsers by traffic share, regulatory requirement, and team budget, then runs different tiers at different cadences.

This skill is a pure reference consumed by the cloud-grid skills (browserstack-automate, saucelabs-automate, lambdatest-automate)

• the self-hosted selenium-grid-4-runner
• existing browser-matrix-runner (bundled engines).

When to use

• New project - defining the initial browser matrix.
• Quarterly review - re-tier browsers as traffic shifts (e.g., IE11 traffic drops to <0.1%, demote from T2 to T3).
• Audit / compliance - defending why specific browsers were tested (or weren't).
• Cost optimisation - moving low-tier browsers from cloud grid (paid) to bundled engines (free).

The three-tier model

Data sources for traffic share

Use your own analytics for traffic-share decisions; supplement with StatCounter when slicing geographies you don't have own data for.

Worked matrix template

For a hypothetical SaaS B2B web app:

# Browser test matrix — Q2 2026

**Owner:** QA lead
**Reviewed:** YYYY-MM-DD
**Next review:** Q3 2026

## Tier 1 — must pass every PR (3 combos)

| Browser | Version | OS | Why T1 | Where tested |
|---|---|---|---|---|
| Chrome | latest stable | Linux | 68% of traffic | qa-web-e2e/playwright-testing on CI |
| Chrome | latest-1 (N-1) | Linux | Lag-behind users (Group Policy) | qa-web-e2e/playwright-testing on CI |
| Firefox | latest stable | Linux | 12% of traffic | qa-compatibility/browser-matrix-runner (bundled) |

## Tier 2 — nightly + pre-release (5 combos)

| Browser | Version | OS | Why T2 | Where tested |
|---|---|---|---|---|
| Safari | 17 | macOS Sonoma | 8% traffic; Apple-platform-only | BrowserStack Automate |
| Safari iOS | 17 | iOS 17 | Mobile Safari = 9% mobile traffic | BrowserStack Automate (real device) |
| Edge | latest | Windows 11 | 5% traffic | BrowserStack Automate |
| Chrome | latest | Android 14 | 5% mobile traffic | BrowserStack Automate (real device) |
| Firefox | ESR | Linux | Enterprise users on long-support track | qa-compatibility/browser-matrix-runner |

## Tier 3 — quarterly / on-demand (4 combos)

| Browser | Version | OS | Why T3 | Where tested |
|---|---|---|---|---|
| Safari iOS | 16 | iOS 16 | 2% mobile traffic; declining | BrowserStack quarterly |
| Chrome | latest | Linux ARM | <1% but B2B request | BrowserStack quarterly |
| Samsung Internet | latest | Android | <1% but high in some regions | BrowserStack quarterly |
| Opera | latest | Linux | <1%; on-demand only when customer reports | LambdaTest on-demand |

## Not in matrix (out of scope)

- IE11: customer survey 0 users; statutory not required → out of
  scope. Re-evaluate Q1 2027.
- Brave / Vivaldi: Chromium-based; covered by Chrome T1.
- Older Firefox versions (Firefox 100 and below): negligible traffic.

## Tier-change log

- 2026-Q1 → Q2: Safari iOS 15 retired (0.4% traffic, below
  threshold); Safari iOS 17 added.
- 2026-Q1 → Q2: Edge promoted to T2 (was T3) — traffic grew 3 →
  5%.

Tier-membership heuristic

A browser belongs in:

• T1 if any of:

  • Currently ≥5% of own traffic, AND
  • "Most users" (statistically reasoned, not gut)
  • Browser the team uses internally (dogfooding)

• T2 if any of:

  • 1-5% of own traffic
  • Required by stakeholder commitment / SLA
  • Required by regulation (e.g., accessibility on Safari iOS for AppStore review)
  • Mobile platform with significant traffic

• T3 if any of:

  • <1% traffic but customer-requested
  • Long-tail enterprise (ESR / extended-support tracks)
  • Coverage-completeness for audit defensibility

• Out of scope if all of:

  • <0.1% traffic
  • No customer commitment
  • No regulatory requirement
  • Already covered by an engine-compatible browser (e.g., Brave covered by Chrome)

Cost-tier mapping

Where to test each tier:

For very high-volume + cost-sensitive teams, self-hosted Selenium Grid 4 for T1 + cloud grid for T2 / T3 is the optimum mix.

Engine vs runner distinction

A T1 entry "Chrome latest" can mean:

• Bundled Chromium engine in Playwright - fast, free, but engineered for testing (not 1:1 with stable Chrome)
• Real Chrome stable channel via cloud grid - slower, paid, but exactly what users have

For T1 the bundled-engine path is usually sufficient; for T2 / T3, real-browser via cloud grid is the choice when the difference matters (some bugs are real-Chrome-only).

Anti-patterns

Limitations

• Own-analytics blindness. Bot traffic + anonymised users can skew traffic counts; sanity-check via two sources.
• Cross-region traffic differs. Global average + your specific market may diverge; segment.
• Bundled engines aren't real browsers. Playwright's Chromium / WebKit have engineering-for-testing edits; for some bugs only real-browser cloud grids catch.
• Real-device matrices shift. Cloud grids retire devices / add new ones; quarterly check.

References

• StatCounter GlobalStats - gs.statcounter.com.
• MDN browser-compat-data - github.com/mdn/browser-compat-data.
• caniuse.com - caniuse.com.
• web-platform-tests / web-platform.org - web-platform-tests.org.
• Web feature status - webstatus.dev.
• GitHub Actions runner-images - github.com/actions/runner-images.
• W3C WebDriver specification - w3.org/TR/webdriver2/.
• Sibling skills: browser-matrix-runner (existing - bundled engines), selenium-grid-4-runner, browserstack-automate, saucelabs-automate, lambdatest-automate.