browser-matrix-runner
Configures a CI matrix that runs the smoke / regression suite across multiple browsers per Playwright's three-engine support (Chromium, Firefox, WebKit / Safari) plus branded variants (chrome, msedge channels). Wires GitHub Actions / GitLab CI matrix syntax, captures per-browser screenshots, and aggregates per-browser pass/fail. Use when the product targets multiple browsers and the team wants automated cross-browser testing or browser-compatibility regression - e.g. a 'works in Chrome, broken in Safari' bug that needs Chrome / Firefox / Safari coverage.
Install with skills.sh (any agent)
npx skills add testland/qa --skill browser-matrix-runnerbrowser-matrix-runner
Overview
Per pw-browsers (opens in new window):
Playwright supports three browser engines:
Plus branded variants (pw-browsers (opens in new window)):
"Playwright can operate against Google Chrome and Microsoft Edge when installed on the machine. Available channels include:
chrome,chrome-beta,chrome-dev,chrome-canary,msedge,msedge-beta,msedge-dev,msedge-canary."
When to use
Step 1 - Install browsers
Per pw-browsers (opens in new window):
# All Playwright-bundled browsers
npx playwright install
# Specific browsers
npx playwright install chromium firefox webkit
# With system dependencies (CI-friendly)
npx playwright install --with-depsBranded:
npx playwright install msedge # only on Windows / macOS
npx playwright install chromeStep 2 - Configure projects
// playwright.config.ts
import { defineConfig, devices } from '@playwright/test';
export default defineConfig({
projects: [
{ name: 'chromium', use: { ...devices['Desktop Chrome'] } },
{ name: 'firefox', use: { ...devices['Desktop Firefox'] } },
{ name: 'webkit', use: { ...devices['Desktop Safari'] } },
{ name: 'edge', use: { ...devices['Desktop Edge'], channel: 'msedge' } },
{ name: 'chrome', use: { ...devices['Desktop Chrome'], channel: 'chrome' } },
],
});Step 3 - Run
# All browsers
npx playwright test
# Specific browser
npx playwright test --project=firefox
# Multiple
npx playwright test --project=chromium --project=webkitStep 4 - CI matrix
# .github/workflows/cross-browser.yml
jobs:
test:
strategy:
fail-fast: false
matrix:
browser: [chromium, firefox, webkit]
include:
- browser: edge
os: windows-latest # msedge on Windows
- browser: chrome
os: ubuntu-latest
runs-on: ${{ matrix.os || 'ubuntu-latest' }}
steps:
- uses: actions/checkout@v5
- uses: actions/setup-node@v4
with: { node-version: '20' }
- run: npm ci
- run: npx playwright install --with-deps ${{ matrix.browser }}
- run: npx playwright test --project=${{ matrix.browser }}
- uses: actions/upload-artifact@v4
if: always()
with:
name: playwright-${{ matrix.browser }}
path: playwright-report/fail-fast: false ensures Chromium failure doesn't cancel WebKit / Firefox.
Verify each browser leg: assert npx playwright install --with-deps exited 0 and a playwright-report/ artifact was produced before counting the browser as passed. If install fails, the cause is almost always a missing system dependency - re-run with --with-deps (wired above) and surface the missing package from the installer's error output, then re-run the leg.
Step 5 - Per-browser failure analysis
When per-browser failures appear:
## Cross-browser results - `<sha>`
| Browser | Tests | Pass | Fail | Time |
|-----------|------:|-----:|-----:|-------:|
| Chromium | 42 | 42 | 0 | 120s |
| Firefox | 42 | 42 | 0 | 135s |
| WebKit | 42 | 38 | 4 | 140s | ← WebKit-specific issues
| Edge | 42 | 42 | 0 | 125s |
| Chrome | 42 | 42 | 0 | 120s |
### WebKit-only failures
| Test | Symptom |
|-------------------------------------|-------------------------------------------|
| `checkout.spec.ts > apply promo` | `100vh` calculated wrong on iOS Safari |
| `cart.spec.ts > add item` | `IntersectionObserver` callback timing diff |
| ... | |WebKit-only failures often cluster around well-known iOS Safari quirks (viewport units, scroll behavior, font rendering).
Step 6 - Engine vs channel choice
| Use the engine (Playwright-bundled) when | Use the branded channel when |
|---|---|
| Default; reproducible across team / CI | Need to test specific Chrome / Edge release behavior |
| Faster CI (lighter download per pw-browsers (opens in new window)) | Compliance / contract requires the branded build |
| Pre-release behavior (Chromium N+1) | Bug reports against branded versions |
For most teams: bundled engines for CI; branded channels for manual / spot-check.
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Single-browser CI | Cross-browser regressions invisible. | Matrix (Step 4). |
fail-fast: true on the matrix | Chromium fails; team can't see WebKit / Firefox. | fail-fast: false (Step 4). |
Skipping --with-deps install | Linux runner missing browser dependencies; tests fail. | Always --with-deps in CI (Step 1). |
| Per-browser conditional code in tests | Couples tests to browser implementation; fragile. | Test what users observe; abstract per-browser quirks in production code. |
| Not testing branded channels at all | Misses Edge / Chrome-specific bugs. | Schedule weekly branded-channel runs (Step 6). |
Limitations
References
Related skills
browser-matrix-strategy-reference
Pure-reference for designing and reviewing a browser / OS / device test matrix from traffic data - the T1/T2/T3 tier-membership heuristics (T1 >=5% traffic, T2 1-5% or statutory, T3 <1% with customer demand), the traffic-share sources (own analytics, StatCounter, MDN browser-compat-data), a worked matrix template with tier-change log, and how to justify dropping a legacy browser (IE11, old iOS Safari). Use when designing an initial matrix, running a quarterly re-tier review, or making the case to drop a browser. This is the WHAT-to-test strategy reference - to execute the matrix use the runners browser-matrix-runner (bundled engines) or selenium-grid-4-runner (self-hosted); to cap and publish committed support tiers use compatibility-budget.
compatibility-budget
Pure-reference for deciding how large a compatibility matrix a team can afford and for publishing that commitment - defines tier-1 (must work; per-PR) vs tier-2 (must work; nightly) vs tier-3 (should work; pre-release) vs unsupported, with example budgets per product type (web / desktop / mobile / library), the matrix-size cost / coverage trade-off, and 'what we support' external templates. Use when a team must cap how many browser / OS / runtime combos it commits to, or must publish a support policy. This is the BUDGET and support-statement gate - for the traffic-share analysis that picks WHICH specific browsers belong in each tier use browser-matrix-strategy-reference; to execute the resulting matrix use the runners browser-matrix-runner (bundled engines) or selenium-grid-4-runner (self-hosted).
os-matrix-runner
Configures a CI matrix that runs tests across operating systems (Linux / macOS / Windows) and runtime versions (Node 18/20/22; Python 3.10/3.11/3.12; Java 17/21; .NET 6/8). Wires GitHub Actions matrix syntax, addresses OS-specific quirks (path separators, line endings, file permissions). Use when the product ships across OS / runtime combinations and the team needs continuous cross-platform coverage.
selenium-grid-4-runner
Author and operate Selenium Grid 4 - self-hosted distributed WebDriver. Covers the six-component architecture (Router / Distributor / Session Map / Event Bus / New Session Queue / Node), standalone vs hub-and-node modes, the Docker-image stack (selenium/standalone-chrome, selenium/hub, selenium/node-chrome), node registration, session-queue tuning, and observability. Use for self-hosted cross-browser testing when data residency or cost-control require an on-prem grid. This is the self-hosted execution RUNNER - for the zero-infra alternative use browser-matrix-runner (Playwright bundled engines); for managed cloud grids use browserstack-automate, saucelabs-automate, or lambdatest-automate; to decide WHICH browsers and tiers to run use browser-matrix-strategy-reference.