mobile-device-matrix-toolkit

Dispatches mobile UI test runs across a 3-tier device matrix (smoke per-PR, regression per-merge, full farm at release) to control CI cost: generates per-target Appium capability configs from a central YAML, parallelises via GitHub Actions matrix strategy, and aggregates JUnit XML into a cross-device pass/fail table. Use when deciding which iOS / Android devices and OS versions to run tests on and at which stage (smoke / regression / full farm), not how to configure a specific test framework (for that, use xcuitest-suite, espresso-suite, etc.).

Install with skills.sh (any agent)

npx skills add testland/qa --skill mobile-device-matrix-toolkit

View source

mobile-device-matrix-toolkit

Overview

Mobile testing has a combinatorial explosion problem:

iOS: 5+ active OS versions × 10+ device sizes ≈ 50+ configs.
Android: 8+ API levels × 100+ device profiles ≈ 800+ configs.

Running every test on every config = CI cost / time disaster.

This skill is a dispatcher that picks the right subset per cadence. It wraps the per-platform test runners (xcuitest-suite, espresso-suite, appium-testing, detox-testing, maestro-flows) and orchestrates matrix execution.

When to use

The team's mobile suite needs cross-device / cross-OS coverage but blanket "run everywhere" is too expensive.
A new device support tier (foldables, larger tablets) needs to be added; the matrix should grow without exploding cost.
A device farm subscription (Firebase Test Lab, BrowserStack, AWS Device Farm, Sauce Labs) is paid for and the team needs a workflow to use it efficiently.

Step 1 - Define the three-tier matrix

# .matrix/devices.yaml
tier_smoke:
  description: "Per-PR - every commit. Cheap, fast feedback."
  ios:
    - { device: "iPhone 15", os: "17.4" }
  android:
    - { device: "Pixel 7", api: 34 }

tier_regression:
  description: "Per-merge to main. Wider coverage."
  ios:
    - { device: "iPhone 15", os: "17.4" }
    - { device: "iPhone SE 3rd", os: "16.0" }
    - { device: "iPad Pro 12.9", os: "17.4" }
  android:
    - { device: "Pixel 7", api: 34 }
    - { device: "Pixel 5", api: 31 }
    - { device: "Galaxy Tab S8", api: 34 }

tier_release:
  description: "Pre-release. Full matrix; runs on device farm."
  ios:
    - device: "iPhone 15", os: "17.4"
    - device: "iPhone 15 Pro Max", os: "17.4"
    - device: "iPhone 14", os: "17.0"
    - device: "iPhone 13", os: "16.0"
    - device: "iPhone SE 3rd", os: "15.0"
    - device: "iPad Pro 12.9", os: "17.4"
    - device: "iPad Mini", os: "16.0"
  android:
    - { device: "Pixel 8 Pro", api: 34 }
    - { device: "Pixel 7", api: 34 }
    - { device: "Pixel 5", api: 31 }
    - { device: "Galaxy S23", api: 33 }
    - { device: "Galaxy A54", api: 33 }
    - { device: "Pixel 4a", api: 30 }
    - { device: "Galaxy Tab S8", api: 34 }

Per-tier guidance:

Tier	When	iOS / Android count	Wall time	Cost (est)
Smoke	Every PR push	1 / 1	~5 min	local sim
Regression	Merge to main	3 / 3	~20 min	local sim
Release	Pre-release tag	7 / 7	~60 min	farm

Step 2 - Generate per-target capabilities

# scripts/gen-matrix.py
import yaml, json, sys

cfg = yaml.safe_load(open(sys.argv[1]))
tier = sys.argv[2]   # smoke | regression | release

targets = []
for ios in cfg[f'tier_{tier}']['ios']:
    targets.append({
        'name': f"ios-{ios['device'].replace(' ', '-')}-{ios['os']}",
        'platform': 'iOS',
        'capabilities': {
            'platformName': 'iOS',
            'appium:deviceName': ios['device'],
            'appium:platformVersion': ios['os'],
            'appium:automationName': 'XCUITest',
        },
    })
for and_ in cfg[f'tier_{tier}']['android']:
    targets.append({
        'name': f"android-{and_['device'].replace(' ', '-')}-api{and_['api']}",
        'platform': 'Android',
        'capabilities': {
            'platformName': 'Android',
            'appium:deviceName': and_['device'],
            'appium:platformVersion': str(and_['api']),
            'appium:automationName': 'UiAutomator2',
        },
    })

print(json.dumps(targets))

CI uses this output as a matrix:

jobs:
  generate-matrix:
    outputs:
      targets: ${{ steps.gen.outputs.targets }}
    steps:
      - id: gen
        run: |
          targets=$(python scripts/gen-matrix.py .matrix/devices.yaml ${{ inputs.tier }})
          echo "targets=$targets" >> "$GITHUB_OUTPUT"

  test:
    needs: generate-matrix
    strategy:
      fail-fast: false
      matrix:
        target: ${{ fromJSON(needs.generate-matrix.outputs.targets) }}
    runs-on: ${{ matrix.target.platform == 'iOS' && 'macos-15' || 'ubuntu-latest' }}
    name: ${{ matrix.target.name }}
    steps:
      - run: ./scripts/run-tests.sh ${{ matrix.target.name }} '${{ toJSON(matrix.target.capabilities) }}'

Step 3 - Per-tier dispatch

# .github/workflows/mobile-tests.yml
on:
  pull_request:
    paths: ['mobile/**']
  push:
    branches: [main]
  release:
    types: [created]

jobs:
  smoke:
    if: github.event_name == 'pull_request'
    uses: ./.github/workflows/run-matrix.yml
    with:
      tier: smoke

  regression:
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    uses: ./.github/workflows/run-matrix.yml
    with:
      tier: regression

  release:
    if: github.event_name == 'release'
    uses: ./.github/workflows/run-matrix.yml
    with:
      tier: release

Step 4 - Aggregate per-target results

Before trusting the roll-up: a shard that fails to upload its JUnit XML (device timeout, farm hiccup) reads as 0 tests, not failure - detect shards with a missing or empty report, re-run only those shards, then re-aggregate. Each matrix shard uploads its JUnit XML; an aggregator job combines:

# scripts/aggregate-matrix.py
import xml.etree.ElementTree as ET
from collections import defaultdict
import sys

per_target = defaultdict(lambda: {'tests': 0, 'failures': 0, 'errors': 0, 'time': 0.0})

for f in sys.argv[1:]:
    target = f.split('/')[-2]   # extract from path
    tree = ET.parse(f)
    for ts in tree.iter('testsuite'):
        per_target[target]['tests'] += int(ts.get('tests', 0))
        per_target[target]['failures'] += int(ts.get('failures', 0))
        per_target[target]['errors'] += int(ts.get('errors', 0))
        per_target[target]['time'] += float(ts.get('time', 0))

# Render matrix
print('| Target | Tests | Pass | Fail | Time |')
print('|--------|------:|-----:|-----:|-----:|')
for target, m in sorted(per_target.items()):
    pass_ = m['tests'] - m['failures'] - m['errors']
    print(f"| {target} | {m['tests']} | {pass_} | {m['failures']+m['errors']} | {m['time']:.1f}s |")

Output:

| Target                          | Tests | Pass | Fail | Time   |
|---------------------------------|------:|-----:|-----:|-------:|
| ios-iPhone-15-17.4               |   42  |   42 |    0 | 320.4s |
| ios-iPhone-SE-3rd-16.0           |   42  |   41 |    1 | 295.1s |   ← Pixel 5 only
| android-Pixel-7-api34            |   42  |   42 |    0 | 280.5s |
| android-Galaxy-Tab-S8-api34      |   42  |   40 |    2 | 290.0s |   ← tablet layout issues

Step 5 - Device-farm vs local emulator decision

Use device farm when	Use local emulator/simulator when
Real-device behavior matters (camera, sensors)	UI logic only
Specific OEM device under test (Samsung, foldable)	Stock OS suffices
iOS testing on Linux CI	macOS CI runner available
Release-tier matrix (cost amortizes over fewer runs)	Per-PR (cost per run too high)
Regulatory / certification testing	Rapid iteration

Per-farm wiring:

# Firebase Test Lab (Android)
- run: |
    gcloud firebase test android run \
      --type instrumentation \
      --app app/build/outputs/apk/debug/app-debug.apk \
      --test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk \
      --device model=Pixel7,version=34,locale=en \
      --device model=GalaxyS23,version=33,locale=en

# BrowserStack (iOS + Android)
- run: |
    npx browserstack-runner --config browserstack.json

Anti-patterns

Anti-pattern	Why it fails	Fix
Same matrix for every commit	CI cost explosion; team disables.	Three-tier dispatch (Step 1, 3).
Single device per platform	Misses tablet / older-OS regressions until release.	Regression tier covers 3 / 3 (Step 1).
Device-farm runs on every PR	Per-minute cost; budget exhausts mid-month.	Farm only for release tier.
`fail-fast: true` on the matrix	One failing target cancels others; lose coverage signal.	`fail-fast: false` (Step 2).
Hard-coded device list in CI yaml	Devices added / OS versions deprecated → manual yaml updates everywhere.	Centralized `.matrix/devices.yaml` (Step 1).
No aggregated report	Per-target results buried in CI logs; reviewer can't see the matrix.	Aggregator job (Step 4).

Limitations

Cost-coverage trade-off. Even the release tier doesn't cover every config. Choose representative devices per OS family.
Per-platform CI runner availability. GitHub Actions macOS runners are paid (after free quota); large iOS matrix is expensive even on local sims.
Farm SLA varies. Farms have queue times; release tier may take 1-3 hours wall-time even with parallelism.
Device-farm flakiness. Real devices on shared infrastructure have intermittent issues; pair with retries (cautiously) and per-device flake tracking.

References

xcuitest-suite, espresso-suite, appium-testing, detox-testing, maestro-flows - per-platform runners this dispatcher orchestrates.
junit-xml-analysis (in the qa-test-reporting plugin) - parser for the per-target JUnit XML aggregation.
Per Mike Cohn's test pyramid (cited in test-pyramid-balancer), mobile UI tests are the most expensive layer; matrix dispatch is the cost-management discipline.

Related skills

appium-testing

Wires Appium for cross-platform mobile UI automation - uses the WebDriver protocol, picks a driver per platform (XCUITest for iOS, UiAutomator2 / Espresso for Android, Mac2 for macOS, Windows for desktop), authors tests in JS / Python / Java / Ruby / .NET, configures `desiredCapabilities`, runs against simulators / emulators / device farms. Use when a single test suite must cover both iOS and Android, or when the team's stack is multi-platform (iOS + Android + Mac + Windows).

detox-testing

Authors React Native E2E tests with Detox (Wix) - gray-box architecture (runs in-process with the app), `element(by.id|by.text|by.label)` matchers, `waitFor()` for explicit sync beyond Detox's automatic async tracking, Jest runner. Use when the app is React Native and speed matters. For Flutter use flutter-testing; for black-box cross-platform use appium-testing; for YAML-declarative flows use maestro-flows; for non-RN native use xcuitest-suite or espresso-suite.

espresso-suite

Authors Espresso UI tests for Android - uses `onView(withId(...)).perform(...).check(matches(...))`, leans on Espresso's automatic synchronization (no `Thread.sleep`), wires `IdlingResource` for app-specific async, runs via `./gradlew connectedAndroidTest` and parses the JUnit XML output. Use when an Android app needs UI tests in Google's first-party framework.

flutter-testing

Authors Flutter tests across the three-layer pyramid - unit (`flutter test`), widget (`testWidgets` + `WidgetTester`), integration (`integration_test` on simulator/emulator/device). Picks the right layer per change, mocks via `mockito` + `build_runner`, LCOV coverage, CI with the Flutter Action. Use when the app is Flutter and the team wants its first-party stack. For React Native use detox-testing; for black-box cross-platform use appium-testing; for YAML-declarative flows use maestro-flows.

maestro-flows

Authors Maestro YAML flow files (`.maestro/*.yaml`) for mobile + web UI automation: declarative `tapOn`, `inputText`, `assertVisible`, `swipe`, supported targets (iOS, Android, Flutter, React Native, web), nested flow imports, JavaScript hooks for complex conditions. Use when the team has already chosen Maestro, is coming from an existing `.maestro/` directory, or explicitly wants YAML-declarative tests readable by non-engineers without a compile step. For framework selection or authoring tests in XCUITest / Espresso / Detox / Appium / Flutter, use a mobile driver-selection or per-flow mobile test-authoring step instead.

mobile-a11y-test-author

Authors native mobile accessibility tests covering iOS (Accessibility Inspector, XCUITest `performAccessibilityAudit()` introduced in iOS 17, VoiceOver label/trait/hint verification) and Android (Espresso `AccessibilityChecks.enable()`, Accessibility Scanner, TalkBack traversal, `contentDescription` labelling) with WCAG-aligned checks for element labels, 44pt/48dp touch targets, contrast ratios, and focus order. Use when an iOS or Android app needs automated and manual accessibility test coverage beyond what `xcuitest-suite` or `espresso-suite` provide.

mobile-web-emulation-runner

Builds a workflow to run web E2E tests under mobile viewports + DPRs (device pixel ratios): Playwright's `devices` catalog (iPhone 15, Pixel 7), suite run per-device as matrix shards, per-device screenshots, mobile assertions (`.tap()`, viewport-conditional layout). Use when a responsive web app needs mobile-breakpoint regression without a real-device farm. Mobile WEB only - for native apps use appium-testing, detox-testing, or flutter-testing; for cross-shard aggregation use mobile-device-matrix-toolkit; for gesture sequences use touch-gesture-tester.

mobile-web-perf-budget

Pure-reference skill for mobile-web performance budgets - Core Web Vitals at the 75th percentile mobile (LCP ≤2.5s, INP ≤200ms, CLS ≤0.1; FID retired March 2024 in favor of INP), Lighthouse mobile profile config, per-route resource budgets (JS bundle, image weight, font load). Use as the team's reference for "what should the mobile perf gate enforce" - paired with `lighthouse-perf` (the runner) and `lighthouse-budget-author` (the per-route author).

touch-gesture-tester

Verifies touch-gesture handlers (tap, double-tap, long-press, swipe, pinch, rotate, pan) work as expected under both mobile-emulation (Playwright) and native (XCUITest / Espresso / Detox) - distinguishes "mouse click handler also fires on tap" from "real touch event fired with correct properties." Use when the app has bespoke gesture handlers (custom carousels, sliders, drag-drop, pull-to-refresh) and the team needs targeted gesture verification beyond generic UI assertions.

xcuitest-suite

Authors XCUIest UI tests for iOS / iPadOS / tvOS - uses the three-class XCUIApplication / XCUIElement / XCUIElementQuery pattern, sets accessibility identifiers on production code, runs via `xcodebuild test` with destination, parses the `xcresult` bundle. Use when an iOS app needs UI tests in Apple's first-party framework (no external runtime; native to Xcode).