qa-shift-right
Production-side QA per ISTQB-canonical shift right ("a test approach to test a system continuously in production"): 4 skills (feature-flag-experiment-validator, prod-canary-validator, rum-to-synthetic-gap-analyzer, synthetic-monitor-author) and 2 agents (canary-and-experiment-coordinator, observability-to-test).
Install this plugin
/plugin install qa-shift-right@testland-qaPart of role bundle: qa-role-performance
qa-shift-right
Production-side QA per ISTQB-canonical shift right ("a test approach to test a system continuously in production"). Synthetic monitors that exercise critical journeys (with RUM-derived coverage gap analysis), canary-deploy validators with statistical comparison vs baseline, A/B / feature-flag experiment significance validators, release runbooks with the multi-team cutover sequence in references, and the loop from production-side incident → regression test added.
Components
| Type | Name | Description |
|---|---|---|
| Skill | synthetic-monitor-author | Build-an-X synthetic monitor: pick journey + platform (Datadog/Checkly/Pingdom/etc.) + Playwright-style script + per-step assertions + multi-region cadence + alert thresholds. Includes the RUM-coverage gap method: score real-user journeys from RUM / CrUX data, diff against the monitor inventory, emit a ranked gap list. |
| Skill | prod-canary-validator | Build-an-X canary verdict: per-metric absolute + relative thresholds, two-sample statistical tests (chi-square / Welch's t-test), promote/pause/rollback verdict. |
| Skill | feature-flag-experiment-validator | Build-an-X A/B test analysis: chi-square / Welch's / Mann-Whitney U per metric, FDR multiple-comparisons correction, practical-vs-statistical significance, ship/don't-ship verdict. |
| Skill | release-runbook-author | Writes one service's six-phase release runbook (pre-flight, smoke gate, canary observation, human promote gate, rollout, post-release) with each phase gated on a delta against a recorded baseline. The multi-team cutover sequence (dependency-ordered gates, named owners, timeboxes, reverse-order rollback) lives in references/. |
| Agent | observability-to-test | Closes the loop: production-signal → regression test (cheapest catching layer per test pyramid) + fix PR + postmortem update. |
Install
/plugin marketplace add testland/qa
/plugin install qa-shift-right@testland-qaSkills
feature-flag-experiment-validator
Validates the statistical significance of an A/B / feature-flag experiment result - computes per-metric effect size + p-value (chi-square for proportions, Welch's t-test for continuous metrics), applies a multiple-comparison correction (Bonferroni / Benjamini-Hochberg) when N>1 metric, surfaces practical-vs-statistical-significance distinction, and emits a ship/don't-ship verdict per metric. Use when an experiment has finished and someone is about to ship the winning variant off a dashboard readout, when a result rests on a small sample, or when more than one metric was compared - the rigorous version of "the variant looks better in the dashboard."
prod-canary-validator
Builds a canary-validation workflow that compares a canary deploy's metrics against the baseline (current main) - picks the metric set (error rate, p50/p95/p99 latency, business KPIs like checkout-completion), defines per-metric thresholds (absolute + relative-to-baseline), runs a statistical-comparison check (effect size + significance) over the canary's observation window, and emits a promote/rollback verdict. Use as the gate between canary deploy and full rollout - the deterministic version of "the on-call eyeballs the dashboard for 30 min.
release-runbook-author
Turns one service's release into a written six-phase runbook: pre-flight checks, a smoke gate, a canary observation window, a named human promote gate, progressive rollout, and post-release verification. Fixes each phase's pass criteria as a delta against a recorded baseline rather than a bare absolute number, gives canary and rollout separate windows and separate thresholds, and emits a per-phase evidence table that becomes the release record. The multi-team cutover-sequence procedure - dependency-ordered gates with one named owner each, hard timeboxes, written rollback triggers, and the reverse-order rollback path - is worked in references for windows where several teams cut over interdependent services. Use when a single service is about to ship and its release steps exist only as tribal knowledge or a chat thread, or when a shared release window needs its cutover order, gate owners, and rollback path written down.
synthetic-monitor-author
Drafts a synthetic monitor configuration for one critical user journey - picks the platform (Datadog Synthetics, Pingdom, Checkly, New Relic, etc.), authors the scripted-transaction body (Playwright-style for browser checks; HTTP-step for API checks), wires the cadence (typical 1-15 min), defines per-step assertions (DOM presence, API status, response shape) and aggregate alert thresholds (consecutive-failure count + on-call routing). Includes the RUM-coverage gap method for deciding which journeys to monitor: score real-user journeys from RUM / CrUX data by session volume times business value, diff against the existing monitor inventory, and emit a ranked gap list. Use when a critical journey needs continuous-in-production verification per ISTQB-canonical shift-right ("a test approach to test a system continuously in production"), or when synthetic coverage was never systematically derived from real usage data.