qa-feature-flags
Feature-flag platform testing + experimentation harness testing: SDK-specific tests for LaunchDarkly plus the OpenFeature umbrella (Unleash, Flagsmith, GrowthBook); the feature-flag test matrix reference; the experiment-sdk-testing umbrella (Statsig, Optimizely, Split.io, Amplitude Experiment, VWO); AB-test validity checklist; experiment-results interpreter (peeking + guardrail references); stale-flag detector with the removal runbook; flag-coverage-gap detector; sample-ratio-mismatch (SRM) detector. Distinct from qa-test-environment/feature-flag-test-harness (generic flag-aware test harness) and qa-shift-right/feature-flag-experiment-validator (validates production experiment results); this plugin scopes to platform-SDK testing, flag-lifecycle hygiene, and experimentation-harness integrity.
Install this plugin
/plugin install qa-feature-flags@testland-qaPart of role bundle: qa-role-backend
qa-feature-flags
Feature-flag platform testing + experimentation harness testing: LaunchDarkly SDK tests, the OpenFeature SDK umbrella (with Unleash / Flagsmith / GrowthBook vendor references), the feature-flag test matrix reference (coverage-suite building + kill-switch test categories), the experiment-sdk-testing umbrella (Statsig, Optimizely, Split.io, Amplitude Experiment, VWO hermetic harnesses), the AB-test validity checklist, the experiment-results interpreter (peeking + guardrail methodology in its references), and three agents: flag-coverage-gap detection, stale-flag detection with the removal runbook, and sample-ratio-mismatch (SRM) detection. Distinct from qa-test-environment/feature-flag-test-harness (generic flag-aware test harness) and qa-shift-right/feature-flag-experiment-validator (validates production experiment results); this plugin scopes to platform-SDK testing, flag-lifecycle hygiene, and testing the experimentation harness itself.
Components
| Type | Name | Description |
|---|---|---|
| Skill | launchdarkly-testing | Wraps LaunchDarkly server-side SDK testing patterns: TestData data source for hermetic tests (no network), file-based data source for fix... |
| Skill | openfeature-sdk-testing | OpenFeature (CNCF vendor-neutral) SDK testing umbrella: InMemoryProvider hermetic tests, EvaluationDetails assertions, hooks; plus Unleash / Flagsmith / GrowthBook vendor references. |
| Skill | feature-flag-test-matrix-reference | Flag-state combinatorics + coverage strategies, the coverage-suite building workflow, and kill-switch test categories (references/killswitch.md). |
| Skill | ab-test-validity-checklist | Workflow-driven skill that builds an A/B test validity checklist from an experiment proposal. |
| Skill | experiment-sdk-testing | Shared offline-datafile / hermetic-init pattern with per-vendor references for Statsig, Optimizely, Split.io, Amplitude Experiment, and VWO. |
| Skill | experiment-results-interpreter | Interprets valid experiment results; peeking-problem and guardrail-metric methodology in references/. |
| Agent | flag-coverage-gap-detector | Read-only adversarial critic that scans code for flag-evaluation call sites (isEnabled / getBooleanValue / variation / variationDetail) a... |
| Agent | stale-flag-detector | Read-only specialist that ranks stale feature flags for removal and attaches the safe-removal runbook (verification, code + platform removal, rollback). |
| Agent | sample-ratio-mismatch-detector | Read-only specialist that detects Sample Ratio Mismatch (SRM) in an A/B test by running a chi-square test against the observed-vs-expecte... |
Install
/plugin marketplace add testland/qa
/plugin install qa-feature-flags@testland-qaSkills
ab-test-validity-checklist
Workflow skill that builds an A/B-test validity checklist from an experiment proposal, walking the canonical design-correctness gates - pre-registered OEC/power/guardrails, randomization unit + SRM check, assignment integrity, telemetry, peeking discipline, novelty/primacy, post-experiment SRM re-check - into a per-experiment checklist + sign-off form. Use when launching, auditing, or governing an experiment. For pitfall mechanics (guardrails, peeking) see experiment-results-interpreter's references; to read an already-valid result use experiment-results-interpreter; for per-SDK harness tests use experiment-sdk-testing - this gates DESIGN, not SDK code.
experiment-results-interpreter
Interprets the results of a valid online controlled experiment, one whose harness, SRM, and telemetry have already been confirmed. Covers the distinction between practical and statistical significance, reading confidence intervals instead of binary p-values, novelty and primacy week-over-week decay that causes post-ship reversion, interaction effects from concurrent experiments, Simpson's paradox in segmented results, and the ordered guardrail-check sequence required before a ship decision - with the deep methodology in references/: the peeking problem and its corrections (fixed-horizon, alpha-spending, always-valid mSPRT) in references/peeking.md, and guardrail-metric methodology (taxonomy, OEC relationship, pre-commitment, thresholds) in references/guardrails.md. Use when a data scientist or PM is ready to draw conclusions from an experiment, when designing a stop-early policy, or when declaring an experiment's guardrail set. Distinct from ab-test-validity-checklist (harness setup and SRM detection).
experiment-sdk-testing
Umbrella for experimentation-SDK test harnesses: the shared offline-datafile / hermetic-init pattern (commit a point-in-time flag/experiment config fixture, initialize the SDK with no network, pin arms per test, assert assignment integrity), with per-vendor references for Statsig (localMode + overrideGate), Optimizely (datafile + forced decisions), Split.io / Harness FME (localhost mode + features map or YAML fixture), Amplitude Experiment (local evaluation + bootstrap), and VWO (settings file + deterministic bucketing). Use when writing tests for application code instrumented with any of these five experimentation SDKs; for experiment DESIGN gates use ab-test-validity-checklist, and to read results use experiment-results-interpreter.
feature-flag-test-matrix-reference
Feature-flag test matrix design: the flag-state combinatorics problem (N flags × M variants × K user-segments = N×M×K test cases), the canonical coverage strategies (pairwise interaction coverage; default-only smoke; full matrix; risk-driven matrix), the workflow for building the coverage suite from a flag inventory (grep-based inventory, per-flag classification, PICT pairwise generation, per-cell test skeletons), the dedicated kill-switch test categories (references/killswitch.md: graceful degradation, fail-static default, kill latency, mid-flight consistency), and the flags-vs-experiments distinction. Use when designing the flag-test surface for a new project, building or auditing flag-test coverage, or authoring kill-switch tests.
launchdarkly-testing
Wraps LaunchDarkly server-side SDK testing patterns: TestData data source for hermetic tests (no network), file-based data source for fixture-driven tests, flag override patterns (TestData.update for per-test flag values), and assignment-integrity tests. Use when writing tests for code that uses LaunchDarkly flags; to decide which flag combinations those tests should cover in the first place, see feature-flag-test-matrix-reference.
openfeature-sdk-testing
Wraps OpenFeature (CNCF vendor-neutral SDK abstraction) testing patterns: the InMemoryProvider for hermetic tests without network calls, provider registration via OpenFeature.setProvider, the getBooleanValue/getBooleanDetails evaluation API with EvaluationDetails (value, variant, reason, errorCode), hooks for evaluation side-effects, and evaluation context for targeting-rule tests. Covers TypeScript, Java, and Python SDKs, plus per-vendor hermetic-bootstrap references for Unleash (bootstrap toggles), Flagsmith (offline LocalFileHandler), and GrowthBook (initSync payload). Use when writing tests for code that resolves feature flags through the OpenFeature SDK or the Unleash / Flagsmith / GrowthBook native SDKs; LaunchDarkly has its own skill (launchdarkly-testing).
Agents
flag-coverage-gap-detector
Read-only adversarial critic that scans code for flag-evaluation call sites (isEnabled / getBooleanValue / variation / variationDetail) and identifies flag branches whose OFF path, FALLTHROUGH path, or non-default variants have no corresponding test exercising them. Emits a ranked list of untested flag branches and an overall coverage-gap verdict. Use after adding or changing a feature flag to confirm test coverage exists for every reachable branch - distinct from stale-flag-detector (which finds flags whose AGE or rollout state suggests removal) and feature-flag-test-matrix-reference (whose workflow BUILDS a new coverage matrix from scratch); this agent audits what is already in the test suite against what branches the production code actually reaches.
sample-ratio-mismatch-detector
Read-only specialist that detects Sample Ratio Mismatch (SRM) in an A/B test by running a chi-square test against the observed-vs-expected allocation. Returns a verdict (clean / SRM detected) and, if SRM detected, a taxonomy of likely root causes per the Microsoft Research KDD 2019 paper 'Diagnosing Sample Ratio Mismatch' (logging bugs, bot filtering, redirects, telemetry drops, randomization bugs). Use proactively at experiment-end before any ship decision, or when investigating surprising results. Preloads experiment-results-interpreter (guardrail + peeking methodology in its references/).
stale-flag-detector
Read-only specialist that scans a codebase for stale feature flags - flags at 100% rollout for long enough to remove, kill-switches that haven't been touched in months, experiments that have already shipped, and orphan flags (referenced in code but not in the platform, or vice versa) - and returns a ranked removal-candidate list plus the safe-removal runbook for each (pre-removal verification, code + platform-side removal steps, post-removal checks, rollback plan). Use proactively monthly / quarterly as flag-debt audit, or as a pre-flight check before a major refactor. Preloads feature-flag-test-matrix-reference.