feature-flag-test-matrix-reference

Feature-flag test matrix design: the flag-state combinatorics problem (N flags × M variants × K user-segments = N×M×K test cases), the canonical coverage strategies (pairwise interaction coverage; default-only smoke; full matrix; risk-driven matrix), the workflow for building the coverage suite from a flag inventory (grep-based inventory, per-flag classification, PICT pairwise generation, per-cell test skeletons), the dedicated kill-switch test categories (references/killswitch.md: graceful degradation, fail-static default, kill latency, mid-flight consistency), and the flags-vs-experiments distinction. Use when designing the flag-test surface for a new project, building or auditing flag-test coverage, or authoring kill-switch tests.

Install with skills.sh (any agent)

npx skills add testland/qa --skill feature-flag-test-matrix-reference

View source

feature-flag-test-matrix-reference

Overview

A codebase with N feature flags, each having M variants, and users in K segments, has N × M × K possible flag-state-segment combinations. At realistic numbers (50 flags, 2 variants each, 5 segments) that's 500 - and at 50 flags with 3 variants and 10 segments, it's 1500. Testing every combination is infeasible, so the matrix has to be sampled deliberately rather than enumerated.

This skill is both the reference (the combinatorics + strategies below) and the coverage-suite-building workflow (see Building the coverage suite); the per-SDK test mechanics live in launchdarkly-testing and openfeature-sdk-testing.

When to use

Designing the test surface for a new flag-heavy product.
Auditing existing flag-test coverage - are critical combinations covered?
PR review of a new flag - does it create a coverage gap?
Investigating an "only happens with flag X + flag Y on" incident.

How to use

Inventory the flags: list every flag, its variant count (M), and the user segments (K) it targets.
Size the problem: compute N × M × K to confirm full enumeration is infeasible and sampling is required.
Discover interactions: mark flags as inert (independent) or known-interacting (auth + permissions, billing + plan-tier), pulling from a risk register where one exists.
Pick a coverage strategy from the five below - default-only smoke, per-flag isolation, pairwise, full matrix, or risk-driven - matching the flag set's risk.
Add the special flag-state categories (kill-switch, percentage-rollout, sticky-assignment, default-on-error) as their own tests.
Layer the resulting cases across unit, integration, E2E, and production-smoke.
Hand the matrix to the platform SDK skill (launchdarkly-testing, or openfeature-sdk-testing for OpenFeature / Unleash / Flagsmith / GrowthBook) to implement each case.

The combinatorics

Variable	Typical scale
Total flags in codebase	20-500
Variants per flag	2 (most), 3-5 (experiments), 10+ (multivariate)
User segments	5-20 (free, paid, enterprise, internal, beta, etc.)
Combinatorial total	Quickly enters thousands

Insight: most flag combinations are inert (independent). Only a small subset interact - the test matrix should target interactions.

Five coverage strategies

1. Default-only smoke

Test only the default-value combination ("all flags off" or "all flags at default"). Fast but misses everything.

Use when: flag-heavy codebase where defaults change rarely.

2. Per-flag isolation

For each flag, test default + each variant in isolation. N × M tests; ignores interactions.

Use when: flags are mostly independent (UI tweaks, language strings, low-risk).

3. Pairwise interaction

Test every pair of flags (combinatorial 2-way coverage). Per NIST SP 800-142 on combinatorial testing, pairwise catches ~67% of real defects with O(N²) combinations.

Use when: flags are known-interacting (auth + permissions, billing + plan-tier).

Implementation: tools like pict (Microsoft) generate the pairwise matrix from a flag inventory.

4. Full matrix

Every combination. N^M tests for boolean flags.

Use when: small (≤10) flag count with strong interaction; financial / regulatory paths.

5. Risk-driven

Custom matrix targeting (flag, segment) cells with known risk (per a risk register per risk-matrix in the qa-process plugin).

Use when: any non-trivial codebase. Best in practice.

Building the coverage suite

The workflow that turns the strategies above into a committed matrix + test skeletons:

Inventory flags - grep for SDK calls and emit a flag inventory:

grep -rn 'isOn\|isEnabled\|variation\|getFeatureValue' --include='*.{ts,js,py,go,java}' .

flags:
  - name: show-new-ui
    platform: launchdarkly
    type: boolean
    found_at: [src/components/Header.tsx:42, src/pages/Dashboard.tsx:88]
  - name: checkout-experiment
    type: multi-variant
    variants: [control, treatment-a, treatment-b]

Classify each flag - kill-switch (naming: *-kill, disable-*, emergency-*), experiment (multi-variant + analytics), permission-gated, UI tweak, migration (use-new-*), plan/tier gating. The class picks the strategy: default-only smoke for UI tweaks, per-flag isolation for migrations, pairwise for permission/plan interactions, full matrix for kill-switches + regulatory paths, risk-driven for the rest.
Generate the matrix - for pairwise use PICT (opens in new window) (pict pict.txt > matrix.tsv emits a pairwise-covering matrix, e.g. ≤12 tests instead of 24 for full); for risk-driven, combine with the risk register (risk-matrix in the qa-process plugin) - high-impact x high-likelihood cells become required tests.
Emit a per-cell test skeleton - one describe block per flag (or flag-pair), one test per cell, with the SDK pinned via the platform's test data source:

describe('auth flag matrix', () => {
  test('free user, new auth on → new flow', () => {
    td.update(td.flag('use-new-auth').booleanFlag().on(true));
    expect(authFlow({ plan: 'free' })).toBe('new');
  });
});

Add the special-category tests regardless of matrix coverage: kill-switch deactivation latency (see references/killswitch.md), default-on-error (SDK failure returns the call-site default), and sticky-assignment (same user, same variant across evaluations).
Commit the matrix + document the gaps - flag-coverage.yaml in the repo, plus a coverage doc listing covered cells and deliberate gaps with reasons, so drift is reviewable.

Special flag-state test categories

Category	Test
Kill-switch	Setting flag → off must halt the feature within N seconds (cache TTL); full four-category treatment in references/killswitch.md
Percentage rollout	Flag at 10% → ~10% of users in 'on' bucket; SDK assignment stable per user
Targeted rollout	Targeting `region=EU` → only EU users get treatment
Sticky assignment	Same user → same variant across sessions and re-launches
Override hierarchy	User-specific override > segment override > default
Default-on-error	SDK fails / network down → default value returned
Fast-deactivate	Toggle flag off → live users see new state on next evaluation

These are per-platform behaviours (LaunchDarkly, Unleash, Flagsmith, GrowthBook implement them differently); test per platform per the SDK skills.

Flag-test layering

Tests should run at multiple layers:

Layer	Coverage
Unit	Resolver / handler logic gated on flag value (mock SDK)
Integration	SDK + handler together (test SDK against local-eval / fixture)
E2E	Real flag toggle → real user sees the change
Production smoke	After flag change → assert expected behaviour live

Flag-experiment distinction

Per ab-test-validity-checklist (in this plugin):

Flag	Experiment
Toggles behaviour	Measures outcome
Boolean / multi-variant	Multi-arm with metrics
Test the behaviour change	Test the assignment + outcome correlation
Ship decision on engineering judgment	Ship decision on statistical result

A feature flag can power an experiment (the flag is the allocation mechanism), but tests are layered: flag tests verify correct behaviour per variant; experiment tests verify assignment + analytics.

Worked example

A billing service has 3 boolean flags - new_checkout, annual_discount, tax_engine_v2 - across 3 segments (free, paid, enterprise). The team knows annual_discount and tax_engine_v2 interact: a discount changes the taxable amount.

Inventory: 3 flags, 2 variants each, 3 segments; the full flag-state-by-segment cross-product is too large to enumerate.
Interaction discovery marks the annual_discount × tax_engine_v2 pair as known-interacting; new_checkout is a UI change treated as independent.
Strategy: risk-driven. Per-flag isolation covers new_checkout; a pairwise cell covers (annual_discount on, tax_engine_v2 on) on the paid and enterprise segments where money moves.
Special categories: a kill-switch test asserts toggling tax_engine_v2 off reverts to v1 within the cache TTL; a sticky-assignment test asserts the annual_discount percentage rollout keeps each user in the same bucket across sessions.
Layering: unit tests exercise the tax resolver per flag value; an E2E test toggles tax_engine_v2 and asserts an enterprise invoice total.

Result: a handful of targeted cases instead of the full cross-product, with the discount × tax interaction covered explicitly.

Anti-patterns

Anti-pattern	Why it fails	Fix
Test only default-value path	Misses every flag-on case	Per-flag isolation minimum
Mock the SDK to return constant	Misses targeting / rollout logic	Local-eval mode or fixture-based SDK
Same test for every flag combination	Slow; flaky; opaque failures	Per-combination assertion logs
No kill-switch test	Production incident has no rehearsed response	Test deactivation latency
Don't test percentage-rollout sticky-assignment	Rollout produces non-deterministic UX	Per `ab-test-validity-checklist`
Tests assume flag-on default	Real default-off behaviour untested in CI	Test both paths
No cleanup test for removed flags	Stale flags accumulate	Periodic audit via the `stale-flag-detector` agent
Pairwise without flag-interaction discovery	Some pairs spuriously interact	Couple with risk-register input

Limitations

Pairwise misses 3-way+ interactions. Some real-world bugs need 3-way coverage.
Real-world matrices have ordering effects. Flag A enabled THEN flag B may differ from B then A; test ordering needs separate coverage.
Coverage tooling lags. PICT / ACTS exist but integration with flag platforms is bespoke.
Stale flags pollute the matrix. Cleanup pairs with the stale-flag-detector agent's removal runbook.

References

LaunchDarkly flag-testing strategy: launchdarkly.com/blog/ (opens in new window).
NIST SP 800-142 (combinatorial testing): csrc.nist.gov/publications/detail/sp/800-142/final (opens in new window).
GrowthBook test docs: docs.growthbook.io/lib/node (opens in new window).
Kill-switch test categories: references/killswitch.md.
Consumed by: launchdarkly-testing, openfeature-sdk-testing.
Removal side: stale-flag-detector agent.
Cross-plugin: ab-test-validity-checklist, feature-flag-test-harness.

Kill-switch (ops-toggle) test authoring

View source

Kill-switch (ops-toggle) test authoring

The four test categories specific to kill-switch flags: switch-OFF graceful degradation, fail-static default when the flag service is unreachable, latency budget for the kill decision, and no-data-corruption mid-flight.

A kill-switch flag (also called an ops toggle) is a long-lived feature flag whose purpose is to let operators immediately disable functionality in production during an incident, per Martin Fowler's feature-toggle taxonomy at martinfowler.com/articles/feature-toggles.html (opens in new window). Fowler identifies them as "manually-managed circuit breakers" that must be reconfigurable without a deployment.

Kill-switch flags share the same SDK infrastructure as other flags but carry distinct testing obligations because they are incident-response tools, not just delivery mechanisms.

Author these when: a new kill-switch flag is introduced (naming signals: disable-*, emergency-*, *-kill, circuit-*, shutdown-*); a production incident exposed that a kill-switch was flipped but the feature did not degrade cleanly; a high-traffic launch pre-positions kill-switches; or a PR touches a kill-switch-gated code path.

Category 1 - Switch-OFF path degrades gracefully

The feature code must handle the off variant without throwing, panicking, or producing a broken UI state.

// LaunchDarkly TestData source - from launchdarkly.com/docs/sdk/features/test-data-sources:
// "The test data source allows you to mock the behavior of a LaunchDarkly
//  SDK so it has predictable behavior when evaluating flags."
import { TestData } from '@launchdarkly/node-server-sdk';

describe('checkout-kill-switch: switch-OFF', () => {
  let td: TestData;

  beforeEach(() => {
    td = TestData.dataSource();
    // Flag starts ON (normal production state)
    td.update(td.flag('checkout-kill').boolVariation(true));
  });

  test('flag OFF - checkout shows maintenance message, not an error', async () => {
    td.update(td.flag('checkout-kill').boolVariation(false));
    const result = await renderCheckout({ flagClient: clientWith(td) });
    expect(result.status).toBe('degraded');
    expect(result.userMessage).toMatch(/temporarily unavailable/i);
    expect(result.errorThrown).toBe(false);
  });

  test('flag OFF - degraded path does not call payment provider', async () => {
    td.update(td.flag('checkout-kill').boolVariation(false));
    const paymentSpy = jest.spyOn(paymentProvider, 'charge');
    await renderCheckout({ flagClient: clientWith(td) });
    expect(paymentSpy).not.toHaveBeenCalled();
  });
});

Document what the degraded state must and must not do; include it as a code comment or test description so the test doubles as operational runbook.

Category 2 - Fail-static default when the flag service is unreachable

Per the OpenFeature specification at openfeature.dev/docs/reference/concepts/evaluation-api (opens in new window): "In the case of any error during flag evaluation, the default value will be returned, so give consideration to your default values!"

The LaunchDarkly SDK aligns: "The fallback value is defined in your code... and is only returned if an error occurs" including "LaunchDarkly service is unreachable" (source: launchdarkly.com/docs/sdk/features/evaluating (opens in new window)).

The test must assert two things: the SDK returns the correct default, and the application continues operating with that default rather than throwing.

import { OpenFeature, InMemoryProvider } from '@openfeature/server-sdk';

describe('checkout-kill-switch: fail-static default', () => {
  test('provider unavailable - default OFF is served, app does not crash', async () => {
    // Register a provider that always throws
    await OpenFeature.setProviderAndWait(new AlwaysErrorProvider());
    const client = OpenFeature.getClient();

    // Default is false (feature disabled) - the safe side for a kill-switch
    const value = await client.getBooleanValue('checkout-kill', false);
    expect(value).toBe(false);

    // Application layer must handle the default without throwing
    await expect(renderCheckout({ featureEnabled: value })).resolves.not.toThrow();
  });

  test('provider unavailable - error hook fires and is logged', async () => {
    const errorEvents: string[] = [];
    OpenFeature.addHooks({
      error: (_ctx, err) => { errorEvents.push(err.message); }
    });
    await OpenFeature.setProviderAndWait(new AlwaysErrorProvider());
    const client = OpenFeature.getClient();
    await client.getBooleanValue('checkout-kill', false);
    expect(errorEvents.length).toBeGreaterThan(0);
  });
});

The default value for a kill-switch MUST be the safe side: false for a flag that enables a feature (disable it on error) or true for a flag that disables a feature (keep it disabled on error). Document the chosen default and its rationale as a comment at the flag call site.

Category 3 - Latency budget for the kill decision

A kill-switch flipped in the operator console must reach running processes within an acceptable window. The window depends on the streaming/polling configuration of the SDK.

Per LaunchDarkly's documentation at launchdarkly.com/docs/sdk/concepts/client-side-server-side (opens in new window): "Server-side SDKs open a streaming connection to LaunchDarkly and receive flag configuration changes over the stream." The cached values have "no expiration or time-to-live (TTL) value" - propagation speed depends on the streaming connection, not a TTL. Client-side SDKs that use polling have an interval-bound lag.

Because in-process latency tests against a live SDK are environment-dependent and slow, the recommended approach is two tests:

A unit test that asserts the flag-evaluation result changes synchronously when the in-memory test source is updated (validates that the application re-reads the flag on each request rather than caching it locally).
An integration test comment documenting the expected propagation window for your SDK configuration, to be verified in a staging smoke test.

describe('checkout-kill-switch: kill latency', () => {
  test('flag re-evaluated per request - not cached application-side', async () => {
    const td = TestData.dataSource();
    td.update(td.flag('checkout-kill').boolVariation(true));
    const client = clientWith(td);

    const before = await isCheckoutEnabled(client);
    expect(before).toBe(true);

    // Flip the kill-switch
    td.update(td.flag('checkout-kill').boolVariation(false));

    // Next evaluation reflects the flip without a process restart
    const after = await isCheckoutEnabled(client);
    expect(after).toBe(false);
  });

  // Integration note: with LaunchDarkly streaming (server SDK default),
  // flag changes propagate in near-real-time over the SSE stream.
  // With Unleash polling (default 15s interval per unleash.io/docs),
  // the worst-case lag equals the polling interval.
  // Agree on the acceptable window with SRE and add a staging smoke test.
});

If the application caches the flag value (e.g., in a request-scoped singleton), this test will catch it.

Category 4 - No data corruption mid-flight

When the kill-switch flips while an operation is already in progress (a multi-step transaction, a streaming response, a batch job), the in-progress operation must complete cleanly or roll back - it must not leave partial state.

This is the most scenario-specific of the four categories. The general pattern is to snapshot the flag value at the start of the operation and hold it for the operation's duration rather than re-evaluating mid-operation.

describe('checkout-kill-switch: mid-flight flip', () => {
  test('in-progress order is not corrupted when kill-switch flips mid-checkout', async () => {
    const td = TestData.dataSource();
    td.update(td.flag('checkout-kill').boolVariation(true));
    const client = clientWith(td);

    // Begin a multi-step checkout; flip the flag after step 1
    const order = await startCheckout(client);  // Step 1: reserve inventory

    // Simulate operator flipping the kill-switch during step 2
    td.update(td.flag('checkout-kill').boolVariation(false));

    const result = await completeCheckout(order);  // Step 2: charge + confirm

    // Either fully committed or fully rolled back - never partial
    expect(['committed', 'rolled_back']).toContain(result.state);
    if (result.state === 'rolled_back') {
      // Inventory reservation must be released
      expect(await inventoryReserved(order.itemId)).toBe(false);
    }
  });
});

Document the consistency contract in a test description or comment: what "no partial state" means for this specific operation.

Anti-patterns

Anti-pattern	Why it fails	Fix
Test only the ON path	OFF path is untested; incident reveals broken degradation	Category 1 test
Default value is ON (feature enabled)	SDK failure enables a feature that should be disabled	Default must be the safe-off side
Application caches the flag value	Kill-switch flip takes minutes not seconds	Re-evaluate per request; Category 3 test catches it
Mid-flight test omitted	Flip during a transaction causes partial writes	Category 4 test for any stateful operation
Rely on live SDK in unit tests	Flaky; requires network	Use TestData (LD) or InMemory provider (OpenFeature)
No error-hook assertion	Unreachable SDK is silent; ops loses visibility	Assert error hook fires in Category 2 test

Limitations

Category 3 (latency) cannot be fully tested without a real SDK connection. The unit test verifies application-side re-evaluation; the propagation window (streaming lag or poll interval) requires a staging integration test.
Mid-flight semantics are application-specific. Category 4 must be authored per operation; no generic skeleton fits all cases.
Multi-process deployments. Each server process holds an independent SDK connection. The kill propagates to each independently; the worst-case latency is per-process.

Sources

Kill-switch / ops-toggle definition and taxonomy: Pete Hodgson, "Feature Toggles (aka Feature Flags)": martinfowler.com/articles/feature-toggles.html (opens in new window).
Fail-static default value behavior (OpenFeature): openfeature.dev/docs/reference/concepts/evaluation-api (opens in new window).
Fallback values on SDK error or service unreachability (LaunchDarkly): launchdarkly.com/docs/sdk/features/evaluating (opens in new window).
LaunchDarkly TestData mock source for SDK-level tests: launchdarkly.com/docs/sdk/features/test-data-sources (opens in new window).
Streaming vs polling propagation speed and in-memory cache behavior (LaunchDarkly): launchdarkly.com/docs/sdk/concepts/client-side-server-side (opens in new window).
Circuit breaker as automated kill-switch pattern: Martin Fowler, "CircuitBreaker": martinfowler.com/bliki/CircuitBreaker.html (opens in new window).

Related skills

ab-test-validity-checklist

Workflow skill that builds an A/B-test validity checklist from an experiment proposal, walking the canonical design-correctness gates - pre-registered OEC/power/guardrails, randomization unit + SRM check, assignment integrity, telemetry, peeking discipline, novelty/primacy, post-experiment SRM re-check - into a per-experiment checklist + sign-off form. Use when launching, auditing, or governing an experiment. For pitfall mechanics (guardrails, peeking) see experiment-results-interpreter's references; to read an already-valid result use experiment-results-interpreter; for per-SDK harness tests use experiment-sdk-testing - this gates DESIGN, not SDK code.

experiment-results-interpreter

Interprets the results of a valid online controlled experiment, one whose harness, SRM, and telemetry have already been confirmed. Covers the distinction between practical and statistical significance, reading confidence intervals instead of binary p-values, novelty and primacy week-over-week decay that causes post-ship reversion, interaction effects from concurrent experiments, Simpson's paradox in segmented results, and the ordered guardrail-check sequence required before a ship decision - with the deep methodology in references/: the peeking problem and its corrections (fixed-horizon, alpha-spending, always-valid mSPRT) in references/peeking.md, and guardrail-metric methodology (taxonomy, OEC relationship, pre-commitment, thresholds) in references/guardrails.md. Use when a data scientist or PM is ready to draw conclusions from an experiment, when designing a stop-early policy, or when declaring an experiment's guardrail set. Distinct from ab-test-validity-checklist (harness setup and SRM detection).

experiment-sdk-testing

Umbrella for experimentation-SDK test harnesses: the shared offline-datafile / hermetic-init pattern (commit a point-in-time flag/experiment config fixture, initialize the SDK with no network, pin arms per test, assert assignment integrity), with per-vendor references for Statsig (localMode + overrideGate), Optimizely (datafile + forced decisions), Split.io / Harness FME (localhost mode + features map or YAML fixture), Amplitude Experiment (local evaluation + bootstrap), and VWO (settings file + deterministic bucketing). Use when writing tests for application code instrumented with any of these five experimentation SDKs; for experiment DESIGN gates use ab-test-validity-checklist, and to read results use experiment-results-interpreter.

launchdarkly-testing

Wraps LaunchDarkly server-side SDK testing patterns: TestData data source for hermetic tests (no network), file-based data source for fixture-driven tests, flag override patterns (TestData.update for per-test flag values), and assignment-integrity tests. Use when writing tests for code that uses LaunchDarkly flags; to decide which flag combinations those tests should cover in the first place, see feature-flag-test-matrix-reference.

openfeature-sdk-testing

Wraps OpenFeature (CNCF vendor-neutral SDK abstraction) testing patterns: the InMemoryProvider for hermetic tests without network calls, provider registration via OpenFeature.setProvider, the getBooleanValue/getBooleanDetails evaluation API with EvaluationDetails (value, variant, reason, errorCode), hooks for evaluation side-effects, and evaluation context for targeting-rule tests. Covers TypeScript, Java, and Python SDKs, plus per-vendor hermetic-bootstrap references for Unleash (bootstrap toggles), Flagsmith (offline LocalFileHandler), and GrowthBook (initSync payload). Use when writing tests for code that resolves feature flags through the OpenFeature SDK or the Unleash / Flagsmith / GrowthBook native SDKs; LaunchDarkly has its own skill (launchdarkly-testing).