feature-flag-test-harness

Overview

A test that hits the production flag service is non-deterministic by definition - the answer depends on whoever toggled the flag last. And a test that asks "did we test the feature with the flag off?" needs both runs side by side.

This skill builds a harness that:

1. Replaces the production OpenFeature provider with an in-memory provider the test owns (openfeature-providers (opens in new window)).
2. Enumerates the relevant flag combinations (not the full 2^N cartesian product - the long tail isn't worth running).
3. Runs the suite once per combination, as a separate CI shard.
4. Aggregates results into a matrix the reviewer can read.

The skill's reference architecture targets OpenFeature because it standardizes the SDK across LaunchDarkly, Flagsmith, ConfigCat, self-hosted, etc. - the harness works identically against any provider (openfeature-overview (opens in new window)).

When to use

• A new feature lives behind a release toggle and the team needs to verify both code paths (toggle off = old behavior unchanged; toggle on = new behavior correct).
• An experiment toggle has multiple variants (A / B / control) and each needs an integration-test pass.
• An ops toggle (kill switch / degradation flag) needs a "service fails open" pass.
• The flag set already exists in OpenFeature, LaunchDarkly, Flagsmith, ConfigCat, GrowthBook, or any SDK with an in-memory / test provider.

If the team has only one or two flags and a flat "always on for test" config works, this skill is overkill - set the test environment's flag values once in setup and stop there.

How to use

1. Classify each flag by Hodgson category (Step 1) and mark which flag pairs actually interact.
2. Enumerate the minimum cover - single-flag variants plus the declared interaction tuples, never the full 2^N product.
3. Wire the OpenFeature in-memory provider so the suite never hits the production flag service - see references/provider-wiring.md.
4. Generate the combination matrix and run one CI shard per combination - see references/matrix-and-ci.md.
5. Keep fail-fast: false on the matrix so every failing combination surfaces in one run, not just the first.
6. Aggregate each shard's JUnit XML into a per-combination pass/fail matrix the reviewer reads at a glance.
7. Split cadence: PR runs only combinations whose flags touch changed files; nightly runs the full matrix.

Step 1 - Classify each flag (Hodgson taxonomy)

Per feature-toggles (opens in new window), flags fall into four categories with different test needs:

For experiment and permissioning toggles, the harness simulates each cohort by seeding the EvaluationContext (feature-toggles (opens in new window)).

Don't run all 2^N combinations. Author marks the interactions worth testing:

# tests/flag-matrix.yaml
flags:
  new_checkout:        { kind: release,  test: [off, on] }
  promo_codes:         { kind: release,  test: [off, on] }
  ranking_experiment:  { kind: experiment, variants: [control, treatment_a, treatment_b] }
  payment_kill_switch: { kind: ops,      test: [on, off] }   # off = degraded

interactions:
  # The author asserts these flag pairs interact; run their combinations explicitly.
  - [new_checkout, promo_codes]
  # Ranking experiment doesn't interact with checkout; don't bloat the matrix.

The harness enumerates every flag's variants individually plus the listed interaction tuples, never the full 2^N product. See the Worked example for the run count this schema yields.

Provider wiring, matrix generation, CI

Wire the in-memory provider so the suite reads flag values from FLAGS_JSON instead of the production service (Node shown; Python and Java variants in the reference):

// tests/harness/flag-harness.ts
import { OpenFeature, InMemoryProvider } from '@openfeature/server-sdk';

export function withFlags(flags: Record<string, unknown>) {
  return OpenFeature.setProviderAndWait(new InMemoryProvider(
    Object.fromEntries(Object.entries(flags).map(([k, v]) => [k, {
      defaultVariant: 'configured', variants: { configured: v }, disabled: false,
    }])),
  ));
}
// beforeAll(() => withFlags(JSON.parse(process.env.FLAGS_JSON || '{}')));

Generate one shard per combination, then feed the JSON to the CI matrix:

python scripts/gen-flag-matrix.py tests/flag-matrix.yaml   # -> JSON array of {name, flags}

• Full provider wiring (Node / Python / Java), evaluation API, and default-value behavior: references/provider-wiring.md.
• The gen-flag-matrix.py source, CI matrix YAML, result aggregation, and the PR / nightly cadence split: references/matrix-and-ci.md.

Worked example

A checkout team ships new_checkout (release) and promo_codes (release) behind flags, runs a ranking_experiment (control / treatment_a / treatment_b), and guards payments with a payment_kill_switch (ops). They author tests/flag-matrix.yaml exactly as in Step 1 and declare the one interaction that matters: [new_checkout, promo_codes].

gen-flag-matrix.py enumerates 9 single-flag runs (2 + 2 + 3 + 2) plus 4 interaction runs (new_checkout × promo_codes) = 13 shards, not the 24 of the full 2^N product. Each shard boots the suite with the in-memory provider pinned to that combination via FLAGS_JSON, so no run touches the production flag service.

CI runs all 13 shards in parallel with fail-fast: false. The aggregated matrix shows promo_codes=on red at checkout.spec.ts:42 and ranking_experiment=treatment_b red at cart.spec.ts:18, while the baseline and every other combination pass. The team reads two flag-specific failures in one glance instead of re-running to find the second.

Anti-patterns and limitations

The full anti-pattern table (2^N blowup, hitting the prod provider, asserting flag value instead of behavior, fail-fast: true, missing baseline row) and the harness's limitations (no targeting rules, author-declared interactions, per-request dynamism, matrix-size bound) are in references/anti-patterns-and-limits.md.

References

• openfeature-overview (opens in new window) - OpenFeature SDK + provider model.
• openfeature-providers (opens in new window) - Provider interface, in-memory test provider.
• openfeature-eval (opens in new window) - getBooleanValue / getStringValue / getNumberValue / getObjectValue signatures + default-value behavior.
• feature-toggles (opens in new window) - Hodgson's taxonomy: release / experiment / ops / permissioning; longevity vs dynamism.
• testcontainers, docker-compose-tests - the surrounding stack the harness drives per combination.