openfeature-sdk-testing

Overview

A codebase with N feature flags, each having M variants, and users in K segments, has N × M × K possible flag-state-segment combinations. At realistic numbers (50 flags, 2 variants each, 5 segments) that's 500 - and at 50 flags with 3 variants and 10 segments, it's 1500. Testing every combination is infeasible, so the matrix has to be sampled deliberately rather than enumerated.

This file is both the reference (the combinatorics + strategies below) and the coverage-suite-building workflow (see Building the coverage suite); the per-SDK test mechanics live in launchdarkly-testing and in the main openfeature-sdk-testing skill.

When to read this

• Designing the test surface for a new flag-heavy product.
• Auditing existing flag-test coverage - are critical combinations covered?
• PR review of a new flag - does it create a coverage gap?
• Investigating an "only happens with flag X + flag Y on" incident.

How to use

1. Inventory the flags: list every flag, its variant count (M), and the user segments (K) it targets.
2. Size the problem: compute N × M × K to confirm full enumeration is infeasible and sampling is required.
3. Discover interactions: mark flags as inert (independent) or known-interacting (auth + permissions, billing + plan-tier), pulling from a risk register where one exists.
4. Pick a coverage strategy from the five below - default-only smoke, per-flag isolation, pairwise, full matrix, or risk-driven - matching the flag set's risk.
5. Add the special flag-state categories (kill-switch, percentage-rollout, sticky-assignment, default-on-error) as their own tests.
6. Layer the resulting cases across unit, integration, E2E, and production-smoke.
7. Hand the matrix to the platform SDK skill (launchdarkly-testing, or openfeature-sdk-testing for OpenFeature / Unleash / Flagsmith / GrowthBook) to implement each case.

The combinatorics

Insight: most flag combinations are inert (independent). Only a small subset interact - the test matrix should target interactions.

Five coverage strategies

1. Default-only smoke

Test only the default-value combination ("all flags off" or "all flags at default"). Fast but misses everything.

Use when: flag-heavy codebase where defaults change rarely.

2. Per-flag isolation

For each flag, test default + each variant in isolation. N × M tests; ignores interactions.

Use when: flags are mostly independent (UI tweaks, language strings, low-risk).

3. Pairwise interaction

Test every pair of flags (combinatorial 2-way coverage). Per NIST SP 800-142 on combinatorial testing, pairwise catches ~67% of real defects with O(N²) combinations.

Use when: flags are known-interacting (auth + permissions, billing + plan-tier).

Implementation: tools like pict (Microsoft) generate the pairwise matrix from a flag inventory.

4. Full matrix

Every combination. N^M tests for boolean flags.

Use when: small (≤10) flag count with strong interaction; financial / regulatory paths.

5. Risk-driven

Custom matrix targeting (flag, segment) cells with known risk (per a risk register per risk-matrix in the qa-process plugin).

Use when: any non-trivial codebase. Best in practice.

Building the coverage suite

The workflow that turns the strategies above into a committed matrix + test skeletons:

1. Inventory flags - grep for SDK calls and emit a flag inventory:

grep -rn 'isOn\|isEnabled\|variation\|getFeatureValue' --include='*.{ts,js,py,go,java}' .

flags:
  - name: show-new-ui
    platform: launchdarkly
    type: boolean
    found_at: [src/components/Header.tsx:42, src/pages/Dashboard.tsx:88]
  - name: checkout-experiment
    type: multi-variant
    variants: [control, treatment-a, treatment-b]

1. Classify each flag - kill-switch (naming: *-kill, disable-*, emergency-*), experiment (multi-variant + analytics), permission-gated, UI tweak, migration (use-new-*), plan/tier gating. The class picks the strategy: default-only smoke for UI tweaks, per-flag isolation for migrations, pairwise for permission/plan interactions, full matrix for kill-switches + regulatory paths, risk-driven for the rest.
2. Generate the matrix - for pairwise use PICT (opens in new window) (pict pict.txt > matrix.tsv emits a pairwise-covering matrix, e.g. ≤12 tests instead of 24 for full); for risk-driven, combine with the risk register (risk-matrix in the qa-process plugin) - high-impact x high-likelihood cells become required tests.
3. Emit a per-cell test skeleton - one describe block per flag (or flag-pair), one test per cell, with the SDK pinned via the platform's test data source:

describe('auth flag matrix', () => {
  test('free user, new auth on → new flow', () => {
    td.update(td.flag('use-new-auth').booleanFlag().on(true));
    expect(authFlow({ plan: 'free' })).toBe('new');
  });
});

1. Add the special-category tests regardless of matrix coverage: kill-switch deactivation latency (see killswitch.md (opens in new window)), default-on-error (SDK failure returns the call-site default), and sticky-assignment (same user, same variant across evaluations).
2. Commit the matrix + document the gaps - flag-coverage.yaml in the repo, plus a coverage doc listing covered cells and deliberate gaps with reasons, so drift is reviewable.

Special flag-state test categories

These are per-platform behaviours (LaunchDarkly, Unleash, Flagsmith, GrowthBook implement them differently); test per platform per the SDK skills.

Flag-test layering

Tests should run at multiple layers:

Flag-experiment distinction

Per ab-test-validity-checklist (in this plugin):

A feature flag can power an experiment (the flag is the allocation mechanism), but tests are layered: flag tests verify correct behaviour per variant; experiment tests verify assignment + analytics.

Worked example

A billing service has 3 boolean flags - new_checkout, annual_discount, tax_engine_v2 - across 3 segments (free, paid, enterprise). The team knows annual_discount and tax_engine_v2 interact: a discount changes the taxable amount.

1. Inventory: 3 flags, 2 variants each, 3 segments; the full flag-state-by-segment cross-product is too large to enumerate.
2. Interaction discovery marks the annual_discount × tax_engine_v2 pair as known-interacting; new_checkout is a UI change treated as independent.
3. Strategy: risk-driven. Per-flag isolation covers new_checkout; a pairwise cell covers (annual_discount on, tax_engine_v2 on) on the paid and enterprise segments where money moves.
4. Special categories: a kill-switch test asserts toggling tax_engine_v2 off reverts to v1 within the cache TTL; a sticky-assignment test asserts the annual_discount percentage rollout keeps each user in the same bucket across sessions.
5. Layering: unit tests exercise the tax resolver per flag value; an E2E test toggles tax_engine_v2 and asserts an enterprise invoice total.

Result: a handful of targeted cases instead of the full cross-product, with the discount × tax interaction covered explicitly.

Anti-patterns

Limitations

• Pairwise misses 3-way+ interactions. Some real-world bugs need 3-way coverage.
• Real-world matrices have ordering effects. Flag A enabled THEN flag B may differ from B then A; test ordering needs separate coverage.
• Coverage tooling lags. PICT / ACTS exist but integration with flag platforms is bespoke.
• Stale flags pollute the matrix. Cleanup pairs with the stale-flag-detector agent's removal runbook.

References

• LaunchDarkly flag-testing strategy: launchdarkly.com/blog/ (opens in new window).
• NIST SP 800-142 (combinatorial testing): csrc.nist.gov/publications/detail/sp/800-142/final (opens in new window).
• GrowthBook test docs: docs.growthbook.io/lib/node (opens in new window).
• Kill-switch test categories: killswitch.md (opens in new window).
• Used by: launchdarkly-testing and the main skill.
• Removal side: stale-flag-detector agent.
• Cross-plugin: ab-test-validity-checklist, feature-flag-test-harness.

Deeper detail for flagsmith.md (opens in new window). Offline mode (covered in the skill spine) is the default for tests; the modes below are secondary.

Local-evaluation mode

Polls the Flagsmith API periodically and evaluates flags locally between refreshes (no per-request network, but not zero network):

flagsmith = Flagsmith(
    environment_key="server-key",
    enable_local_evaluation=True,
    environment_refresh_interval_seconds=60,
)

Local mode polls; offline mode does not. For tests, offline is usually preferred.

default_flag_handler - per-flag mock

Programmatic fallback used when the offline environment.json does not have the flag under test yet:

from flagsmith import Flagsmith
from flagsmith.models import DefaultFlag

def default_flag_handler(feature_name: str) -> DefaultFlag:
    if feature_name == "secret_button":
        return DefaultFlag(enabled=False, value='{"colour": "#b8b8b8"}')
    return DefaultFlag(enabled=False, value=None)

flagsmith = Flagsmith(
    environment_key="test-key",
    default_flag_handler=default_flag_handler,
)

Useful when the offline environment.json does not have the flag you are testing yet.

Anti-patterns

Limitations

• environment.json is point-in-time. Drift invisible.
• default_flag_handler only fires for missing flags in local-eval mode. In offline mode it can be used for fallback.
• No granular per-user override API. Use traits + segments via the environment.json.
• Doesn't validate Flagsmith's own logic. Platform-side evaluation is separate.

Wraps Flagsmith server-side SDK testing patterns so tests run without calling the Flagsmith API: offline mode with LocalFileHandler + a downloaded environment.json snapshot, local-evaluation mode (no per-request network), and default_flag_handler for per-feature mocked fallbacks, via the get_environment_flags / get_identity_flags evaluation paths.

Flagsmith (open-source, also SaaS at flagsmith.com) supports three test-friendly modes per docs.flagsmith.com/clients/server-side (opens in new window):

1. Offline mode with LocalFileHandler - loads a downloaded environment.json snapshot; zero network.
2. Local-evaluation mode - fetches environment + flags periodically, evaluates locally without per-request network.
3. Default flag handler - programmatic fallback for any flag (mock-flag-only mode).

Offline mode is the default choice for tests; local-evaluation and default_flag_handler detail live in flagsmith-modes.md (opens in new window).

How to use

1. Install the SDK: pip install flagsmith (Python) or npm install --save-dev flagsmith-nodejs (Node).
2. Download the environment snapshot with the Flagsmith CLI (command below) and commit it as a test fixture.
3. Construct the client in offline mode with LocalFileHandler pointed at that fixture, so tests make zero network calls.
4. For flags not yet in the snapshot, register a default_flag_handler returning a DefaultFlag with the value under test (see flagsmith-modes.md (opens in new window)).
5. Evaluate via get_environment_flags() for environment-scoped flags, or get_identity_flags(identifier, traits=...) for per-user flags.
6. Assert on is_feature_enabled(...) and get_feature_value(...), and add an integrity test that the same identity resolves consistently.
7. Verify: with offline_mode=True and no environment_key set, get_environment_flags() must return the fixture's flags with zero network calls (run the suite with the machine offline or under a network-blocking fixture to confirm). If it raises or attempts an API call, the LocalFileHandler environment_document_path is wrong or offline_mode is unset - fix the path and re-run.
8. Wire pytest tests/flagsmith/ into CI - offline mode needs no env vars.

Offline mode with LocalFileHandler

Per docs.flagsmith.com (opens in new window):

from flagsmith import Flagsmith
from flagsmith.offline_handlers import LocalFileHandler

local_file_handler = LocalFileHandler(environment_document_path="tests/fixtures/flagsmith-environment.json")

flagsmith = Flagsmith(offline_mode=True, offline_handler=local_file_handler)

Download the environment.json via the Flagsmith CLI, then commit it (refresh deliberately):

flagsmith environment-document --api-key=<server-key> --output=tests/fixtures/flagsmith-environment.json

Evaluate flags

def test_environment_flag():
    flags = flagsmith.get_environment_flags()
    assert flags.is_feature_enabled("secret_button") is False
    assert flags.get_feature_value("secret_button") == '{"colour": "#b8b8b8"}'

def test_identity_flag():
    flags = flagsmith.get_identity_flags(
        identifier="user@example.com",
        traits={"plan": "premium"},
    )
    assert flags.is_feature_enabled("premium_feature") is True

Integrity test - the same identity must resolve consistently:

def test_same_identity_consistent():
    f1 = flagsmith.get_identity_flags("u1")
    f2 = flagsmith.get_identity_flags("u1")
    assert f1.is_feature_enabled("flag-x") == f2.is_feature_enabled("flag-x")

CI integration

No env vars needed in offline mode:

jobs:
  flagsmith-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-python@v5
      - run: pip install flagsmith
      - run: pytest tests/flagsmith/

Worked example

A service reads a secret_button flag (boolean plus a JSON colour value) and a premium_feature flag gated on a plan trait, tested fully offline:

1. Download the environment document and commit it (step 2 above).
2. Build the offline client with LocalFileHandler (offline-mode snippet above).
3. secret_button is not in the snapshot yet, so register a default_flag_handler for it (see flagsmith-modes.md (opens in new window)).
4. Assert on get_environment_flags() and get_identity_flags(...) (evaluate snippet above).

Result: pytest tests/flagsmith/ runs green in CI with zero network access and no analytics pollution.

Sources

• Flagsmith server-side docs: docs.flagsmith.com/clients/server-side (opens in new window).
• Local-evaluation mode, default_flag_handler, anti-patterns, and limitations: flagsmith-modes.md (opens in new window).

Wraps GrowthBook Node SDK testing patterns: GrowthBookClient initialization with direct payload (initSync; no network), isOn / getFeatureValue / evalFeature, scoped instances (createScopedInstance) for per-request user context, inline experiment (runInlineExperiment) tests, and tracking-callback assertion patterns.

Per docs.growthbook.io/lib/node (opens in new window), the GrowthBook Node SDK's GrowthBookClient supports an initSync({ payload }) pattern that fully bypasses the network - pass the feature definitions directly. createScopedInstance lets each test (or request) bind its own user context cleanly.

Install

npm install --save-dev @growthbook/growthbook

Initialize with payload (offline)

Per docs.growthbook.io (opens in new window):

import { GrowthBookClient } from '@growthbook/growthbook';

const gbClient = new GrowthBookClient().initSync({
  payload: {
    features: {
      'show-new-ui': { defaultValue: true },
      'experiment-x': {
        defaultValue: false,
        rules: [{ condition: { id: 'test-user-1' }, force: true }],
      },
    },
  },
});

initSync is purpose-built for tests - no async wait.

isOn / getFeatureValue

test('flag on', () => {
  const userContext = { attributes: { id: 'user-1' } };
  expect(gbClient.isOn('show-new-ui', userContext)).toBe(true);
});

test('typed feature value', () => {
  const userContext = { attributes: { id: 'user-1' } };
  const color = gbClient.getFeatureValue('button-color', 'blue', userContext);
  expect(['blue', 'red', 'green']).toContain(color);
});

Scoped instance per test

test('user-specific evaluation', () => {
  const instance = gbClient.createScopedInstance({
    attributes: { id: 'user-1', plan: 'premium' },
  });
  expect(instance.isOn('premium-feature')).toBe(true);
});

Avoids passing context everywhere.

Inline experiments

test('inline experiment returns one variant', () => {
  const userContext = { attributes: { id: 'user-1' } };
  const { value } = gbClient.runInlineExperiment({
    key: 'my-experiment',
    variations: ['red', 'blue', 'green'],
    coverage: 1.0,
    weights: [0.33, 0.34, 0.33],
  }, userContext);
  expect(['red', 'blue', 'green']).toContain(value);
});

Tracking-callback assertions

test('experiment tracking fires', () => {
  const tracked: any[] = [];
  const client = new GrowthBookClient({
    trackingCallback: (exp, result, ctx) => {
      tracked.push({ key: exp.key, variation: result.key });
    },
  }).initSync({ payload: {} });

  client.runInlineExperiment(
    { key: 'exp-x', variations: [0, 1] },
    { attributes: { id: 'user-1' } }
  );

  expect(tracked).toHaveLength(1);
  expect(tracked[0].key).toBe('exp-x');
});

Feature-usage callback

const evaluated: any[] = [];

test('feature usage logged', () => {
  const userContext = {
    attributes: { id: 'user-1' },
    onFeatureUsage: (key: string, result: any) => {
      evaluated.push({ key, value: result.value });
    },
  };

  gbClient.evalFeature('feature-x', userContext);
  expect(evaluated).toContainEqual(expect.objectContaining({ key: 'feature-x' }));
});

TypeScript-strict feature contract

interface AppFeatures {
  'button-color': string;
  'font-size': number;
  'newForm': boolean;
}

const gbClient = new GrowthBookClient<AppFeatures>({}).initSync({ payload: {} });

const color = gbClient.getFeatureValue('button-color', 'blue', ctx);  // typed
// gbClient.isOn('buton-color', ctx);  // typo → compile error

CI integration

Fully offline; no GrowthBook key needed:

jobs:
  growthbook-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v4
      - run: npm ci && npm test

Anti-patterns

Limitations

• Payload is point-in-time. Drift from GrowthBook UI invisible.
• No targeting-rule-test helper beyond writing conditions. Complex targeting tests need many fixture variants.
• Tracking-callback is synchronous. Async exposure logging needs a separate test.
• TypeScript strict-mode required for the typed feature contract.

Sources

• GrowthBook Node SDK: docs.growthbook.io/lib/node (opens in new window).
• Inline experiments: docs.growthbook.io/lib/node#inline-experiments (opens in new window).

The four test categories specific to kill-switch flags: switch-OFF graceful degradation, fail-static default when the flag service is unreachable, latency budget for the kill decision, and no-data-corruption mid-flight.

A kill-switch flag (also called an ops toggle) is a long-lived feature flag whose purpose is to let operators immediately disable functionality in production during an incident, per Martin Fowler's feature-toggle taxonomy at martinfowler.com/articles/feature-toggles.html (opens in new window). Fowler identifies them as "manually-managed circuit breakers" that must be reconfigurable without a deployment.

Kill-switch flags share the same SDK infrastructure as other flags but carry distinct testing obligations because they are incident-response tools, not just delivery mechanisms.

Author these when: a new kill-switch flag is introduced (naming signals: disable-*, emergency-*, *-kill, circuit-*, shutdown-*); a production incident exposed that a kill-switch was flipped but the feature did not degrade cleanly; a high-traffic launch pre-positions kill-switches; or a PR touches a kill-switch-gated code path.

Category 1 - Switch-OFF path degrades gracefully

The feature code must handle the off variant without throwing, panicking, or producing a broken UI state.

// LaunchDarkly TestData source - from launchdarkly.com/docs/sdk/features/test-data-sources:
// "The test data source allows you to mock the behavior of a LaunchDarkly
//  SDK so it has predictable behavior when evaluating flags."
import { TestData } from '@launchdarkly/node-server-sdk';

describe('checkout-kill-switch: switch-OFF', () => {
  let td: TestData;

  beforeEach(() => {
    td = TestData.dataSource();
    // Flag starts ON (normal production state)
    td.update(td.flag('checkout-kill').boolVariation(true));
  });

  test('flag OFF - checkout shows maintenance message, not an error', async () => {
    td.update(td.flag('checkout-kill').boolVariation(false));
    const result = await renderCheckout({ flagClient: clientWith(td) });
    expect(result.status).toBe('degraded');
    expect(result.userMessage).toMatch(/temporarily unavailable/i);
    expect(result.errorThrown).toBe(false);
  });

  test('flag OFF - degraded path does not call payment provider', async () => {
    td.update(td.flag('checkout-kill').boolVariation(false));
    const paymentSpy = jest.spyOn(paymentProvider, 'charge');
    await renderCheckout({ flagClient: clientWith(td) });
    expect(paymentSpy).not.toHaveBeenCalled();
  });
});

Document what the degraded state must and must not do; include it as a code comment or test description so the test doubles as operational runbook.

Category 2 - Fail-static default when the flag service is unreachable

Per the OpenFeature specification at openfeature.dev/docs/reference/concepts/evaluation-api (opens in new window): "In the case of any error during flag evaluation, the default value will be returned, so give consideration to your default values!"

The LaunchDarkly SDK aligns: "The fallback value is defined in your code... and is only returned if an error occurs" including "LaunchDarkly service is unreachable" (source: launchdarkly.com/docs/sdk/features/evaluating (opens in new window)).

The test must assert two things: the SDK returns the correct default, and the application continues operating with that default rather than throwing.

import { OpenFeature, InMemoryProvider } from '@openfeature/server-sdk';

describe('checkout-kill-switch: fail-static default', () => {
  test('provider unavailable - default OFF is served, app does not crash', async () => {
    // Register a provider that always throws
    await OpenFeature.setProviderAndWait(new AlwaysErrorProvider());
    const client = OpenFeature.getClient();

    // Default is false (feature disabled) - the safe side for a kill-switch
    const value = await client.getBooleanValue('checkout-kill', false);
    expect(value).toBe(false);

    // Application layer must handle the default without throwing
    await expect(renderCheckout({ featureEnabled: value })).resolves.not.toThrow();
  });

  test('provider unavailable - error hook fires and is logged', async () => {
    const errorEvents: string[] = [];
    OpenFeature.addHooks({
      error: (_ctx, err) => { errorEvents.push(err.message); }
    });
    await OpenFeature.setProviderAndWait(new AlwaysErrorProvider());
    const client = OpenFeature.getClient();
    await client.getBooleanValue('checkout-kill', false);
    expect(errorEvents.length).toBeGreaterThan(0);
  });
});

The default value for a kill-switch MUST be the safe side: false for a flag that enables a feature (disable it on error) or true for a flag that disables a feature (keep it disabled on error). Document the chosen default and its rationale as a comment at the flag call site.

Category 3 - Latency budget for the kill decision

A kill-switch flipped in the operator console must reach running processes within an acceptable window. The window depends on the streaming/polling configuration of the SDK.

Per LaunchDarkly's documentation at launchdarkly.com/docs/sdk/concepts/client-side-server-side (opens in new window): "Server-side SDKs open a streaming connection to LaunchDarkly and receive flag configuration changes over the stream." The cached values have "no expiration or time-to-live (TTL) value" - propagation speed depends on the streaming connection, not a TTL. Client-side SDKs that use polling have an interval-bound lag.

Because in-process latency tests against a live SDK are environment-dependent and slow, the recommended approach is two tests:

1. A unit test that asserts the flag-evaluation result changes synchronously when the in-memory test source is updated (validates that the application re-reads the flag on each request rather than caching it locally).
2. An integration test comment documenting the expected propagation window for your SDK configuration, to be verified in a staging smoke test.

describe('checkout-kill-switch: kill latency', () => {
  test('flag re-evaluated per request - not cached application-side', async () => {
    const td = TestData.dataSource();
    td.update(td.flag('checkout-kill').boolVariation(true));
    const client = clientWith(td);

    const before = await isCheckoutEnabled(client);
    expect(before).toBe(true);

    // Flip the kill-switch
    td.update(td.flag('checkout-kill').boolVariation(false));

    // Next evaluation reflects the flip without a process restart
    const after = await isCheckoutEnabled(client);
    expect(after).toBe(false);
  });

  // Integration note: with LaunchDarkly streaming (server SDK default),
  // flag changes propagate in near-real-time over the SSE stream.
  // With Unleash polling (default 15s interval per unleash.io/docs),
  // the worst-case lag equals the polling interval.
  // Agree on the acceptable window with SRE and add a staging smoke test.
});

If the application caches the flag value (e.g., in a request-scoped singleton), this test will catch it.

Category 4 - No data corruption mid-flight

When the kill-switch flips while an operation is already in progress (a multi-step transaction, a streaming response, a batch job), the in-progress operation must complete cleanly or roll back - it must not leave partial state.

This is the most scenario-specific of the four categories. The general pattern is to snapshot the flag value at the start of the operation and hold it for the operation's duration rather than re-evaluating mid-operation.

describe('checkout-kill-switch: mid-flight flip', () => {
  test('in-progress order is not corrupted when kill-switch flips mid-checkout', async () => {
    const td = TestData.dataSource();
    td.update(td.flag('checkout-kill').boolVariation(true));
    const client = clientWith(td);

    // Begin a multi-step checkout; flip the flag after step 1
    const order = await startCheckout(client);  // Step 1: reserve inventory

    // Simulate operator flipping the kill-switch during step 2
    td.update(td.flag('checkout-kill').boolVariation(false));

    const result = await completeCheckout(order);  // Step 2: charge + confirm

    // Either fully committed or fully rolled back - never partial
    expect(['committed', 'rolled_back']).toContain(result.state);
    if (result.state === 'rolled_back') {
      // Inventory reservation must be released
      expect(await inventoryReserved(order.itemId)).toBe(false);
    }
  });
});

Document the consistency contract in a test description or comment: what "no partial state" means for this specific operation.

Anti-patterns

Limitations

• Category 3 (latency) cannot be fully tested without a real SDK connection. The unit test verifies application-side re-evaluation; the propagation window (streaming lag or poll interval) requires a staging integration test.
• Mid-flight semantics are application-specific. Category 4 must be authored per operation; no generic skeleton fits all cases.
• Multi-process deployments. Each server process holds an independent SDK connection. The kill propagates to each independently; the worst-case latency is per-process.

Sources

• Kill-switch / ops-toggle definition and taxonomy: Pete Hodgson, "Feature Toggles (aka Feature Flags)": martinfowler.com/articles/feature-toggles.html (opens in new window).
• Fail-static default value behavior (OpenFeature): openfeature.dev/docs/reference/concepts/evaluation-api (opens in new window).
• Fallback values on SDK error or service unreachability (LaunchDarkly): launchdarkly.com/docs/sdk/features/evaluating (opens in new window).
• LaunchDarkly TestData mock source for SDK-level tests: launchdarkly.com/docs/sdk/features/test-data-sources (opens in new window).
• Streaming vs polling propagation speed and in-memory cache behavior (LaunchDarkly): launchdarkly.com/docs/sdk/concepts/client-side-server-side (opens in new window).
• Circuit breaker as automated kill-switch pattern: Martin Fowler, "CircuitBreaker": martinfowler.com/bliki/CircuitBreaker.html (opens in new window).

Companion reference for openfeature-sdk-testing. The SKILL.md spine shows the full TypeScript/Node.js flow; the flow is identical here, only the API names differ.

Java

Install (Maven, per github.com/open-feature/java-sdk README (opens in new window)):

<dependency>
  <groupId>dev.openfeature</groupId>
  <artifactId>sdk</artifactId>
  <version>1.20.2</version>
</dependency>

Configure the InMemoryProvider:

import dev.openfeature.sdk.OpenFeatureAPI;
import dev.openfeature.sdk.Client;
import dev.openfeature.sdk.providers.memory.Flag;
import dev.openfeature.sdk.providers.memory.InMemoryProvider;

Map<String, Flag<?>> flags = new HashMap<>();
flags.put("show-new-ui", Flag.builder()
    .variant("on", true)
    .variant("off", false)
    .defaultVariant("on")
    .build());

OpenFeatureAPI api = OpenFeatureAPI.getInstance();
api.setProviderAndWait(new InMemoryProvider(flags));
Client client = api.getClient();

Evaluate flags:

boolean enabled = client.getBooleanValue("show-new-ui", false);

FlagEvaluationDetails<Boolean> details =
    client.getBooleanDetails("show-new-ui", false);
// details.getValue()    - resolved value
// details.getVariant()  - "on" or "off"
// details.getReason()   - "STATIC", "DEFAULT", etc.
// details.getErrorCode()- ErrorCode.FLAG_NOT_FOUND, TYPE_MISMATCH, etc.

Per-invocation evaluation context:

Map<String, Value> attrs = new HashMap<>();
attrs.put("email", new Value("user@example.com"));
EvaluationContext ctx = new ImmutableContext("user-42", attrs);
boolean value = client.getBooleanValue("beta-access", false, ctx);

Python

Install (per github.com/open-feature/python-sdk README (opens in new window)):

pip install openfeature-sdk==0.10.0

Configure the InMemoryProvider:

from openfeature import api
from openfeature.provider.in_memory_provider import InMemoryFlag, InMemoryProvider

flags = {
    "show-new-ui": InMemoryFlag(
        default_variant="on",
        variants={"on": True, "off": False}
    ),
}

api.set_provider_and_wait(InMemoryProvider(flags))
client = api.get_client()

Evaluate flags:

enabled = client.get_boolean_value("show-new-ui", False)

details = client.get_boolean_details("show-new-ui", False)
# details.value       - resolved value
# details.variant     - "on" / "off"
# details.reason      - "STATIC", "DEFAULT", etc.
# details.error_code  - "FLAG_NOT_FOUND", "TYPE_MISMATCH", etc.

Per-invocation evaluation context:

from openfeature.evaluation_context import EvaluationContext
ctx = EvaluationContext(targeting_key="user-42",
                        attributes={"email": "user@example.com"})
details = client.get_boolean_details("beta-access", False, ctx)

EvaluationDetails and reason codes

Per the OpenFeature specification (openfeature.dev/specification/sections/flag-evaluation (opens in new window), Requirements 1.4.3-1.4.15), EvaluationDetails carries:

Canonical reason values (per openfeature.dev/specification/sections/providers (opens in new window), Requirement 2.2.5): STATIC, DEFAULT, TARGETING_MATCH, SPLIT, CACHED, DISABLED, UNKNOWN, STALE, ERROR.

Canonical error codes include FLAG_NOT_FOUND, TYPE_MISMATCH, PARSE_ERROR, TARGETING_KEY_MISSING, INVALID_CONTEXT, GENERAL, PROVIDER_NOT_READY, PROVIDER_FATAL.

Hook lifecycle stages and execution order

Per openfeature.dev/docs/reference/concepts/hooks (opens in new window) and the specification (openfeature.dev/specification/sections/hooks (opens in new window), Requirement 4.3.1-4.3.8), the four stages are:

Execution order for before: API - Client - Invocation. For after, error, finally: reverse order (Invocation - Client - API), per specification Requirement 4.4.2.

Wraps Unleash (Open Source / SaaS) SDK testing patterns: bootstrap with a static toggles array (no network), the test mode (disableMetrics + disablePolling), the custom strategy testing pattern (implement a Strategy class + assert isEnabled), and assignment-integrity tests.

Install

npm install --save-dev unleash-client
pip install UnleashClient

Bootstrap with toggles (offline)

import { initialize } from 'unleash-client';

const unleash = initialize({
  url: 'http://localhost:4242/api/',
  appName: 'test-app',
  disableMetrics: true,        // No metrics upload
  disablePolling: true,        // No background polling
  bootstrap: {
    data: [
      {
        name: 'show-new-ui',
        enabled: true,
        strategies: [
          { name: 'default' },
        ],
      },
    ],
  },
});

// Wait for initialization
await new Promise<void>((resolve) => unleash.once('synchronized', resolve));

The bootstrap.data array is the SDK's initial flag state; since polling is disabled, that state persists for the test session.

isEnabled tests

test('flag enabled', () => {
  expect(unleash.isEnabled('show-new-ui')).toBe(true);
});

test('flag with context', () => {
  expect(unleash.isEnabled('premium-only', { userId: 'u1', properties: { tier: 'premium' } })).toBe(true);
});

Custom strategy

Unleash's extensibility: custom strategies implement an isEnabled(parameters, context) method.

import { Strategy } from 'unleash-client';

class TenantStrategy extends Strategy {
  constructor() { super('tenantStrategy'); }
  isEnabled(parameters: any, context: any): boolean {
    const allowedTenants = (parameters.tenants ?? '').split(',');
    return allowedTenants.includes(context.tenantId);
  }
}

const unleash = initialize({
  url: '...',
  appName: 'test',
  strategies: [new TenantStrategy()],
  bootstrap: {
    data: [
      {
        name: 'flag-x',
        enabled: true,
        strategies: [{ name: 'tenantStrategy', parameters: { tenants: 'A,B' } }],
      },
    ],
  },
});

test('strategy allows tenant A', () => {
  expect(unleash.isEnabled('flag-x', { tenantId: 'A' })).toBe(true);
});

test('strategy rejects tenant C', () => {
  expect(unleash.isEnabled('flag-x', { tenantId: 'C' })).toBe(false);
});

Percentage-rollout determinism

const unleash = initialize({
  // ...
  bootstrap: {
    data: [
      {
        name: 'gradual-rollout',
        enabled: true,
        strategies: [{ name: 'flexibleRollout', parameters: { rollout: '50', stickiness: 'userId' } }],
      },
    ],
  },
});

test('rollout deterministic per user', () => {
  const r1 = unleash.isEnabled('gradual-rollout', { userId: 'u1' });
  const r2 = unleash.isEnabled('gradual-rollout', { userId: 'u1' });
  expect(r1).toBe(r2);
});

Teardown

afterAll(() => unleash.destroy());

CI integration

Fully offline - no Unleash server URL needed since polling is disabled:

jobs:
  unleash-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v4
      - run: npm ci && npm test

Anti-patterns

Limitations

• Bootstrap is point-in-time. Drift between fixture + Unleash UI is invisible.
• Custom strategies tested in isolation lose context. Integration test with the full SDK still needed.
• Metric-side tests need real server. If you're testing the metrics export path, bootstrap mode is wrong.
• Stickiness is parameter-driven. A misconfigured stickiness parameter shows up in production, not in tests.

Sources

• Unleash docs: docs.getunleash.io (opens in new window).
• Node SDK: docs.getunleash.io/sdks/node (opens in new window).
• Custom strategies: docs.getunleash.io/concepts/activation-strategies (opens in new window).