openfeature-sdk-testing
Wraps OpenFeature (CNCF vendor-neutral SDK abstraction) testing patterns: the InMemoryProvider for hermetic tests without network calls, provider registration via OpenFeature.setProvider, the getBooleanValue/getBooleanDetails evaluation API with EvaluationDetails (value, variant, reason, errorCode), hooks for evaluation side-effects, and evaluation context for targeting-rule tests. Covers TypeScript, Java, and Python SDKs, plus per-vendor hermetic-bootstrap references for Unleash (bootstrap toggles), Flagsmith (offline LocalFileHandler), and GrowthBook (initSync payload). Use when writing tests for code that resolves feature flags through the OpenFeature SDK or the Unleash / Flagsmith / GrowthBook native SDKs; LaunchDarkly has its own skill (launchdarkly-testing).
Install with skills.sh (any agent)
npx skills add testland/qa --skill openfeature-sdk-testingopenfeature-sdk-testing
Overview
The OpenFeature SDK ships an InMemoryProvider in every language that substitutes real flag-management infrastructure with in-process flag state, letting unit and integration tests run without any network call. The production evaluation path (targeting logic, type coercion, defaults) is exercised in full; only the data source is swapped. Per openfeature.dev/docs/reference/concepts/provider (opens in new window), "an application integrator can register one provider at a time."
Scope: this skill is the SDK-testing umbrella. The body covers the vendor-neutral OpenFeature layer teams adopt to keep application code decoupled from a specific provider; the vendor-native hermetic-bootstrap patterns (each an offline data-source variant of the same idea) live in references/: references/unleash.md (bootstrap toggles + custom strategies), references/flagsmith.md (offline LocalFileHandler + default_flag_handler), and references/growthbook.md (initSync payload + inline experiments). LaunchDarkly's TestData source has enough distinct surface for its own skill: launchdarkly-testing.
When to use
How to use
TypeScript/Node.js is the canonical language below. The Java and Python flow is identical; only the API names differ, listed in references/multi-language-and-spec.md.
TypeScript / Node.js
Install (per github.com/open-feature/js-sdk README (opens in new window)):
npm install --save @openfeature/server-sdkConfigure the InMemoryProvider with flag variants and a default variant:
import { OpenFeature, InMemoryProvider } from '@openfeature/server-sdk';
const flags = {
'show-new-ui': {
variants: { on: true, off: false },
disabled: false,
defaultVariant: 'on',
},
'checkout-v2': {
variants: { enabled: true, disabled: false },
disabled: false,
defaultVariant: 'disabled',
},
} as const;
await OpenFeature.setProvider(new InMemoryProvider(flags));
const client = OpenFeature.getClient();Evaluate flags using the typed evaluation API (per openfeature.dev/docs/reference/concepts/evaluation-api (opens in new window)):
// Returns the resolved value; falls back to default on error
const enabled = await client.getBooleanValue('show-new-ui', false);
// Returns full EvaluationDetails
const details = await client.getBooleanDetails('show-new-ui', false);
// details.value - the resolved boolean
// details.variant - e.g. "on"
// details.reason - e.g. "STATIC", "DEFAULT", "TARGETING_MATCH"
// details.errorCode- e.g. "FLAG_NOT_FOUND" when the flag is absentTypical test pattern:
import { OpenFeature, InMemoryProvider } from '@openfeature/server-sdk';
describe('checkout feature', () => {
let client: Client;
beforeAll(async () => {
await OpenFeature.setProvider(new InMemoryProvider({
'checkout-v2': {
variants: { enabled: true, disabled: false },
disabled: false,
defaultVariant: 'disabled',
},
}));
client = OpenFeature.getClient();
});
afterAll(() => OpenFeature.close());
it('returns false when flag defaults to disabled', async () => {
const value = await client.getBooleanValue('checkout-v2', false);
expect(value).toBe(false);
});
it('returns details with reason STATIC for static flags', async () => {
const details = await client.getBooleanDetails('checkout-v2', false);
expect(details.reason).toBe('STATIC');
});
it('returns FLAG_NOT_FOUND error code for unknown flag', async () => {
const details = await client.getBooleanDetails('unknown-flag', false);
expect(details.errorCode).toBe('FLAG_NOT_FOUND');
});
});Java and Python install/configure/evaluate blocks live in references/multi-language-and-spec.md.
EvaluationDetails and reason codes
EvaluationDetails carries value, flagKey, variant, reason, errorCode, errorMessage, and flagMetadata. Canonical reason values include STATIC, DEFAULT, TARGETING_MATCH, SPLIT, CACHED, DISABLED, UNKNOWN, STALE, ERROR; canonical error codes include FLAG_NOT_FOUND, TYPE_MISMATCH, TARGETING_KEY_MISSING, and PROVIDER_NOT_READY. The full field table with spec requirement numbers is in references/multi-language-and-spec.md.
Evaluation context for targeting-rule tests
Evaluation context is "a container for arbitrary contextual data that can be used as a basis for dynamic evaluation" (per openfeature.dev/docs/reference/concepts/evaluation-context (opens in new window)). The targeting key is a unique identifier (user ID, session ID) that providers use for deterministic bucketing. Custom attributes carry additional data (email, plan, region).
Context can be set at three levels: global (via the API object), client, and per invocation. Lower levels override duplicate keys from higher levels.
// TypeScript - per-invocation context for targeting-rule tests
const ctx = { targetingKey: 'user-42', email: 'user@example.com' };
const details = await client.getBooleanDetails('beta-access', false, ctx);
expect(details.reason).toBe('TARGETING_MATCH');Java and Python context builders (ImmutableContext, EvaluationContext) are in references/multi-language-and-spec.md.
Hooks for test-time side-effects
Hooks intercept the flag evaluation lifecycle at four stages - before (can modify evaluation context), after (validate the returned value), error (on resolution failure), and finally (unconditionally). The stage table and execution order (per specification Requirement 4.4.2) are in references/multi-language-and-spec.md.
Register hooks at global, client, or invocation level:
// Global hook - runs for every flag evaluation
OpenFeature.addHooks({
before(ctx) {
// ctx carries flagKey, flagValueType, defaultValue, evaluationContext
console.log(`Evaluating ${ctx.flagKey}`);
},
after(ctx, details) {
// details is the EvaluationDetails for this evaluation
expect(details.errorCode).toBeUndefined();
},
error(ctx, err) {
console.error(`Flag ${ctx.flagKey} failed: ${err.message}`);
},
});Test use case: attach an after hook to assert that no evaluation returns an error code, surfacing FLAG_NOT_FOUND regressions across the entire test run without asserting each flag individually.
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Registering a real (networked) provider in unit tests | Network calls; non-deterministic; slow | InMemoryProvider for unit tests |
Evaluating without setProviderAndWait / await setProvider | Returns default with PROVIDER_NOT_READY error code before init | Always await provider readiness |
Sharing a single InMemoryProvider instance across test files | Cross-test state pollution | Create a fresh provider per describe block |
Asserting only value; ignoring reason and errorCode | Hides fallback-to-default failures (flag absent, type mismatch) | Assert details.reason and details.errorCode explicitly |
| Testing provider internals (variant weighting, targeting logic) | That is the provider's responsibility, not the application's | Test what the application does with the evaluated value |
Omitting OpenFeature.close() in teardown | Leaks provider state and background threads | Always call close() / shutdown() in afterAll |
Limitations
References
Flagsmith test modes, anti-patterns, and limitations
View source (opens in new window)Flagsmith test modes, anti-patterns, and limitations
Deeper detail for flagsmith.md (opens in new window). Offline mode (covered in the skill spine) is the default for tests; the modes below are secondary.
Local-evaluation mode
Polls the Flagsmith API periodically and evaluates flags locally between refreshes (no per-request network, but not zero network):
flagsmith = Flagsmith(
environment_key="server-key",
enable_local_evaluation=True,
environment_refresh_interval_seconds=60,
)Local mode polls; offline mode does not. For tests, offline is usually preferred.
default_flag_handler - per-flag mock
Programmatic fallback used when the offline environment.json does not have the flag under test yet:
from flagsmith import Flagsmith
from flagsmith.models import DefaultFlag
def default_flag_handler(feature_name: str) -> DefaultFlag:
if feature_name == "secret_button":
return DefaultFlag(enabled=False, value='{"colour": "#b8b8b8"}')
return DefaultFlag(enabled=False, value=None)
flagsmith = Flagsmith(
environment_key="test-key",
default_flag_handler=default_flag_handler,
)Useful when the offline environment.json does not have the flag you are testing yet.
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Production env_key in tests | Real API calls + analytics pollution | offline_mode + LocalFileHandler |
| environment.json not committed | Test flakes when prod changes | Commit; refresh deliberately |
| Mixing offline_mode + local_evaluation_mode | Conflicting; one takes precedence | Pick one |
| default_flag_handler returns DefaultFlag with no value | Tests for value-based flags fail silently | Always set value |
Skipping flagsmith.get_identity_flags for identity-scoped tests | Bypasses per-user logic | Use identity API |
| Per-test new Flagsmith client | Slow init | Session-scoped fixture |
Limitations
Flagsmith SDK testing
View source (opens in new window)Flagsmith SDK testing
Wraps Flagsmith server-side SDK testing patterns so tests run without calling the Flagsmith API: offline mode with LocalFileHandler + a downloaded environment.json snapshot, local-evaluation mode (no per-request network), and default_flag_handler for per-feature mocked fallbacks, via the get_environment_flags / get_identity_flags evaluation paths.
Flagsmith (open-source, also SaaS at flagsmith.com) supports three test-friendly modes per docs.flagsmith.com/clients/server-side (opens in new window):
Offline mode is the default choice for tests; local-evaluation and default_flag_handler detail live in flagsmith-modes.md (opens in new window).
How to use
Offline mode with LocalFileHandler
Per docs.flagsmith.com (opens in new window):
from flagsmith import Flagsmith
from flagsmith.offline_handlers import LocalFileHandler
local_file_handler = LocalFileHandler(environment_document_path="tests/fixtures/flagsmith-environment.json")
flagsmith = Flagsmith(offline_mode=True, offline_handler=local_file_handler)Download the environment.json via the Flagsmith CLI, then commit it (refresh deliberately):
flagsmith environment-document --api-key=<server-key> --output=tests/fixtures/flagsmith-environment.jsonEvaluate flags
def test_environment_flag():
flags = flagsmith.get_environment_flags()
assert flags.is_feature_enabled("secret_button") is False
assert flags.get_feature_value("secret_button") == '{"colour": "#b8b8b8"}'
def test_identity_flag():
flags = flagsmith.get_identity_flags(
identifier="user@example.com",
traits={"plan": "premium"},
)
assert flags.is_feature_enabled("premium_feature") is TrueIntegrity test - the same identity must resolve consistently:
def test_same_identity_consistent():
f1 = flagsmith.get_identity_flags("u1")
f2 = flagsmith.get_identity_flags("u1")
assert f1.is_feature_enabled("flag-x") == f2.is_feature_enabled("flag-x")CI integration
No env vars needed in offline mode:
jobs:
flagsmith-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/setup-python@v5
- run: pip install flagsmith
- run: pytest tests/flagsmith/Worked example
A service reads a secret_button flag (boolean plus a JSON colour value) and a premium_feature flag gated on a plan trait, tested fully offline:
Result: pytest tests/flagsmith/ runs green in CI with zero network access and no analytics pollution.
Sources
GrowthBook SDK testing
View source (opens in new window)GrowthBook SDK testing
Wraps GrowthBook Node SDK testing patterns: GrowthBookClient initialization with direct payload (initSync; no network), isOn / getFeatureValue / evalFeature, scoped instances (createScopedInstance) for per-request user context, inline experiment (runInlineExperiment) tests, and tracking-callback assertion patterns.
Per docs.growthbook.io/lib/node (opens in new window), the GrowthBook Node SDK's GrowthBookClient supports an initSync({ payload }) pattern that fully bypasses the network - pass the feature definitions directly. createScopedInstance lets each test (or request) bind its own user context cleanly.
Install
npm install --save-dev @growthbook/growthbookInitialize with payload (offline)
Per docs.growthbook.io (opens in new window):
import { GrowthBookClient } from '@growthbook/growthbook';
const gbClient = new GrowthBookClient().initSync({
payload: {
features: {
'show-new-ui': { defaultValue: true },
'experiment-x': {
defaultValue: false,
rules: [{ condition: { id: 'test-user-1' }, force: true }],
},
},
},
});initSync is purpose-built for tests - no async wait.
isOn / getFeatureValue
test('flag on', () => {
const userContext = { attributes: { id: 'user-1' } };
expect(gbClient.isOn('show-new-ui', userContext)).toBe(true);
});
test('typed feature value', () => {
const userContext = { attributes: { id: 'user-1' } };
const color = gbClient.getFeatureValue('button-color', 'blue', userContext);
expect(['blue', 'red', 'green']).toContain(color);
});Scoped instance per test
test('user-specific evaluation', () => {
const instance = gbClient.createScopedInstance({
attributes: { id: 'user-1', plan: 'premium' },
});
expect(instance.isOn('premium-feature')).toBe(true);
});Avoids passing context everywhere.
Inline experiments
test('inline experiment returns one variant', () => {
const userContext = { attributes: { id: 'user-1' } };
const { value } = gbClient.runInlineExperiment({
key: 'my-experiment',
variations: ['red', 'blue', 'green'],
coverage: 1.0,
weights: [0.33, 0.34, 0.33],
}, userContext);
expect(['red', 'blue', 'green']).toContain(value);
});Tracking-callback assertions
test('experiment tracking fires', () => {
const tracked: any[] = [];
const client = new GrowthBookClient({
trackingCallback: (exp, result, ctx) => {
tracked.push({ key: exp.key, variation: result.key });
},
}).initSync({ payload: {} });
client.runInlineExperiment(
{ key: 'exp-x', variations: [0, 1] },
{ attributes: { id: 'user-1' } }
);
expect(tracked).toHaveLength(1);
expect(tracked[0].key).toBe('exp-x');
});Feature-usage callback
const evaluated: any[] = [];
test('feature usage logged', () => {
const userContext = {
attributes: { id: 'user-1' },
onFeatureUsage: (key: string, result: any) => {
evaluated.push({ key, value: result.value });
},
};
gbClient.evalFeature('feature-x', userContext);
expect(evaluated).toContainEqual(expect.objectContaining({ key: 'feature-x' }));
});TypeScript-strict feature contract
interface AppFeatures {
'button-color': string;
'font-size': number;
'newForm': boolean;
}
const gbClient = new GrowthBookClient<AppFeatures>({}).initSync({ payload: {} });
const color = gbClient.getFeatureValue('button-color', 'blue', ctx); // typed
// gbClient.isOn('buton-color', ctx); // typo → compile errorCI integration
Fully offline; no GrowthBook key needed:
jobs:
growthbook-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/setup-node@v4
- run: npm ci && npm testAnti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
Live apiHost in tests | Network requests; flaky | initSync({ payload }) |
| Skipping tracking-callback assertion | Exposure-event regressions silent | Per-test callback + assert |
| Sharing scopedInstance across tests | Cross-test state | Per-test create |
| TypeScript any-typed features | Lose compile-time safety | Generic AppFeatures |
coverage: 0.1 in test without large N | Not enough samples to see all variations | Use coverage: 1.0 for deterministic tests |
| Missing user.id in context | Bucketing degenerate | Always pass attributes.id |
Tests assume default weights is 50/50 | Default behaviour drifts | Explicit weights |
Limitations
Sources
OpenFeature testing: Java / Python SDKs and specification tables
View source (opens in new window)OpenFeature testing: Java / Python SDKs and specification tables
Companion reference for openfeature-sdk-testing. The SKILL.md spine shows the full TypeScript/Node.js flow; the flow is identical here, only the API names differ.
Java
Install (Maven, per github.com/open-feature/java-sdk README (opens in new window)):
<dependency>
<groupId>dev.openfeature</groupId>
<artifactId>sdk</artifactId>
<version>1.20.2</version>
</dependency>Configure the InMemoryProvider:
import dev.openfeature.sdk.OpenFeatureAPI;
import dev.openfeature.sdk.Client;
import dev.openfeature.sdk.providers.memory.Flag;
import dev.openfeature.sdk.providers.memory.InMemoryProvider;
Map<String, Flag<?>> flags = new HashMap<>();
flags.put("show-new-ui", Flag.builder()
.variant("on", true)
.variant("off", false)
.defaultVariant("on")
.build());
OpenFeatureAPI api = OpenFeatureAPI.getInstance();
api.setProviderAndWait(new InMemoryProvider(flags));
Client client = api.getClient();Evaluate flags:
boolean enabled = client.getBooleanValue("show-new-ui", false);
FlagEvaluationDetails<Boolean> details =
client.getBooleanDetails("show-new-ui", false);
// details.getValue() - resolved value
// details.getVariant() - "on" or "off"
// details.getReason() - "STATIC", "DEFAULT", etc.
// details.getErrorCode()- ErrorCode.FLAG_NOT_FOUND, TYPE_MISMATCH, etc.Per-invocation evaluation context:
Map<String, Value> attrs = new HashMap<>();
attrs.put("email", new Value("user@example.com"));
EvaluationContext ctx = new ImmutableContext("user-42", attrs);
boolean value = client.getBooleanValue("beta-access", false, ctx);Python
Install (per github.com/open-feature/python-sdk README (opens in new window)):
pip install openfeature-sdk==0.10.0Configure the InMemoryProvider:
from openfeature import api
from openfeature.provider.in_memory_provider import InMemoryFlag, InMemoryProvider
flags = {
"show-new-ui": InMemoryFlag(
default_variant="on",
variants={"on": True, "off": False}
),
}
api.set_provider_and_wait(InMemoryProvider(flags))
client = api.get_client()Evaluate flags:
enabled = client.get_boolean_value("show-new-ui", False)
details = client.get_boolean_details("show-new-ui", False)
# details.value - resolved value
# details.variant - "on" / "off"
# details.reason - "STATIC", "DEFAULT", etc.
# details.error_code - "FLAG_NOT_FOUND", "TYPE_MISMATCH", etc.Per-invocation evaluation context:
from openfeature.evaluation_context import EvaluationContext
ctx = EvaluationContext(targeting_key="user-42",
attributes={"email": "user@example.com"})
details = client.get_boolean_details("beta-access", False, ctx)EvaluationDetails and reason codes
Per the OpenFeature specification (openfeature.dev/specification/sections/flag-evaluation (opens in new window), Requirements 1.4.3-1.4.15), EvaluationDetails carries:
| Field | Type | Meaning |
|---|---|---|
value | T | Resolved flag value (spec req. 1.4.3) |
flagKey | string | The requested flag identifier (1.4.5) |
variant | string | Provider-supplied variant name (1.4.6) |
reason | string | Resolution rationale (1.4.7) |
errorCode | enum | Failure classification (1.4.8) |
errorMessage | string | Optional context for the error (1.4.13) |
flagMetadata | map | Immutable provider-supplied data (1.4.14) |
Canonical reason values (per openfeature.dev/specification/sections/providers (opens in new window), Requirement 2.2.5): STATIC, DEFAULT, TARGETING_MATCH, SPLIT, CACHED, DISABLED, UNKNOWN, STALE, ERROR.
Canonical error codes include FLAG_NOT_FOUND, TYPE_MISMATCH, PARSE_ERROR, TARGETING_KEY_MISSING, INVALID_CONTEXT, GENERAL, PROVIDER_NOT_READY, PROVIDER_FATAL.
Hook lifecycle stages and execution order
Per openfeature.dev/docs/reference/concepts/hooks (opens in new window) and the specification (openfeature.dev/specification/sections/hooks (opens in new window), Requirement 4.3.1-4.3.8), the four stages are:
| Stage | Runs when |
|---|---|
before | Before flag resolution; can modify evaluation context |
after | After successful resolution; can validate the returned value |
error | On resolution failure or unhandled before-hook error |
finally | Unconditionally after all other stages |
Execution order for before: API - Client - Invocation. For after, error, finally: reverse order (Invocation - Client - API), per specification Requirement 4.4.2.
Unleash SDK testing
View source (opens in new window)Unleash SDK testing
Wraps Unleash (Open Source / SaaS) SDK testing patterns: bootstrap with a static toggles array (no network), the test mode (disableMetrics + disablePolling), the custom strategy testing pattern (implement a Strategy class + assert isEnabled), and assignment-integrity tests.
Install
npm install --save-dev unleash-client
pip install UnleashClientBootstrap with toggles (offline)
import { initialize } from 'unleash-client';
const unleash = initialize({
url: 'http://localhost:4242/api/',
appName: 'test-app',
disableMetrics: true, // No metrics upload
disablePolling: true, // No background polling
bootstrap: {
data: [
{
name: 'show-new-ui',
enabled: true,
strategies: [
{ name: 'default' },
],
},
],
},
});
// Wait for initialization
await new Promise<void>((resolve) => unleash.once('synchronized', resolve));The bootstrap.data array is the SDK's initial flag state; since polling is disabled, that state persists for the test session.
isEnabled tests
test('flag enabled', () => {
expect(unleash.isEnabled('show-new-ui')).toBe(true);
});
test('flag with context', () => {
expect(unleash.isEnabled('premium-only', { userId: 'u1', properties: { tier: 'premium' } })).toBe(true);
});Custom strategy
Unleash's extensibility: custom strategies implement an isEnabled(parameters, context) method.
import { Strategy } from 'unleash-client';
class TenantStrategy extends Strategy {
constructor() { super('tenantStrategy'); }
isEnabled(parameters: any, context: any): boolean {
const allowedTenants = (parameters.tenants ?? '').split(',');
return allowedTenants.includes(context.tenantId);
}
}
const unleash = initialize({
url: '...',
appName: 'test',
strategies: [new TenantStrategy()],
bootstrap: {
data: [
{
name: 'flag-x',
enabled: true,
strategies: [{ name: 'tenantStrategy', parameters: { tenants: 'A,B' } }],
},
],
},
});
test('strategy allows tenant A', () => {
expect(unleash.isEnabled('flag-x', { tenantId: 'A' })).toBe(true);
});
test('strategy rejects tenant C', () => {
expect(unleash.isEnabled('flag-x', { tenantId: 'C' })).toBe(false);
});Percentage-rollout determinism
const unleash = initialize({
// ...
bootstrap: {
data: [
{
name: 'gradual-rollout',
enabled: true,
strategies: [{ name: 'flexibleRollout', parameters: { rollout: '50', stickiness: 'userId' } }],
},
],
},
});
test('rollout deterministic per user', () => {
const r1 = unleash.isEnabled('gradual-rollout', { userId: 'u1' });
const r2 = unleash.isEnabled('gradual-rollout', { userId: 'u1' });
expect(r1).toBe(r2);
});Teardown
afterAll(() => unleash.destroy());CI integration
Fully offline - no Unleash server URL needed since polling is disabled:
jobs:
unleash-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/setup-node@v4
- run: npm ci && npm testAnti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
disableMetrics: false in CI | Spurious metrics POSTs | Always disableMetrics: true in tests |
disablePolling: false without test-only URL | Network calls; CI flakes | Always disable polling in offline tests |
bootstrap.data is stale | Drift from prod definitions | Pull from Unleash periodically; commit fixture |
| Custom strategies not unit-tested | Logic bugs in the strategy itself | Test the strategy class in isolation too |
Skipping synchronized event wait | Race: isEnabled returns default | Always await synchronized |
Forgetting unleash.destroy() | Goroutine / timer leak | Always destroy in afterAll |
| Tests use real Unleash server | Slow; flaky if server down | Bootstrap mode |
Limitations
Sources
Related skills
ab-test-validity-checklist
Workflow skill that builds an A/B-test validity checklist from an experiment proposal, walking the canonical design-correctness gates - pre-registered OEC/power/guardrails, randomization unit + SRM check, assignment integrity, telemetry, peeking discipline, novelty/primacy, post-experiment SRM re-check - into a per-experiment checklist + sign-off form. Use when launching, auditing, or governing an experiment. For pitfall mechanics (guardrails, peeking) see experiment-results-interpreter's references; to read an already-valid result use experiment-results-interpreter; for per-SDK harness tests use experiment-sdk-testing - this gates DESIGN, not SDK code.
experiment-results-interpreter
Interprets the results of a valid online controlled experiment, one whose harness, SRM, and telemetry have already been confirmed. Covers the distinction between practical and statistical significance, reading confidence intervals instead of binary p-values, novelty and primacy week-over-week decay that causes post-ship reversion, interaction effects from concurrent experiments, Simpson's paradox in segmented results, and the ordered guardrail-check sequence required before a ship decision - with the deep methodology in references/: the peeking problem and its corrections (fixed-horizon, alpha-spending, always-valid mSPRT) in references/peeking.md, and guardrail-metric methodology (taxonomy, OEC relationship, pre-commitment, thresholds) in references/guardrails.md. Use when a data scientist or PM is ready to draw conclusions from an experiment, when designing a stop-early policy, or when declaring an experiment's guardrail set. Distinct from ab-test-validity-checklist (harness setup and SRM detection).
experiment-sdk-testing
Umbrella for experimentation-SDK test harnesses: the shared offline-datafile / hermetic-init pattern (commit a point-in-time flag/experiment config fixture, initialize the SDK with no network, pin arms per test, assert assignment integrity), with per-vendor references for Statsig (localMode + overrideGate), Optimizely (datafile + forced decisions), Split.io / Harness FME (localhost mode + features map or YAML fixture), Amplitude Experiment (local evaluation + bootstrap), and VWO (settings file + deterministic bucketing). Use when writing tests for application code instrumented with any of these five experimentation SDKs; for experiment DESIGN gates use ab-test-validity-checklist, and to read results use experiment-results-interpreter.
feature-flag-test-matrix-reference
Feature-flag test matrix design: the flag-state combinatorics problem (N flags × M variants × K user-segments = N×M×K test cases), the canonical coverage strategies (pairwise interaction coverage; default-only smoke; full matrix; risk-driven matrix), the workflow for building the coverage suite from a flag inventory (grep-based inventory, per-flag classification, PICT pairwise generation, per-cell test skeletons), the dedicated kill-switch test categories (references/killswitch.md: graceful degradation, fail-static default, kill latency, mid-flight consistency), and the flags-vs-experiments distinction. Use when designing the flag-test surface for a new project, building or auditing flag-test coverage, or authoring kill-switch tests.
launchdarkly-testing
Wraps LaunchDarkly server-side SDK testing patterns: TestData data source for hermetic tests (no network), file-based data source for fixture-driven tests, flag override patterns (TestData.update for per-test flag values), and assignment-integrity tests. Use when writing tests for code that uses LaunchDarkly flags; to decide which flag combinations those tests should cover in the first place, see feature-flag-test-matrix-reference.