statsig-test
Wraps Statsig SDK testing patterns - server-side statsig.initialize with an API key, gate / experiment / dynamic-config evaluation (checkGate, getExperiment, getConfig), local-evaluation offline mode, overrideGate / overrideConfig to force a user into an arm, assignment-integrity tests. Use when writing tests for Statsig-instrumented application code. For another experimentation SDK use the matching harness - optimizely-test, vwo-test, amplitude-experiment-test, or split-io-test; for experiment DESIGN gates not SDK code use ab-test-validity-checklist.
Install with skills.sh (any agent)
npx skills add testland/qa --skill statsig-teststatsig-test
Overview
Per docs.statsig.com (opens in new window), the Statsig SDK is available for Node.js, Java, Python, Go, Ruby, .NET, PHP, Rust, and C++ - all with the same conceptual surface: gates, experiments, dynamic configs.
When to use
Authoring
Install
npm install --save-dev statsig-node # Node
pip install statsig # PythonInitialize for testing
import statsig from 'statsig-node';
beforeAll(async () => {
await statsig.initialize(process.env.STATSIG_SERVER_KEY!, {
localMode: true, // No network; all gates / configs return defaults
});
});
afterAll(async () => {
await statsig.shutdown();
});localMode: true is the test-mode flag - gates fall back to the default value, configs return empty, no network calls.
Override gates / experiments per user
test('user in treatment sees new UI', async () => {
statsig.overrideGate('new_ui_gate', true, 'user-1');
const enabled = await statsig.checkGate({ userID: 'user-1' }, 'new_ui_gate');
expect(enabled).toBe(true);
const disabledForOthers = await statsig.checkGate({ userID: 'user-2' }, 'new_ui_gate');
expect(disabledForOthers).toBe(false);
});statsig.overrideGate(gateName, value, userID) - pin a user to a value for the lifetime of the test.
Experiment evaluation
test('user in arm B sees increased font size', async () => {
statsig.overrideConfig('font_size_experiment', { font_size: 20 }, 'user-1');
const exp = await statsig.getExperiment({ userID: 'user-1' }, 'font_size_experiment');
expect(exp.value).toEqual({ font_size: 20 });
});Assignment integrity tests
Per ab-test-validity-checklist Step 3:
test('same user always gets same arm (determinism)', async () => {
const arm1 = await statsig.getExperiment({ userID: 'user-1' }, 'exp-x');
const arm2 = await statsig.getExperiment({ userID: 'user-1' }, 'exp-x');
expect(arm1.value).toEqual(arm2.value);
});
test('different users may get different arms', async () => {
const arms = await Promise.all(
Array.from({ length: 100 }, (_, i) =>
statsig.getExperiment({ userID: `user-${i}` }, 'exp-x').then(e => e.value)
)
);
const uniqueArms = new Set(arms.map(a => JSON.stringify(a)));
expect(uniqueArms.size).toBeGreaterThan(1);
});Exposure event firing
Statsig fires an exposure event per evaluation by default; verify in tests:
test('exposure logged on evaluation', async () => {
const events: any[] = [];
// Statsig SDK exposes a hook for testing event logging
statsig.flush(); // Force flush any pending events
// Inspect via test mock of the event-uploader
});Running
npm testFor CI, use STATSIG_SERVER_KEY set to a test-tier key OR rely on localMode: true for fully-offline tests.
CI integration
jobs:
statsig-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/setup-node@v4
- run: npm ci
- run: npm test
env:
STATSIG_SERVER_KEY: ${{ secrets.STATSIG_TEST_KEY }}Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Tests using production Statsig API key | Production traffic polluted | Per-env keys; or localMode: true |
Skipping statsig.shutdown() | Pending event upload leaks | Always shutdown |
| Asserting on exact internal config IDs | Statsig config IDs change | Assert on returned values |
| Tests rely on real evaluation (no override) | Flaky if Statsig service changes | Override per test |
Forgetting userID in evaluation | Returns default; not the test you wrote | Always pass full user object |
| Sharing one Statsig instance across test files | Override leaks | Per-test cleanup |
Limitations
References
Related skills
ab-test-validity-checklist
Workflow skill that builds an A/B-test validity checklist from an experiment proposal, walking the canonical design-correctness gates - pre-registered OEC/power/guardrails, randomization unit + SRM check, assignment integrity, telemetry, peeking discipline, novelty/primacy, post-experiment SRM re-check - into a per-experiment checklist + sign-off form. Use when launching, auditing, or governing an experiment. For pitfall mechanics use guardrail-metrics-reference or peeking-problem-reference; to read an already-valid result use experiment-results-interpreter; for per-SDK harness tests use optimizely-test or statsig-test - this gates DESIGN, not SDK code.
amplitude-experiment-test
Wraps Amplitude Experiment SDK testing patterns: client initialization with API key (or a bootstrapped local flag config for offline tests), the fetch / variant API, exposure-event suppression in tests, and assignment-integrity tests. Use when writing tests for code that uses Amplitude Experiment for A/B testing or flag management.
experiment-results-interpreter
Interprets the results of a valid online controlled experiment, one whose harness, SRM, and telemetry have already been confirmed. Covers the distinction between practical and statistical significance, reading confidence intervals instead of binary p-values, novelty and primacy week-over-week decay that causes post-ship reversion, interaction effects from concurrent experiments, Simpson's paradox in segmented results, and the ordered guardrail-check sequence required before a ship decision. Use when a data scientist or PM is ready to draw conclusions from an experiment whose telemetry and randomisation have already passed the ab-test-validity-checklist. Distinct from ab-test-validity-checklist (harness setup and SRM detection) and from interaction-effect overlap auditing during experiment design.
guardrail-metrics-reference
Pure-reference catalog of guardrail-metric methodology for online controlled experiments. Defines guardrail metrics (metrics that must NOT degrade for an experiment to ship, even if the primary metric improves), the standard guardrail set (latency / errors / engagement / opt-out), the relationship to OEC (Overall Evaluation Criterion) per Kohavi et al., and pre-commitment of the metric set. The quantitative evaluation mechanics (per-metric alert/block thresholds, Bonferroni / Benjamini-Hochberg multiple-comparison correction) live in references/. Use when designing the metric set for a new experiment, auditing existing experiment configs, or reviewing experiment results before ship-decisions.
optimizely-test
Wraps Optimizely Feature Experimentation SDK testing patterns - client init from a fixture datafile (offline-friendly), the decide / decideAll v5 API, forced-decisions for per-test arm pinning (fixing which variation a user gets), OptimizelyUserContext + activate/track events, assignment-integrity (deterministic bucketing) tests. Use when writing A/B tests or feature-flag tests for Optimizely-instrumented application code. For another experimentation SDK use the matching harness - statsig-test, vwo-test, amplitude-experiment-test, or split-io-test; for experiment DESIGN gates not SDK code use ab-test-validity-checklist.
peeking-problem-reference
Pure-reference catalog of the peeking problem in online A/B testing. Defines the problem (repeatedly looking at experiment results inflates the false-positive rate above the declared alpha because each look is a separate test), the canonical mitigations (fixed-horizon test with pre-declared sample size; sequential testing with alpha-spending functions e.g., O'Brien-Fleming, Pocock; always-valid inference / mSPRT per Johari et al.), and the policy choices (data-peek schedule, stop-early thresholds, decision-time guard rails). Use when designing an experimentation platform's stop-early policy or auditing why a result was declared significant.
split-io-test
Wraps Split.io (Harness FME) SDK testing patterns: hermetic localhost/offline mode with an in-memory features map (JavaScript/browser) or a YAML fixture file (Node.js server-side), getTreatment and getTreatmentWithConfig evaluation, the SDK_READY event and whenReady() promise, impression listener verification, sync.impressionsMode configuration, and CI setup. Use when writing tests for application code instrumented with the Split.io or Harness Feature Management & Experimentation SDK.
vwo-test
Wraps VWO (Visual Website Optimizer) SDK testing patterns: SDK initialization with the settings file (offline-capable), `getFeatureVariableValue` and `activate` API, force-bucketing for per-test assignment, and assignment-integrity tests against the bucketing algorithm. Use when writing tests for VWO-instrumented application code.