Testland
Browse all skills & agents

statsig-test

Wraps Statsig SDK testing patterns - server-side statsig.initialize with an API key, gate / experiment / dynamic-config evaluation (checkGate, getExperiment, getConfig), local-evaluation offline mode, overrideGate / overrideConfig to force a user into an arm, assignment-integrity tests. Use when writing tests for Statsig-instrumented application code. For another experimentation SDK use the matching harness - optimizely-test, vwo-test, amplitude-experiment-test, or split-io-test; for experiment DESIGN gates not SDK code use ab-test-validity-checklist.

Install with skills.sh (any agent)

npx skills add testland/qa --skill statsig-test
View source

statsig-test

Overview

Per docs.statsig.com (opens in new window), the Statsig SDK is available for Node.js, Java, Python, Go, Ruby, .NET, PHP, Rust, and C++ - all with the same conceptual surface: gates, experiments, dynamic configs.

When to use

  • Tests for code that reads a Statsig gate / experiment.
  • Verifying assignment integrity per ab-test-validity-checklist Step 3.
  • Local-evaluation tests when network access is restricted.

Authoring

Install

npm install --save-dev statsig-node       # Node
pip install statsig                        # Python

Initialize for testing

import statsig from 'statsig-node';

beforeAll(async () => {
  await statsig.initialize(process.env.STATSIG_SERVER_KEY!, {
    localMode: true,    // No network; all gates / configs return defaults
  });
});

afterAll(async () => {
  await statsig.shutdown();
});

localMode: true is the test-mode flag - gates fall back to the default value, configs return empty, no network calls.

Override gates / experiments per user

test('user in treatment sees new UI', async () => {
  statsig.overrideGate('new_ui_gate', true, 'user-1');

  const enabled = await statsig.checkGate({ userID: 'user-1' }, 'new_ui_gate');
  expect(enabled).toBe(true);

  const disabledForOthers = await statsig.checkGate({ userID: 'user-2' }, 'new_ui_gate');
  expect(disabledForOthers).toBe(false);
});

statsig.overrideGate(gateName, value, userID) - pin a user to a value for the lifetime of the test.

Experiment evaluation

test('user in arm B sees increased font size', async () => {
  statsig.overrideConfig('font_size_experiment', { font_size: 20 }, 'user-1');

  const exp = await statsig.getExperiment({ userID: 'user-1' }, 'font_size_experiment');
  expect(exp.value).toEqual({ font_size: 20 });
});

Assignment integrity tests

Per ab-test-validity-checklist Step 3:

test('same user always gets same arm (determinism)', async () => {
  const arm1 = await statsig.getExperiment({ userID: 'user-1' }, 'exp-x');
  const arm2 = await statsig.getExperiment({ userID: 'user-1' }, 'exp-x');
  expect(arm1.value).toEqual(arm2.value);
});

test('different users may get different arms', async () => {
  const arms = await Promise.all(
    Array.from({ length: 100 }, (_, i) =>
      statsig.getExperiment({ userID: `user-${i}` }, 'exp-x').then(e => e.value)
    )
  );
  const uniqueArms = new Set(arms.map(a => JSON.stringify(a)));
  expect(uniqueArms.size).toBeGreaterThan(1);
});

Exposure event firing

Statsig fires an exposure event per evaluation by default; verify in tests:

test('exposure logged on evaluation', async () => {
  const events: any[] = [];
  // Statsig SDK exposes a hook for testing event logging
  statsig.flush();  // Force flush any pending events
  // Inspect via test mock of the event-uploader
});

Running

npm test

For CI, use STATSIG_SERVER_KEY set to a test-tier key OR rely on localMode: true for fully-offline tests.

CI integration

jobs:
  statsig-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v4
      - run: npm ci
      - run: npm test
        env:
          STATSIG_SERVER_KEY: ${{ secrets.STATSIG_TEST_KEY }}

Anti-patterns

Anti-patternWhy it failsFix
Tests using production Statsig API keyProduction traffic pollutedPer-env keys; or localMode: true
Skipping statsig.shutdown()Pending event upload leaksAlways shutdown
Asserting on exact internal config IDsStatsig config IDs changeAssert on returned values
Tests rely on real evaluation (no override)Flaky if Statsig service changesOverride per test
Forgetting userID in evaluationReturns default; not the test you wroteAlways pass full user object
Sharing one Statsig instance across test filesOverride leaksPer-test cleanup

Limitations

  • localMode is most-but-not-fully-offline. Some SDK initialization paths still ping Statsig.
  • No deterministic arm assignment without override. Test randomness via override; for hash-stability use the network-mode SDK.
  • Overrides are SDK-instance scoped. Don't help if the app spawns multiple processes.
  • Doesn't validate Statsig's server-side analysis. That's the platform's responsibility; tests verify your code's interaction with the SDK.

References

Related skills

ab-test-validity-checklist

Workflow skill that builds an A/B-test validity checklist from an experiment proposal, walking the canonical design-correctness gates - pre-registered OEC/power/guardrails, randomization unit + SRM check, assignment integrity, telemetry, peeking discipline, novelty/primacy, post-experiment SRM re-check - into a per-experiment checklist + sign-off form. Use when launching, auditing, or governing an experiment. For pitfall mechanics use guardrail-metrics-reference or peeking-problem-reference; to read an already-valid result use experiment-results-interpreter; for per-SDK harness tests use optimizely-test or statsig-test - this gates DESIGN, not SDK code.

amplitude-experiment-test

Wraps Amplitude Experiment SDK testing patterns: client initialization with API key (or a bootstrapped local flag config for offline tests), the fetch / variant API, exposure-event suppression in tests, and assignment-integrity tests. Use when writing tests for code that uses Amplitude Experiment for A/B testing or flag management.

experiment-results-interpreter

Interprets the results of a valid online controlled experiment, one whose harness, SRM, and telemetry have already been confirmed. Covers the distinction between practical and statistical significance, reading confidence intervals instead of binary p-values, novelty and primacy week-over-week decay that causes post-ship reversion, interaction effects from concurrent experiments, Simpson's paradox in segmented results, and the ordered guardrail-check sequence required before a ship decision. Use when a data scientist or PM is ready to draw conclusions from an experiment whose telemetry and randomisation have already passed the ab-test-validity-checklist. Distinct from ab-test-validity-checklist (harness setup and SRM detection) and from interaction-effect overlap auditing during experiment design.

guardrail-metrics-reference

Pure-reference catalog of guardrail-metric methodology for online controlled experiments. Defines guardrail metrics (metrics that must NOT degrade for an experiment to ship, even if the primary metric improves), the standard guardrail set (latency / errors / engagement / opt-out), the relationship to OEC (Overall Evaluation Criterion) per Kohavi et al., and pre-commitment of the metric set. The quantitative evaluation mechanics (per-metric alert/block thresholds, Bonferroni / Benjamini-Hochberg multiple-comparison correction) live in references/. Use when designing the metric set for a new experiment, auditing existing experiment configs, or reviewing experiment results before ship-decisions.

optimizely-test

Wraps Optimizely Feature Experimentation SDK testing patterns - client init from a fixture datafile (offline-friendly), the decide / decideAll v5 API, forced-decisions for per-test arm pinning (fixing which variation a user gets), OptimizelyUserContext + activate/track events, assignment-integrity (deterministic bucketing) tests. Use when writing A/B tests or feature-flag tests for Optimizely-instrumented application code. For another experimentation SDK use the matching harness - statsig-test, vwo-test, amplitude-experiment-test, or split-io-test; for experiment DESIGN gates not SDK code use ab-test-validity-checklist.

peeking-problem-reference

Pure-reference catalog of the peeking problem in online A/B testing. Defines the problem (repeatedly looking at experiment results inflates the false-positive rate above the declared alpha because each look is a separate test), the canonical mitigations (fixed-horizon test with pre-declared sample size; sequential testing with alpha-spending functions e.g., O'Brien-Fleming, Pocock; always-valid inference / mSPRT per Johari et al.), and the policy choices (data-peek schedule, stop-early thresholds, decision-time guard rails). Use when designing an experimentation platform's stop-early policy or auditing why a result was declared significant.

split-io-test

Wraps Split.io (Harness FME) SDK testing patterns: hermetic localhost/offline mode with an in-memory features map (JavaScript/browser) or a YAML fixture file (Node.js server-side), getTreatment and getTreatmentWithConfig evaluation, the SDK_READY event and whenReady() promise, impression listener verification, sync.impressionsMode configuration, and CI setup. Use when writing tests for application code instrumented with the Split.io or Harness Feature Management & Experimentation SDK.

vwo-test

Wraps VWO (Visual Website Optimizer) SDK testing patterns: SDK initialization with the settings file (offline-capable), `getFeatureVariableValue` and `activate` API, force-bucketing for per-test assignment, and assignment-integrity tests against the bucketing algorithm. Use when writing tests for VWO-instrumented application code.