Amplitude Experiment SDK testing

Per amplitude.com/docs/experiment (opens in new window), the Amplitude Experiment SDKs (server-side and client-side) expose fetch + variant APIs: fetch the user's assigned variants, then read each variant on demand.

Amplitude correlates exposure + outcome events via the same user ID space as Amplitude Analytics, so exposure-event suppression in tests is important to avoid polluting analytics.

When to use

• Tests for code that reads an Amplitude Experiment variant.
• Suppressing exposure events in non-production test runs.
• Assignment-integrity tests per ab-test-validity-checklist Step 3.

Authoring

Install

pip install amplitude-experiment           # Python (server-side)
npm install --save-dev @amplitude/experiment-node-server

Initialize (server-side)

import * as Experiment from '@amplitude/experiment-node-server';

const client = Experiment.Experiment.initializeRemote(API_KEY, {
  // Suppress real fetches in tests
  fetchTimeoutMillis: 1000,
});

For fully-offline tests, use the local evaluation mode and seed the flag config via the bootstrap option. Per the local-evaluation docs (opens in new window), start() takes no arguments and always performs an initial network fetch (it throws offline and would clear a bootstrapped cache), so do NOT call it for a no-network test: bootstrap populates the cache in the constructor.

import { LocalEvaluationClient } from '@amplitude/experiment-node-server';
import { readFileSync } from 'fs';

// Commit the flag config the flags endpoint would return, keyed by flag key.
const flagFixture = JSON.parse(readFileSync('fixtures/flags.json', 'utf8'));

const localClient = new LocalEvaluationClient(API_KEY, {
  bootstrap: flagFixture,   // seeds the cache; no start() / no network
});

Read variant (offline, synchronous)

evaluateV2 reads straight from the bootstrapped cache, no fetch required:

const user = { user_id: 'user-1', device_id: 'dev-1' };

test('user variant from local eval', () => {
  const variants = localClient.evaluateV2(user);
  expect(variants['checkout-experiment'].value).toBe('treatment-a');
});

Force a variant for a test

Amplitude Experiment's standard pattern is via the flag config: override the flag's default-variant for a specific user ID by modifying the local-eval fixture. Alternatively, mock the evaluate method:

import { jest } from '@jest/globals';

test('user in treatment', () => {
  jest.spyOn(localClient, 'evaluateV2').mockReturnValue({
    'checkout-experiment': { value: 'treatment-a' } as any,
  });

  const variants = localClient.evaluateV2(user);
  expect(variants['checkout-experiment'].value).toBe('treatment-a');
});

Suppress exposure events in tests

Default behavior fires an exposure event on variant() read. Suppress per amplitude.com/docs/experiment (opens in new window):

// In test setup:
const client = Experiment.Experiment.initializeRemote(API_KEY, {
  // Disable automatic exposure tracking
  automaticExposureTracking: false,
});

Assignment integrity tests

test('deterministic assignment', () => {
  const v1 = localClient.evaluateV2({ user_id: 'user-1' });
  const v2 = localClient.evaluateV2({ user_id: 'user-1' });
  expect(v1).toEqual(v2);
});

Running

npm test

CI integration

jobs:
  amplitude-experiment-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-node@v4
      - run: npm ci
      - run: npm test
        env:
          AMPLITUDE_API_KEY: ${{ secrets.AMPLITUDE_TEST_KEY }}

For fully-offline CI: skip the env var and use local-eval with checked-in flag config JSON.

Anti-patterns

Limitations

• Local-evaluation mode is feature-limited. Some flag types (CMAB, multi-armed bandit) aren't supported offline.
• Mocking variant() loses targeting-rule fidelity. Use real local-eval when targeting matters.
• Exposure suppression is binary. Can't selectively suppress per-test.
• Doesn't validate Amplitude's results analysis. Platform-side statistics separate.

References

• Amplitude Experiment docs: amplitude.com/docs/experiment (opens in new window).
• Local evaluation: amplitude.com/docs/experiment/general/evaluation/local-evaluation (opens in new window).
• Companion: ab-test-validity-checklist; result-trust references (peeking, guardrails) live in experiment-results-interpreter.
• Sibling vendor references: statsig.md (opens in new window), optimizely.md (opens in new window), split-io.md (opens in new window), vwo.md (opens in new window).

Optimizely extended test recipes

Deeper test recipes for optimizely.md (opens in new window). Its spine covers install, datafile init, user context + decide, and forced decisions; this file holds the assignment-integrity and event-tracking tests plus the run/CI wiring.

Assignment integrity

Same user id must resolve to the same variation, and decide_all must return a stable key set across calls.

def test_assignment_deterministic():
    user_a1 = client.create_user_context("user-1")
    user_a2 = client.create_user_context("user-1")
    d1 = user_a1.decide("flag-x")
    d2 = user_a2.decide("flag-x")
    assert d1.variation_key == d2.variation_key

def test_decide_all_returns_consistent_set():
    user = client.create_user_context("user-1")
    decisions_1 = user.decide_all()
    decisions_2 = user.decide_all()
    assert decisions_1.keys() == decisions_2.keys()

Event tracking

Assert conversion events via the notification listener rather than inspecting network calls.

def test_conversion_event_emitted():
    captured_events = []
    # Optimizely supports a notification listener
    client.notification_center.add_notification_listener(
        notification_type="TRACK", notification_callback=lambda *args: captured_events.append(args)
    )

    user = client.create_user_context("user-1")
    user.track_event("checkout_completed")
    assert len(captured_events) == 1

Running

pytest tests/optimizely/

CI integration

jobs:
  optimizely-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-python@v5
      - run: pip install -e ".[test]"
      - run: pytest tests/optimizely/

Datafile lives in the repo; no SDK key needed for tests.

Optimizely SDK testing

Optimizely Feature Experimentation (Optimizely Full Stack / Optimizely X) uses a datafile - a JSON blob describing all flags, experiments, and audiences - that the SDK fetches and evaluates locally. Per docs.developers.optimizely.com/feature-experimentation/docs/python-sdk (opens in new window), the SDK supports datafile-based testing: load a fixture datafile in tests, no network call.

The current API surface is decide (single flag) / decideAll (all flags) per the v5 SDK.

When to use

• Tests for code that reads an Optimizely flag / experiment.
• Datafile-based snapshot tests for assignment matrices.
• Forced-decisions for per-test arm pinning.

How to use

1. Install the Optimizely SDK for your language (pip install optimizely-sdk, or the npm package).
2. Export the datafile from the Optimizely UI or API and commit it as a fixture under tests/fixtures/.
3. Initialize the client offline from that datafile - no SDK key, no network call.
4. Create an OptimizelyUserContext per test with the attributes the audience targeting needs.
5. Verify: assert the fixture drives a real decision - user.decide("new_checkout_flow").enabled is True - before pinning arms. If it fails, the datafile is stale or missing the flag; re-export it and re-run.
6. Pin arms with set_forced_decision where a test needs a fixed variation, then call decide (one flag) or decide_all (all flags).
7. Assert on variation_key / enabled (never variation IDs); add assignment-integrity and event-tracking checks via the notification listener (see optimizely-recipes.md (opens in new window)).
8. Run under pytest in CI; re-export the datafile periodically to catch drift against prod config.

Authoring

Install

pip install optimizely-sdk           # Python
npm install --save-dev @optimizely/optimizely-sdk

Datafile-based initialization

Per Optimizely docs, the datafile path is the canonical offline approach:

import json
from optimizely import optimizely

# Load a checked-in datafile fixture
with open("tests/fixtures/optimizely-datafile.json") as f:
    datafile = json.load(f)

client = optimizely.Optimizely(json.dumps(datafile))

The datafile is downloadable from the Optimizely UI or via the Optimizely API; commit a version-specific copy to the repo for deterministic tests.

Create a user context

def test_get_decision_for_user():
    user = client.create_user_context("user-1", {"plan": "premium"})
    decision = user.decide("new_checkout_flow")
    assert decision.enabled is True
    assert decision.variation_key == "treatment_a"

Forced decisions for per-test pinning

from optimizely.optimizely_user_context import OptimizelyDecisionContext

def test_force_user_to_treatment():
    user = client.create_user_context("user-1")
    context = OptimizelyDecisionContext(flag_key="new_checkout_flow", rule_key=None)
    user.set_forced_decision(context, OptimizelyForcedDecision(variation_key="treatment_a"))

    decision = user.decide("new_checkout_flow")
    assert decision.variation_key == "treatment_a"

Extended recipes - assignment-integrity and event-tracking tests, the pytest run command, and CI integration yaml - are in optimizely-recipes.md (opens in new window).

Worked example

The team ships a new_checkout_flow flag with a treatment_a variation, and QA needs a deterministic test that a premium-plan user is routed into the treatment. Follow the How-to-use steps: export the fixture, init offline, create the {"plan": "premium"} context, assert decide("new_checkout_flow") returns enabled is True / variation_key == "treatment_a", then pin the arm with set_forced_decision for the variant-specific path.

Result: a deterministic pass/fail on the checkout-routing logic with no network round-trip and no SDK key - the fixture drives the whole decision.

Anti-patterns

Limitations

• Datafile is point-in-time. Drift between fixture + prod is invisible. Sync periodically.
• Stickiness depends on bucketing UUID. If a user's bucketing ID changes (e.g., from anonymous → logged-in), the assignment may change.
• Forced decisions are per-context. Across multiple user- contexts in the same test, you may need to re-force.
• Doesn't test Optimizely's results analysis. Platform-side statistics are separate.

References

• Optimizely Python SDK: docs.developers.optimizely.com/feature-experimentation/docs/python-sdk (opens in new window).
• Forced decisions: docs.developers.optimizely.com/feature-experimentation/docs/forced-decision-methods (opens in new window).
• Companion: ab-test-validity-checklist; result-trust references (peeking, guardrails) live in experiment-results-interpreter.
• Sibling vendor references: statsig.md (opens in new window), split-io.md (opens in new window), amplitude.md (opens in new window), vwo.md (opens in new window).

End-to-end example

Full Node.js test (Jest) for a checkout feature guarded by two flags, using localhost/offline mode. See split-io.md (opens in new window) for each pattern in isolation.

import path from 'path';
import { SplitFactory } from '@splitsoftware/splitio';

let factory: SplitIO.ISDK;
let client: SplitIO.IClient;

beforeAll(async () => {
  factory = SplitFactory({
    core: { authorizationKey: 'localhost' },
    features: path.join(__dirname, '__fixtures__/split-flags.yml'),
    sync: { impressionsMode: 'NONE' },
  });
  client = factory.client('user-123');
  await client.whenReady();
});

afterAll(async () => {
  await client.destroy();
});

describe('checkout page feature flags', () => {
  it('shows redesigned checkout for on-treatment users', () => {
    expect(client.getTreatment('user-123', 'checkout_redesign')).toBe('on');
  });

  it('returns config for pricing experiment arm', () => {
    const result = client.getTreatmentWithConfig('user-123', 'pricing_experiment');
    expect(result.treatment).toBe('v2');
    expect(JSON.parse(result.config!)).toMatchObject({ price: 9 });
  });

  it('returns control for an undeclared flag', () => {
    expect(client.getTreatment('user-123', 'unrelated_flag')).toBe('control');
  });
});

The __fixtures__/split-flags.yml file is committed alongside the test. No SPLIT_API_KEY secret is required in CI when using localhost mode.

Split.io (Harness FME) SDK testing

The Split.io SDK (now Harness Feature Management & Experimentation, FME) keeps the surface SplitFactory, getTreatment, getTreatmentWithConfig, and the event system. Both the JavaScript (browser) and Node.js (server-side) SDKs support a localhost/offline mode that eliminates all network calls during tests, making feature-flag evaluation fully hermetic. This skill covers both; all SDK behavior cited below is drawn from the FME SDK docs (see References).

Differentiation from the sibling vendor references: statsig.md (opens in new window) uses localMode: true (gate/config primitives); optimizely.md (opens in new window) uses a JSON datafile; amplitude.md (opens in new window) uses a local-eval JSON fixture. Split.io is distinct: Split.io's offline mechanism uses authorizationKey: 'localhost' paired with an in-memory features map (JS SDK) or a YAML/text fixture file (Node.js SDK), and its evaluation API is getTreatment / getTreatmentWithConfig rather than decide, variant, or checkGate.

When to use

• Tests for code that calls client.getTreatment() or client.getTreatmentWithConfig() against a Split.io-instrumented surface.
• Verifying that impression listeners fire correctly per SDK evaluation.
• Assignment-integrity tests per ab-test-validity-checklist Step 3.
• CI pipelines where network access to Split.io / Harness is unavailable or undesirable.

How to use

Install

npm install --save-dev @splitsoftware/splitio   # Node.js + browser

Localhost/offline mode - JavaScript (browser) SDK

Set authorizationKey to 'localhost' and supply a features map:

import { SplitFactory } from '@splitsoftware/splitio';

const factory = SplitFactory({
  core: {
    authorizationKey: 'localhost',
    key: 'test-user-1',
  },
  features: {
    'checkout_redesign':  'on',
    'dark_mode':          'off',
    'pricing_experiment': { treatment: 'v2', config: '{"price":9}' },
  },
  scheduler: {
    offlineRefreshRate: 15,   // seconds between simulated polls
  },
});
const client = factory.client();

Any flag absent from the features map returns the 'control' treatment automatically - no extra setup needed for flags the test does not care about.

Localhost/offline mode - Node.js (server-side) SDK

The server-side SDK reads offline fixtures from a file path. Use a YAML fixture (supported since SDK v10.7.0) for per-key targeting:

# tests/fixtures/split-flags.yml
- checkout_redesign:
    treatment: "on"
    keys: "test-user-1"
    config: "{}"
- checkout_redesign:
    treatment: "off"
- dark_mode:
    treatment: "off"

import path from 'path';
import { SplitFactory } from '@splitsoftware/splitio';

const factory = SplitFactory({
  core: { authorizationKey: 'localhost' },
  features: path.join(__dirname, 'fixtures/split-flags.yml'),
  scheduler: { offlineRefreshRate: 15 },
});
const client = factory.client('test-user-1');

The plain-text format (two whitespace-separated columns) is also supported for simpler cases where per-key targeting is not needed.

SDK_READY event and whenReady()

The SDK emits client.Event.SDK_READY when its data is loaded. Always wait for this event before evaluating treatments to avoid receiving 'control' prematurely:

// Event-listener style
client.on(client.Event.SDK_READY, () => {
  const treatment = client.getTreatment('checkout_redesign');
  expect(treatment).toBe('on');
});

Or use the promise-based equivalent (cleaner in async test bodies):

// Promise style
beforeAll(async () => {
  await client.whenReady();
});

Additional events:

• SDK_READY_TIMED_OUT: timeout before data loaded; SDK may still become ready later
• SDK_UPDATE: rollout plan changed (useful in localhost mode to test dynamic flag flips)

getTreatment and getTreatmentWithConfig

getTreatment returns a treatment string; getTreatmentWithConfig returns { treatment: string, config: string | null }:

test('user in treatment arm sees new pricing', async () => {
  await client.whenReady();

  const treatment = client.getTreatment('test-user-1', 'pricing_experiment');
  expect(treatment).toBe('v2');

  const result = client.getTreatmentWithConfig(
    'test-user-1',
    'pricing_experiment'
  );
  expect(result.treatment).toBe('v2');
  expect(JSON.parse(result.config!)).toEqual({ price: 9 });
});

test('unknown flag returns control', async () => {
  await client.whenReady();
  const treatment = client.getTreatment('test-user-1', 'nonexistent_flag');
  expect(treatment).toBe('control');
});

Note: the Node.js SDK's getTreatment takes (key, flagName, attributes?, evaluationOptions?). The JavaScript client SDK's getTreatment takes (flagName, attributes?) because the key is bound at factory.client(key) construction time.

Impression listener verification

Attach an impressionListener to SplitFactory options. The logImpression callback receives an object containing impression (feature flag, key, treatment, label), attributes, ip, hostname, and sdkLanguageVersion:

const impressions: any[] = [];

const factory = SplitFactory({
  core: { authorizationKey: 'localhost' },
  features: path.join(__dirname, 'fixtures/split-flags.yml'),
  impressionListener: {
    logImpression(data) {
      impressions.push(data);
    },
  },
});
const client = factory.client('test-user-1');

test('impression fires on getTreatment', async () => {
  impressions.length = 0;
  await client.whenReady();

  client.getTreatment('test-user-1', 'checkout_redesign');

  expect(impressions).toHaveLength(1);
  expect(impressions[0].impression.treatment).toBe('on');
});

Controlling impression mode

sync.impressionsMode has three values:

• 'OPTIMIZED' (default): only unique impressions queued; reduces traffic; suitable for production and most test scenarios.
• 'DEBUG': all impressions queued and sent; use when validating that every evaluation generates a record.
• 'NONE': no impressions tracked; use for flag-only (non-experiment) use cases in CI where impression noise is unwanted.

const factory = SplitFactory({
  core: { authorizationKey: 'localhost' },
  features: path.join(__dirname, 'fixtures/split-flags.yml'),
  sync: { impressionsMode: 'DEBUG' },
});

Assignment integrity tests

Per ab-test-validity-checklist Step 3, verify that the same key always receives the same treatment and that different keys can receive different treatments:

test('same key always gets same treatment (determinism)', async () => {
  await client.whenReady();
  const t1 = client.getTreatment('test-user-1', 'checkout_redesign');
  const t2 = client.getTreatment('test-user-1', 'checkout_redesign');
  expect(t1).toBe(t2);
});

test('treatments match the features map (hermetic)', async () => {
  await client.whenReady();
  expect(client.getTreatment('test-user-1', 'checkout_redesign')).toBe('on');
  // Different key receives the default treatment (no per-key override)
  expect(client.getTreatment('test-user-2', 'checkout_redesign')).toBe('off');
});

Teardown

Call client.destroy() (returns a promise) after tests to release internal resources. Skipping it leaks event-listener handles and impression flush timers:

afterAll(async () => {
  await client.destroy();
});

After destroy() is called, subsequent getTreatment calls return 'control' and factory operations require re-instantiation.

CI integration

jobs:
  split-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm ci
      - run: npm test
        # No SPLIT_API_KEY needed: localhost mode is fully offline

The YAML fixture is committed alongside the test code. No SPLIT_API_KEY secret is required in CI when using localhost mode.

Example

A full end-to-end Node.js/Jest test for a checkout feature guarded by two flags is in split-io-example.md (opens in new window).

Anti-patterns

Limitations

• No arm-pinning override API. Unlike Statsig's overrideGate or Optimizely's setForcedDecision, the Split.io SDK has no per-user override call. Pin treatments by adding a per-key YAML entry in the fixture file or by supplying the exact key that maps to the desired treatment.
• Localhost mode is in-process only. It does not work across multiple processes (e.g., a forked worker) unless each process initializes its own factory with the same fixture.
• YAML per-key targeting requires Node.js SDK v10.7.0+. Earlier versions only support the plain-text two-column format with uniform treatment per flag.
• Dynamic feature-map mutations (JS SDK only). Mutating the features object at runtime triggers SDK_UPDATE and simulates a rollout change. The Node.js file-based fixture does not support runtime mutation without reloading.
• Does not validate Split.io's server-side statistical analysis. Platform analysis is the vendor's responsibility; this skill tests your code's interaction with the SDK only.

References

• Node.js SDK docs (Harness FME): developer.harness.io/docs/feature-management-experimentation/sdks-and-infrastructure/server-side-sdks/nodejs-sdk (opens in new window)
• JavaScript SDK docs (Harness FME): developer.harness.io/docs/feature-management-experimentation/sdks-and-infrastructure/client-side-sdks/javascript-sdk (opens in new window)
• Companion: ab-test-validity-checklist; result-trust references (peeking, guardrails) live in experiment-results-interpreter
• Sibling vendor references: statsig.md (opens in new window), optimizely.md (opens in new window), amplitude.md (opens in new window), vwo.md (opens in new window)

Statsig SDK testing

Per docs.statsig.com (opens in new window), the Statsig SDK is available for Node.js, Java, Python, Go, Ruby, .NET, PHP, Rust, and C++ - all with the same conceptual surface: gates, experiments, dynamic configs.

When to use

• Tests for code that reads a Statsig gate / experiment.
• Verifying assignment integrity per ab-test-validity-checklist Step 3.
• Local-evaluation tests when network access is restricted.

Authoring

Install

npm install --save-dev statsig-node       # Node
pip install statsig                        # Python

Initialize for testing

import statsig from 'statsig-node';

beforeAll(async () => {
  await statsig.initialize(process.env.STATSIG_SERVER_KEY!, {
    localMode: true,    // No network; all gates / configs return defaults
  });
});

afterAll(async () => {
  await statsig.shutdown();
});

localMode: true is the test-mode flag - gates fall back to the default value, configs return empty, no network calls.

Override gates / experiments per user

test('user in treatment sees new UI', async () => {
  statsig.overrideGate('new_ui_gate', true, 'user-1');

  const enabled = await statsig.checkGate({ userID: 'user-1' }, 'new_ui_gate');
  expect(enabled).toBe(true);

  const disabledForOthers = await statsig.checkGate({ userID: 'user-2' }, 'new_ui_gate');
  expect(disabledForOthers).toBe(false);
});

statsig.overrideGate(gateName, value, userID) - pin a user to a value for the lifetime of the test.

Experiment evaluation

test('user in arm B sees increased font size', async () => {
  statsig.overrideConfig('font_size_experiment', { font_size: 20 }, 'user-1');

  const exp = await statsig.getExperiment({ userID: 'user-1' }, 'font_size_experiment');
  expect(exp.value).toEqual({ font_size: 20 });
});

Assignment integrity tests

Per ab-test-validity-checklist Step 3:

test('same user always gets same arm (determinism)', async () => {
  const arm1 = await statsig.getExperiment({ userID: 'user-1' }, 'exp-x');
  const arm2 = await statsig.getExperiment({ userID: 'user-1' }, 'exp-x');
  expect(arm1.value).toEqual(arm2.value);
});

test('different users may get different arms', async () => {
  const arms = await Promise.all(
    Array.from({ length: 100 }, (_, i) =>
      statsig.getExperiment({ userID: `user-${i}` }, 'exp-x').then(e => e.value)
    )
  );
  const uniqueArms = new Set(arms.map(a => JSON.stringify(a)));
  expect(uniqueArms.size).toBeGreaterThan(1);
});

Exposure event firing

Statsig fires an exposure event per evaluation by default; verify in tests:

test('exposure logged on evaluation', async () => {
  const events: any[] = [];
  // Statsig SDK exposes a hook for testing event logging
  statsig.flush();  // Force flush any pending events
  // Inspect via test mock of the event-uploader
});

Running

npm test

For CI, use STATSIG_SERVER_KEY set to a test-tier key OR rely on localMode: true for fully-offline tests.

CI integration

jobs:
  statsig-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v4
      - run: npm ci
      - run: npm test
        env:
          STATSIG_SERVER_KEY: ${{ secrets.STATSIG_TEST_KEY }}

Anti-patterns

Limitations

• localMode is most-but-not-fully-offline. Some SDK initialization paths still ping Statsig.
• No deterministic arm assignment without override. Test randomness via override; for hash-stability use the network-mode SDK.
• Overrides are SDK-instance scoped. Don't help if the app spawns multiple processes.
• Doesn't validate Statsig's server-side analysis. That's the platform's responsibility; tests verify your code's interaction with the SDK.

References

• Statsig docs: docs.statsig.com/ (opens in new window).
• Statsig SDK reference (per language): docs.statsig.com/server-core/ (opens in new window).
• Companion: ab-test-validity-checklist; result-trust references (peeking, guardrails) live in experiment-results-interpreter.
• Sibling vendor references: optimizely.md (opens in new window), split-io.md (opens in new window), amplitude.md (opens in new window), vwo.md (opens in new window).

VWO SDK testing

The VWO (Visual Website Optimizer) server-side SDK uses a settings file (equivalent to Optimizely's datafile) for offline-capable testing. Per developers.vwo.com (opens in new window), the SDK supports multiple languages with a common API: activate, get_feature_variable_value, is_feature_enabled, track, push.

When to use

• Tests for code that reads a VWO feature variable / experiment.
• Force-bucketing for per-test assignment pinning.
• Assignment-integrity tests per ab-test-validity-checklist Step 3.

Authoring

Install

pip install vwo-python-sdk
npm install --save-dev vwo-node-sdk

Settings-file-based init

import json
import vwo

with open("tests/fixtures/vwo-settings.json") as f:
    settings = json.load(f)

client = vwo.launch(settings, is_development_mode=True)

is_development_mode=True disables event tracking to VWO servers - fully-offline tests.

Activate an experiment / variation

def test_get_variation_for_user():
    variation_name = client.activate("checkout-experiment", "user-1")
    assert variation_name in ("Control", "Variation-1")

Feature variable

def test_feature_variable_value():
    value = client.get_feature_variable_value("checkout-experiment", "button_color", "user-1")
    assert value in ("blue", "green")

Force-bucket a user

VWO doesn't have a direct "force decision" API like Optimizely; the canonical approach is to construct user IDs that hash into specific buckets - or use the SDK's userPreSegment callback where supported.

Per VWO docs, the bucketing is deterministic on the user ID. Tests rely on this:

def test_specific_user_id_in_treatment():
    # User IDs are bucketed deterministically; pre-compute and pin
    KNOWN_TREATMENT_USER = "test-user-treatment-12345"
    variation = client.activate("checkout-experiment", KNOWN_TREATMENT_USER)
    assert variation == "Variation-1"

A pre-test step generates user IDs and records their bucket assignments in a fixture; tests reference the fixture.

Assignment integrity tests

def test_same_user_always_same_variation():
    v1 = client.activate("expt", "user-1")
    v2 = client.activate("expt", "user-1")
    assert v1 == v2

def test_bucketing_is_uniform():
    counts = {"Control": 0, "Variation-1": 0}
    for i in range(10000):
        v = client.activate("expt", f"user-{i}")
        if v: counts[v] += 1
    # 50/50 split → within a few percent
    ratio = counts["Variation-1"] / sum(counts.values())
    assert 0.48 < ratio < 0.52

The bucketing-uniformity test is also a unit-level SRM check per ab-test-validity-checklist Step 2.

Event tracking

def test_conversion_tracked():
    client.activate("expt", "user-1")
    success = client.track("expt", "user-1", "checkout_completed")
    assert success

Running

pytest tests/vwo/

CI integration

jobs:
  vwo-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-python@v5
      - run: pip install vwo-python-sdk
      - run: pytest tests/vwo/

Anti-patterns

Limitations

• No direct force-decision API. Must rely on deterministic bucketing or modify the user pre-segmentation.
• Settings file is point-in-time. Drift between fixture + prod invisible.
• Bucketing-uniformity tests need many user IDs. Below 1000 iterations the variance dominates.
• Event tracking is fire-and-forget in offline mode. Can't assert delivery; only the local-call success.

References

• VWO Python SDK: github.com/wingify/vwo-python-sdk (opens in new window).
• VWO developer docs: developers.vwo.com (opens in new window).
• Companion: ab-test-validity-checklist; result-trust references (peeking, guardrails) live in experiment-results-interpreter.
• Sibling vendor references: statsig.md (opens in new window), optimizely.md (opens in new window), split-io.md (opens in new window), amplitude.md (opens in new window).