optimizely-test
Wraps Optimizely Feature Experimentation SDK testing patterns: client initialization with a datafile (offline-friendly), the decide / decideAll API (Optimizely Feature Experimentation, the v5 API), forced-decisions for per-test arm pinning, OptimizelyUserContext + activate / track events, and assignment-integrity tests. Use when writing tests for Optimizely-instrumented application code. Composes guardrail-metrics-reference + peeking-problem-reference + ab-test-validity-checklist.
optimizely-test
Overview
Optimizely Feature Experimentation (Optimizely Full Stack / Optimizely X) uses a datafile - a JSON blob describing all flags, experiments, and audiences - that the SDK fetches and evaluates locally. Per docs.developers.optimizely.com/feature-experimentation/docs/python-sdk, the SDK supports datafile-based testing: load a fixture datafile in tests, no network call.
The current API surface is decide (single flag) / decideAll (all flags) per the v5 SDK.
When to use
Authoring
Install
pip install optimizely-sdk # Python
npm install --save-dev @optimizely/optimizely-sdkDatafile-based initialization
Per Optimizely docs, the datafile path is the canonical offline approach:
import json
from optimizely import optimizely
# Load a checked-in datafile fixture
with open("tests/fixtures/optimizely-datafile.json") as f:
datafile = json.load(f)
client = optimizely.Optimizely(json.dumps(datafile))The datafile is downloadable from the Optimizely UI or via the Optimizely API; commit a version-specific copy to the repo for deterministic tests.
Create a user context
def test_get_decision_for_user():
user = client.create_user_context("user-1", {"plan": "premium"})
decision = user.decide("new_checkout_flow")
assert decision.enabled is True
assert decision.variation_key == "treatment_a"Forced decisions for per-test pinning
from optimizely.optimizely_user_context import OptimizelyDecisionContext
def test_force_user_to_treatment():
user = client.create_user_context("user-1")
context = OptimizelyDecisionContext(flag_key="new_checkout_flow", rule_key=None)
user.set_forced_decision(context, OptimizelyForcedDecision(variation_key="treatment_a"))
decision = user.decide("new_checkout_flow")
assert decision.variation_key == "treatment_a"Assignment integrity
def test_assignment_deterministic():
user_a1 = client.create_user_context("user-1")
user_a2 = client.create_user_context("user-1")
d1 = user_a1.decide("flag-x")
d2 = user_a2.decide("flag-x")
assert d1.variation_key == d2.variation_key
def test_decide_all_returns_consistent_set():
user = client.create_user_context("user-1")
decisions_1 = user.decide_all()
decisions_2 = user.decide_all()
assert decisions_1.keys() == decisions_2.keys()Event tracking
def test_conversion_event_emitted():
captured_events = []
# Optimizely supports a notification listener
client.notification_center.add_notification_listener(
notification_type="TRACK", notification_callback=lambda *args: captured_events.append(args)
)
user = client.create_user_context("user-1")
user.track_event("checkout_completed")
assert len(captured_events) == 1Running
pytest tests/optimizely/CI integration
jobs:
optimizely-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/setup-python@v5
- run: pip install -e ".[test]"
- run: pytest tests/optimizely/Datafile lives in the repo; no SDK key needed for tests.
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Tests with live SDK key | Production data polluted; rate-limited | Use datafile fixture |
| Datafile not version-controlled | Tests flake when prod config changes | Commit the fixture |
| Forced decisions leak across tests | Cross-test pollution | Per-test user context; reset before assertion |
Skipping client.shutdown / network listener cleanup | Goroutine / handle leak | Always cleanup |
| Asserting on variation IDs not keys | IDs change per environment | Use variation_key |
| Manual event tracking in tests vs notification listener | Misses platform-emitted events | Use the listener |
| Tests rely on real decide-network roundtrip | Slow; non-deterministic | Datafile + offline |