feature-flag-experiment-validator

Overview

An experiment toggle A/B test routes each user into a cohort, measures behavior per cohort, and ships the winning variant. A/B tests are "sensitive to variance; they require a large sample size in order to reduce standard error and produce a statistically significant result" (ab-test-wiki (opens in new window); per-cohort toggle routing per feature-toggles (opens in new window)). Without proper analysis, teams ship variants that "look better" but aren't actually better. This skill validates the analysis.

When to use

• An A/B test has run for some time and the team wants the verdict.
• The team's analytics tool (Mixpanel, Amplitude, Statsig) reports a winner but the team wants an independent statistical check.
• A multi-metric experiment needs multiple-comparisons correction before a ship decision.
• A close-call experiment (treatment 2.1% better; p=0.06) needs rigorous interpretation.

Step 1 - Inputs

The validator needs, per variant:

# experiment-data.yml
experiment_id: checkout-promo-banner-v2
running_since: 2026-04-15
running_until: 2026-05-05    # 21 days
hypothesis: "Promo banner increases checkout completion."

variants:
  - name: control
    cohort_size: 12450
    metrics:
      checkout_completion_count: 8523
      avg_session_duration_sec: [...samples...]   # raw samples for continuous metrics
      avg_revenue_per_user: [...samples...]

  - name: treatment_a
    cohort_size: 12380
    metrics:
      checkout_completion_count: 8755
      avg_session_duration_sec: [...samples...]
      avg_revenue_per_user: [...samples...]

Per-metric, identify whether it's:

• Proportion (count / total - e.g. completion rate, sign-up rate, click-through rate).
• Continuous (latency, revenue per user, session duration, page count).

Different statistical tests apply.

Step 2 - Test per metric type

Proportions: chi-square or Fisher's exact

For "did the user convert? yes/no":

from scipy.stats import chi2_contingency

def proportion_test(c_success, c_total, t_success, t_total):
    """Returns (p_value, effect_size_pct)."""
    table = [[c_success, c_total - c_success],
             [t_success, t_total - t_success]]
    chi2, p, dof, _ = chi2_contingency(table)
    c_rate = c_success / c_total
    t_rate = t_success / t_total
    effect = (t_rate - c_rate) / c_rate * 100   # relative lift in %
    return p, effect

For very small cells (< 5 expected per cell), Fisher's exact is more accurate; chi-square otherwise.

Continuous: Welch's t-test or Mann-Whitney U

For "what's the average revenue per user?":

from scipy.stats import ttest_ind, mannwhitneyu

def continuous_test(c_samples, t_samples, parametric=True):
    """Welch's t-test (parametric) or Mann-Whitney U (non-parametric)."""
    if parametric:
        t, p = ttest_ind(c_samples, t_samples, equal_var=False)
    else:
        u, p = mannwhitneyu(c_samples, t_samples, alternative='two-sided')
    c_mean = sum(c_samples) / len(c_samples)
    t_mean = sum(t_samples) / len(t_samples)
    effect = (t_mean - c_mean) / c_mean * 100
    return p, effect

Use Mann-Whitney U when the metric isn't normally distributed (revenue per user - heavy right tail; latency - log-normal). Welch's t-test for approximately-normal metrics.

Step 3 - Multiple-comparisons correction

Per ab-test-wiki (opens in new window)'s "challenges" framing: testing many metrics inflates the false-positive rate. With α=0.05 and 10 independent metrics, P(at least one false positive) ≈ 1 - 0.95^10 = 40%.

Default: Benjamini-Hochberg FDR control - balances false-positive vs false-negative rates; controls the proportion of "wins" that are actually noise. Use Bonferroni when the cost of any false positive is catastrophic (regulatory / safety contexts) and over-conservatism is acceptable.

Benjamini-Hochberg (FDR control)

from statsmodels.stats.multitest import multipletests

reject, p_adj, _, _ = multipletests(p_values, alpha=0.05, method='fdr_bh')
# `reject[i]` is True when metric i is significant after FDR control.

Bonferroni (escape hatch - conservative)

adjusted_alpha = alpha / n_metrics    # e.g. 0.05 / 10 = 0.005
# Each metric must have p < 0.005 to be significant.

Over-conservative - increases false negatives.

For pre-registered single-primary-metric experiments, no correction needed for the primary; correction applies to secondary metrics.

Step 4 - Power analysis (was the experiment big enough?)

A non-significant result might mean "no effect" or "experiment too small." Compute post-hoc power:

from statsmodels.stats.power import NormalIndPower

def required_sample(effect_size, alpha=0.05, power=0.8):
    """How many users per variant to detect this effect with this power?"""
    analysis = NormalIndPower()
    return analysis.solve_power(effect_size=effect_size, alpha=alpha, power=power)

If the observed effect (e.g., 0.5% relative lift) requires N=50,000 users per variant for 80% power and the experiment had N=12,000, the experiment was under-powered. The verdict shouldn't be "no effect"; it should be "inconclusive - re-run at higher N or accept that we can't detect effects this small."

Step 5 - Practical vs statistical significance

A 0.1% lift can be statistically significant at N=10M; that doesn't mean the team should ship.

Define minimum detectable effect (MDE) per metric:

# mde.yml
checkout_completion_rate:
  mde_relative: 1.0   # 1% relative lift to be worth shipping
  mde_absolute: 0.5   # OR a 0.5pp absolute lift

avg_revenue_per_user:
  mde_absolute: 0.50  # $0.50/user; below this, ship cost > revenue

The verdict requires both statistical significance AND practical significance (effect ≥ MDE).

Step 6 - Output

Emit a per-metric results table (type, control, treatment, relative effect, raw and adjusted p-value, MDE met, verdict), then a verdict explanation, a ship/pause recommendation, and a power-analysis note. Full worked report: references/output-example.md.

Step 7 - Recommended cadence

Validate the experiment:

• At pre-defined stop date (preferred - pre-registered).
• At minimum required sample (per Step 4 power analysis).
• NOT at "first day of significance" - peeking at running experiments inflates false positives (the "early stopping" problem).

If continuous monitoring is required (e.g. a regression-detection A/B test), use a sequential testing framework (statsmodels' sequential probability ratio test) instead of repeated significance tests.

Anti-patterns

Seven analysis mistakes and their fixes: references/anti-patterns.md.

Limitations

• Causal inference assumes proper randomization. If users self-select into variants (geography, plan tier), bias is uncorrected.
• No defense against contamination. Users that switch variants mid-experiment violate randomization; flag and exclude.
• Power analysis is parametric. Real distributions deviate; use bootstrap for non-parametric power estimation.
• Network effects unmeasured. A user-level test may underestimate effects when treatment users influence control users (social features, marketplaces).
• Doesn't replace product judgment. A statistically significant win that contradicts brand strategy isn't auto-ship.

References

• ab-test-wiki (opens in new window) - A/B testing definition; "A/B tests are sensitive to variance; they require a large sample size in order to reduce standard error and produce a statistically significant result"; statistical hypothesis testing framing.
• feature-toggles (opens in new window) - experiment toggles: per-cohort routing; "highly dynamic ... requires sufficient runtime to generate statistically valid results."
• feature-flag-test-harness (in the qa-test-environment plugin) - harness that runs the experiment IN test (this skill validates the experiment IN production).
• prod-canary-validator - sibling: same statistical framework, different application (canary verdict vs experiment verdict).
• synthetic-monitor-author - sibling: production-side verification, different role.