Experiments

Statistically Rigorous A/B Testing for AI

Test prompts, models, and parameters with confidence. Winnow supports mSPRT sequential testing, Bayesian analysis, multi-armed bandits, and six workflow templates designed for every stage of the AI lifecycle.

Start Free Read the Docs

run_experiment.pypython

import winnow

# Create an experiment with two arms
experiment = winnow.create_experiment(
    name="gpt4o-vs-claude-summarization",
    hypothesis="Claude produces more concise summaries",
    arms=[
        {"name": "control", "model": "gpt-4o", "prompt": "v2.1"},
        {"name": "variant", "model": "claude-sonnet", "prompt": "v2.1"},
    ],
    statistical_method="msprt",
    traffic_split=[50, 50],
)

# Log an observation from production
winnow.log_observation(
    experiment_id=experiment.id,
    arm="variant",
    user_id="user_8291",
    metrics={
        "latency_ms": 320,
        "quality_score": 0.92,
        "cost_usd": 0.003,
    },
)

Built for AI Experimentation

Every statistical tool you need to make high-confidence decisions about your AI pipeline.

Statistical Methods

Choose from mSPRT sequential testing, Bayesian posterior analysis, Difference-in-Differences (DiD), and CUSUM change detection depending on your experiment design.

Multi-Armed Bandits

Automatically shift traffic to the best-performing arm with Thompson Sampling, Upper Confidence Bound (UCB), or epsilon-greedy exploration strategies.

Workflow Templates

Start fast with 6 lifecycle templates: Shadow Mode, Canary Rollout, Blue/Green, Champion/Challenger, Interleaving, and Feature Gate experiments.

Power Analysis & SRM Checks

Estimate sample size before launch, and automatically detect Sample Ratio Mismatch (SRM) issues that could invalidate your results.

Not your typical A/B testing tool

Traditional feature-flag tools treat every metric the same. Winnow understands that AI outputs are stochastic, high-dimensional, and expensive to evaluate. Our statistical engine accounts for LLM variance, supports composite quality metrics, and lets you monitor a live experiment continuously with anytime-valid confidence intervals — so you can peek without over-reading an interim trend, and decide on the assignment-unit result at a planned look.

Anytime-valid intervals — peek safely without over-reading interim trends

Composite metric support (cost + quality + latency)

Built-in LLM variance modeling

Winner declared on the assignment-unit result, not an interim peek

Full audit trail for every decision

Ship better AI with every experiment

Create your first experiment in under five minutes. Free for up to 10 active experiments.

Start Free