Experiment Methodology
Choose how an experiment is analyzed with Bayesian, fixed-horizon frequentist, CUPED-adjusted, or sequential methods.
Ours Privacy supports three analysis methods for experiments:
- Bayesian is the default and reports each variant's probability of being best, expected loss, and a 95% credible interval.
- Frequentist is a fixed-horizon test that reports the absolute treatment effect, a 95% confidence interval, and an adjusted p-value after the planned sample is complete.
- Sequential uses an always-valid confidence sequence for conversion metrics, so the official result is safe to read anytime while data accrues.
Frequentist experiments can optionally use CUPED to reduce variance with each visitor's pre-experiment history.
Supported Combinations
| Analysis | Conversion metrics | Value metrics | CUPED | Read mid-flight |
|---|---|---|---|---|
| Bayesian | Yes | Yes | No | Bayesian results update as data arrives |
| Frequentist | Yes | Yes | Optional | Descriptive data only until the planned horizon |
| Sequential | Yes | No | No | Yes, with an always-valid confidence sequence |
Bayesian CUPED, sequential value metrics, and sequential CUPED are not supported. A configured experiment never silently switches to another method.
Visitors and Impressions
The statistical unit is a unique exposed visitor. A visitor enters the analysis after their first recorded experiment impression, and repeat impressions do not increase the statistical sample size.
The report keeps both counts visible:
- Visitors are unique people included in statistical inference.
- Impressions are recorded
$experiment_impressionevents and remain useful for delivery diagnostics.
If a visitor appears in more than one variant, the report identifies the crossover as a data-quality issue.
Choose the Method Before Starting
Set the analysis method while the experiment is a draft. Starting a Frequentist or Sequential experiment locks:
- The analysis method, test direction, and primary goal
- The outcome window
- CUPED and sample-planning settings, when applicable
- Variant definitions, weights, traffic allocation, and targeting
The API recalculates POWER-plan visitor targets from the saved design at start. Manual visitor targets are persisted as configured.
To use a different design after data collection begins, clone the experiment and start a new one.
Primary and Exploratory Goals
The primary goal selected before start is the experiment's confirmatory metric. It can produce the official evidence state.
Secondary goals and events selected later in the results report are exploratory. They can show descriptive estimates, but they do not produce the experiment's official winner conclusion.
Outcome Windows
An outcome window gives each visitor time to complete the goal after first exposure.
- Bayesian analysis uses a seven-day outcome window.
- Frequentist and Sequential analysis can use
1,7,14, or28days, selected before start.
Only visitors whose outcome window has closed enter inference. The report distinguishes Collecting data from Finalizing outcomes, so reaching a visitor target is not mistaken for a completed cohort.
Bayesian Analysis
Bayesian analysis supports conversion rates and average value per visitor. The report shows:
- Probability to be best
- Expected loss
- A 95% credible interval
- Observed results and relative lift
Inference uses mature unique visitors. Value results do not claim a certain winner when an arm has too few observations or no estimable variance.
Fixed-Horizon Frequentist Analysis
Frequentist mode requires a sample plan before start. For conversion metrics, plan from a baseline, minimum detectable effect, and 80% power, or enter a target number of visitors per variant directly. Value metrics use manual visitor targets.
The official effect is the treatment conversion rate minus the control conversion rate, shown in percentage points for conversion metrics and value units per visitor for value metrics. Relative lift is contextual; the confidence interval and conclusion use the absolute effect.
Fixed-horizon evidence is available only after every planned cohort has matured. In A/B/n experiments, each treatment is compared with control and Bonferroni correction is applied across comparisons.
One-Sided and Two-Sided Tests
Frequentist and Sequential experiments default to a two-sided test: "Did the treatment change the outcome?" It can identify either improvement or harm.
An advanced one-sided test asks only: "Did the treatment improve the outcome?" It can reach positive evidence with less data, but it cannot turn a negative observation into formal harm evidence. Choose the direction before start.
CUPED Variance Reduction
CUPED is available for fixed-horizon Frequentist experiments. It uses each visitor's history for the same primary metric during the fixed 28-day period before the experiment starts.
- Conversion metrics use the visitor's pre-period count of the goal event.
- Value metrics use the visitor's pre-period total value for that event.
- Visitors with no pre-period activity receive a covariate value of zero.
The report shows covariate coverage, correlation, the adjustment coefficient, raw and adjusted variance, and achieved variance reduction. If the covariate is unavailable or has zero variance, the report uses the unadjusted fallback and names the reason.
Sequential Analysis
Sequential mode supports conversion metrics with a time-uniform, always-valid confidence sequence. You can open the report repeatedly without the fixed-horizon peeking penalty.
The report shows the absolute conversion-rate difference and its current always-valid bounds. Evidence of improvement appears when the lower bound clears zero. A two-sided experiment can also show evidence of harm when the upper bound falls below zero.
Sequential intervals are usually wider than ordinary fixed-horizon intervals because they remain valid across every look. "Safe to read anytime" means the false-positive guarantee is preserved under continuous monitoring; it does not mean every early result is conclusive.
Result States and Data Quality
The report separates readiness from evidence:
- Collecting data: the confirmatory cohort has not reached its requirement.
- Finalizing outcomes: enough visitors were exposed, but some outcome windows remain open.
- Ready: the configured analysis can produce an official conclusion.
- No conclusion: the evidence threshold is not met.
- Evidence of improvement or Evidence of harm: the two-sided threshold is met.
- Inconclusive - stopped early: a fixed-horizon experiment ended before its cohort matured.
- Analysis unavailable: an integrity issue prevents a trustworthy conclusion.
While a cohort is still filling, the report shows counts without an evidence statement:
Data-quality diagnostics include sample-ratio mismatch, missing visitor identifiers, and crossover visitors. Material integrity failures suppress official evidence.
API Access
Use GET /rest/v1/experiments/{id}/analysis for the method-aware result. The response contains common visitor, impression, readiness, evidence, and data-quality fields plus the applicable bayesian, frequentist, or sequential block.
The older results endpoint remains the legacy Bayesian contract and rejects experiments configured with another method.
Related Reading
How is this guide?