Experiment Design for Data Science
Experiment design is the foundation of rigorous data science. Whether testing a new feature, measuring the impact of a marketing campaign, or validating a machine learning model, the quality of the conclusions depends on the quality of the experimental design. Poorly designed experiments produce misleading results, wasted resources, and incorrect business decisions. This guide covers the core principles and practical techniques for designing effective experiments.
The Scientific Method in Data Science
At its core, experiment design applies the scientific method to data-driven decision-making. You start with a hypothesis — a specific, testable statement about the effect of a change. The null hypothesis (H₀) states there is no effect, while the alternative hypothesis (H₁) states there is an effect. The experiment collects data to determine whether the evidence supports rejecting the null hypothesis in favor of the alternative.
Randomization
Randomization is the most important principle of experiment design. By randomly assigning subjects to treatment and control groups, you ensure that observed differences are due to the treatment rather than confounding variables. Randomization eliminates selection bias and provides a valid basis for statistical inference. Without randomization, any difference between groups could be attributed to pre-existing differences rather than the treatment effect.
Control Groups
A control group provides the baseline against which the treatment effect is measured. The control group should be identical to the treatment group in every way except for the intervention being tested. In A/B testing, the control group experiences the original version while the treatment group experiences the variant. Placebo controls are essential in medical and behavioral experiments where the expectation of treatment can produce effects.
Sample Size and Statistical Power
Power Analysis
Statistical power is the probability that a test will detect an effect when one truly exists. Low power means the experiment is unlikely to detect real effects, leading to false negatives (Type II errors). Power depends on three factors: the effect size, the sample size, and the significance level (α). A power analysis before the experiment determines the minimum sample size needed to detect a meaningful effect with acceptable power, typically 80% or higher.
Calculating Sample Size
The required sample size depends on the expected effect size and the desired statistical power. Smaller effects require larger samples. For a two-sample t-test comparing means, the sample size per group is approximately n = 16σ²/δ² for 80% power at α = 0.05, where σ is the standard deviation and δ is the minimum detectable effect. Online calculators and statistical software simplify this calculation, but understanding the underlying trade-offs is essential.
Common Experimental Designs
A/B Testing
A/B testing (also called split testing) is the most common experimental design in industry. Users are randomly assigned to two groups: A (control) and B (treatment). The metric of interest is measured and compared between groups. A/B tests are widely used for website optimization, product feature launches, and marketing campaigns. The simplicity of A/B testing makes it easy to implement, but it requires careful attention to sample size, duration, and multiple testing.
Factorial Designs
Factorial designs test multiple factors simultaneously. A 2×2 factorial design tests two factors, each at two levels, requiring four groups. This design reveals not only the main effects of each factor but also interaction effects — whether the effect of one factor depends on the level of another factor. Factorial designs are more efficient than running separate experiments for each factor.
Crossover Designs
In a crossover design, each subject receives both the treatment and the control at different times. This design controls for between-subject variability, effectively using each subject as their own control. Crossover designs require fewer subjects than parallel designs but are only appropriate when the treatment effect is reversible and there is no carryover effect between periods.
Common Pitfalls
Multiple Testing Problem
When testing multiple hypotheses simultaneously, the probability of at least one false positive increases dramatically. If you test 20 independent hypotheses at α = 0.05, you expect one false positive by chance alone. Corrections like the Bonferroni correction or false discovery rate (FDR) control must be applied. Pre-registering the primary endpoint and analysis plan reduces the temptation to cherry-pick significant results.
Peeking and Early Stopping
Checking experimental results repeatedly and stopping when results become significant inflates the false positive rate. This practice, called peeking, violates the assumptions of frequentist hypothesis testing. Sequential testing methods and Bayesian approaches allow for interim analysis without inflating error rates. If peeking is unavoidable, use a sequential test design with proper stopping rules.
Simpson’s Paradox
Simpson’s paradox occurs when a trend appears in several groups but disappears or reverses when the groups are combined. This paradox arises from confounding variables that differ between groups. Proper randomization and stratification before the experiment prevent this issue. In observational studies, techniques like matching and regression adjustment are necessary.
Conclusion
Rigorous experiment design is the difference between data science that produces reliable insights and data science that produces misleading conclusions. By mastering the principles of randomization, control, power analysis, and appropriate design selection, data scientists can ensure that their experiments generate trustworthy results that drive better decisions.
A/B Testing Fundamentals
A/B testing (randomized controlled trial) is the gold standard for causal inference in data science. Users are randomly assigned to control (A) or treatment (B) groups, and the difference in outcomes measures the treatment effect.
Sample Size Calculation
Running an underpowered test wastes time and risks false negatives. Calculate minimum sample size before starting:
import statsmodels.stats.api as sms
# Detect a 5% conversion increase from 20% baseline
effect_size = sms.proportion_effectsize(0.20, 0.25)
sample_size = sms.NormalIndPower().solve_power(
effect_size, power=0.80, alpha=0.05)
# Returns ~1,500 per variantFactors affecting sample size: baseline conversion rate, minimum detectable effect, desired statistical power (typically 80%), and significance level (typically 5%).
Common Pitfalls
- Peeking — Checking results repeatedly and stopping when significant inflates false positive rates. Pre-register sample size and analysis plan.
- Multiple comparisons — Testing many metrics increases chance of finding a false positive. Apply Bonferroni correction or use false discovery rate control.
- Novelty effect — Users behave differently with new features initially. Run tests long enough (minimum 1-2 weeks) to capture steady-state behavior.
- Network effects — In social products, treatment users affect control users through interactions. Use cluster randomization when network effects are expected.
Bayesian A/B Testing
Bayesian approaches provide intuitive probability statements: “There is a 95% chance that variant B is better than variant A.” Compared to frequentist methods, Bayesian tests allow sequential monitoring without inflating error rates and naturally incorporate prior knowledge. Tools like PyMC3 and Google’s CausalImpact implement Bayesian testing frameworks.
Post-Test Analysis
Analyze results across segments — the average treatment effect may hide heterogeneous effects. A feature that helps new users may hurt power users. Compute confidence intervals for the treatment effect, not just p-values. Consider practical significance, not just statistical significance — a 0.1% improvement on billion-dollar revenue may be worth shipping, while a 1% improvement on a minor metric may not.
Designing Online Experiments
Metric Selection
Choosing the right metric is as important as the experimental design itself. The primary metric should directly reflect the business goal — conversion rate for a checkout flow, retention for a feature launch, or engagement for a content change. Secondary metrics capture unintended effects. A checkout redesign that increases conversion but doubles customer support calls is not an unqualified success.
Good primary metrics are sensitive to the expected change, measurable with reasonable sample sizes, and resistant to gaming. Instrumentation must be validated before the experiment starts — if the metric is not tracked correctly, the entire experiment produces meaningless results. Run a A/A test (control vs control) before the actual experiment to verify that the metric is stable and the tracking is correct.
Variance Reduction
Reducing variance increases statistical power without increasing sample size. Stratification divides subjects into homogeneous blocks (new vs returning users, mobile vs desktop) and randomizes within each block. CUPED (Controlled-experiment Using Pre-Experiment Data) uses pre-experiment data as a covariate to explain post-experiment variance. Both techniques can reduce required sample sizes by 30-50% without changing the experiment design.
Network Effects and Interference
In social products and marketplaces, treatment users affect control users. If a new recommendation algorithm shows different products to treatment users, the control users’ experience also changes because inventory shifts. This SUTVA (Stable Unit Treatment Value Assumption) violation invalidates standard A/B testing. Solutions include cluster randomization (randomize at the geographic or social network level), switchback experiments (randomize at the time level), and difference-in-differences analysis.
Frequently Asked Questions
What is the minimum sample size for an A/B test? The minimum sample size depends on the baseline conversion rate, minimum detectable effect, desired statistical power (typically 80%), and significance level (typically 5%). For a 20% baseline conversion rate and a 5% relative improvement, you need approximately 1,500 users per variant. Use power analysis tools to calculate sample size before starting any experiment.
What is the difference between A/A and A/B tests? An A/A test compares two identical control groups against each other. It validates that the experiment infrastructure is working correctly — there should be no statistically significant difference between two control groups. If an A/A test shows a significant difference, the experiment platform has instrumentation bugs, sample ratio mismatch, or other issues that must be fixed before running A/B tests.
How long should an experiment run? Run experiments for at least one full business cycle — typically one to two weeks — to capture day-of-week effects and novelty effects. Longer experiments increase statistical power but delay decisions. Use sequential testing methods to check results periodically without inflating false positive rates.
What is Simpson’s paradox in experiments? Simpson’s paradox occurs when a trend appears in several subgroups but disappears or reverses when the groups are combined. This happens when the confounding variable differs between groups. Proper randomization prevents this in controlled experiments, but in observational analysis, stratification and causal inference techniques are necessary.
Can I run multiple experiments simultaneously? Yes, but ensure they do not interact. If one experiment changes the checkout flow and another changes pricing, the results may interfere. Use overlapping experimentation platforms that hash users to consistent experiment buckets. Monitor for interaction effects and run orthogonal experiments on independent systems.
For a comprehensive overview, read our article on Bayesian Statistics Guide.
For a comprehensive overview, read our article on Big Data Tools Guide.