The Cost of "No Clear Winner"
Many A/B tests on ad campaigns and conversion rates end with a frustrating "no clear winner." This outcome isn't necessarily a sign of experimental design flaws discovered mid-run; more often, it indicates that the experiment was unwinnable from the start. The underlying effect size was too small for the available data to reliably detect, and no amount of additional runtime would have changed that. The good news is that you can identify these unwinnable experiments in minutes, before spending a single dollar, using a straightforward statistical check.
This method involves calculating the minimum detectable effect (MDE) your current data can support, given your desired statistical power and significance levels, and then comparing it to the expected effect size. If the MDE is larger than any realistic or meaningful improvement, the experiment is likely unwinnable.
Step 1: Compute the Smallest Lift Your Data Can Detect
For a standard two-arm test aiming to detect a difference in conversion rates, the smallest lift your data can reliably see is determined by statistical power and significance. This is often referred to as the Minimum Detectable Effect (MDE). A common setup uses 95% confidence (alpha = 0.05, two-sided) and 80% power (beta = 0.20).
The formula to calculate the MDE for conversion rates (CR) involves the Z-scores for alpha and beta, and the baseline conversion rate. The formula looks like this:
from math import sqrt
Z_ALPHA = 1.96 # two-sided 95% confidence
Z_BETA = 0.84 # 80%% power
def min_detectable_lift(baseline_cr, users_per_variant):
numerator = Z_ALPHA + Z_BETA
denominator = sqrt(users_per_variant)
term1 = (baseline_cr * (1 - baseline_cr))
term2 = ((baseline_cr * (1 - baseline_cr)) / users_per_variant)
se_diff = sqrt(term1 + term2)
mde_absolute = (numerator * se_diff) / users_per_variant
mde_relative = mde_absolute / baseline_cr
return mde_relative
The key inputs here are the baseline_cr (your current conversion rate) and users_per_variant (the number of users you anticipate assigning to each arm of the test). This formula directly calculates the minimum *relative* lift (e.g., a 5% increase over the baseline) that your experiment setup can reliably detect. A higher number of users per variant will decrease the MDE, meaning you can detect smaller effects.
Step 2: Resample Your Existing Data
The crucial insight is that you can estimate this MDE *before* running a new experiment, by resampling from your existing data. If you have historical data from similar ad campaigns or website versions, you can simulate how a smaller sample size would have performed.
The process involves taking your existing conversion data and repeatedly drawing random samples of a specific size (e.g., the number of users you expect per variant in your new experiment). For each sample, you'd run a statistical test (like a two-proportion z-test) to see if a significant difference could be detected. By doing this many times, you build a distribution of results. If, across these resampled tests, you rarely find a statistically significant difference, it strongly suggests that your current data volume is insufficient to detect anything but large effects.
More practically, you can directly use the MDE formula from Step 1. If you have a historical dataset, you can calculate the average conversion rate from a subset of that data (representing your expected sample size per variant). Then, plug this baseline CR and the subset size into the MDE formula. If the resulting MDE is higher than any improvement you realistically expect or care about, the experiment is likely unwinnable with that sample size.
Step 3: Compare MDE to Expected Lift
Once you have your calculated MDE, the final step is to compare it to the lift you are hoping to achieve. Consider your business goals and the typical performance of your ad creatives and landing pages. Is a 1% lift meaningful? Is a 5% lift realistic?
If the MDE calculated from your intended sample size is, for example, 10%, but the most optimistic realistic improvement you could hope for is 3%, then the experiment is almost certainly unwinnable. You would need to significantly increase your sample size (and thus your budget and runtime) to have a chance of detecting a 3% lift. Conversely, if your MDE is 2% and you're aiming for a 5% lift, the experiment is statistically viable.
When to Stop an Experiment Early
This pre-experiment check is invaluable. However, situations arise where an experiment is already running. If an experiment has been running for a substantial period, and the observed effect size is consistently smaller than your MDE, it's time to consider stopping it. This is especially true if the confidence interval around the observed effect does not overlap with your target lift, or if the confidence interval is very wide relative to the observed effect.
The key is to distinguish between an experiment that needs more time to gather data and one that is fundamentally underpowered. If your pre-experiment analysis shows an MDE of 10%, and after running for a week you see a 2% difference with 90% confidence, you haven't learned much. More runtime won't magically make the true effect size larger; it will only help you converge on the true (likely small) effect with more certainty. If that small effect isn't meaningful, the experiment is still unwinnable in practice.
Broader Implications for Experimentation
This approach shifts the focus from simply running experiments to running *winnable* experiments. It forces a more rigorous upfront evaluation of statistical power and sample size requirements. For teams constantly iterating on ad creatives, landing pages, or user flows, this method acts as a crucial gatekeeper, preventing wasted resources on tests where the signal-to-noise ratio is inherently too low to yield actionable insights.
It’s not about avoiding experiments, but about ensuring that the experiments you run have a genuine chance of providing a clear, actionable outcome. This requires a deeper understanding of the relationship between sample size, effect size, and statistical significance. By embracing this proactive statistical check, teams can allocate their experimentation budget more effectively, focusing on changes that have the potential for significant impact and are detectable with their available resources.