Skip to content
Why the 80% of A/B tests that don't win still matter
Most experimentation programmes spend a lot of time thinking about win rate, and rightly so. Far less time goes on whether those results feed data driven decision making anywhere beyond the testing team.
A strong win rate means more experiments are generating measurable improvements in customer experience and commercial performance. Better research, stronger hypotheses and more effective prioritisation should all increase the chances of identifying changes that make a difference.
But there’s another question that gets much less attention.
What happens to everything you learn from the tests that don’t win?
Industry benchmarks suggest that the majority of A/B tests don’t produce a positive result on their primary metric. Optimizely’s analysis of 127,000 experiments across 1,100 companies found that only 12% won on their primary metric, with average win rates of around 20% across all experiments. Other industry benchmarks put typical win rates at around 25 to 30%.
We’ve achieved an average 48% win rate across our experimentation programmes over the last five years.
That’s something we’re proud of. We believe it reflects the importance we place on research, prioritisation and developing strong hypotheses before an experiment is launched.
But it still leaves an important question.
What can we learn from the other 52%?
A test doesn't have to win to create value
It’s easy to think of experimentation in binary terms.
The test won, so we implement it. The test lost, so we move on.
But that misses a large part of the value experimentation can create.
A losing test can decisively disprove something the business previously believed about its customers. An inconclusive test can challenge a strongly held hypothesis. And a series of experiments can reveal patterns in customer behaviour that no individual experiment would have uncovered on its own.
That distinction matters.
A test can win, be implemented and never influence another decision. Equally, a test can lose and provide evidence that changes what a business tests, designs or prioritises next.
The opportunity isn’t to make win rate less important.
It’s to make sure we’re extracting as much value as possible from every experiment we run.
Where does all that learning go?
This is where many experimentation programmes have a problem.
Results are often recorded in the experimentation platform, presented back to the immediate team and categorised as a win, loss or inconclusive result. Then everyone moves on to the next test.
The statistical result has been recorded, but the insight may never travel beyond the people directly involved in the experiment. That’s a very different thing from organisational learning, and it’s where data driven decision making tends to break down.
For an experiment to influence data driven decision making across the business, the result needs to be translated from a statistical outcome into something useful. That means answering five questions.
What did we learn about our customers?
Which assumption did this support or challenge?
What should we do differently as a result?
Does this reinforce something we’ve seen in other experiments?
Who else in the organisation needs to know?
Without that translation, potentially valuable customer evidence can disappear into a testing platform.
Individual tests tell you something. Patterns tell you much more.
There’s another reason documenting every experiment matters.
A single test rarely gives you enough evidence to make a significant strategic decision. But imagine six experiments across different areas of a website all pointing towards the same underlying customer behaviour.
Perhaps customers consistently respond positively when delivery information is presented earlier.
Perhaps simplifying choice repeatedly improves engagement.
Or perhaps several apparently different experiments suggest that customers don’t understand a particular proposition as well as the business assumed.
Individually, those experiments may produce relatively tactical insights. Collectively, they can tell you something much more important.
That’s when experimentation starts moving beyond optimisation and becomes an evidence base for data driven decision making. But that only works if the organisation can connect the dots.
Four things strong experimentation programmes should do
There are four practical things that help turn experimentation results into reusable evidence for data driven decision making.
1. Create a shared evidence library
Experiment results shouldn’t live solely in individual testing platforms, presentations or people’s heads.
They should form part of a structured, searchable record that allows teams to understand what has already been tested, what was learned and how that evidence relates to future hypotheses.
2. Translate experimentation metrics into business outcomes
A movement in conversion rate, engagement or another experiment metric may be meaningful to the experimentation team.
It becomes much more useful to the wider organisation when it can be connected to the commercial or customer outcomes stakeholders care about. That translation helps experimentation insight travel beyond the CRO team, which is where joining test results to wider business data makes the biggest difference.
3. Document losses and inconclusive results as rigorously as wins
If only winning experiments are properly documented, a significant proportion of the evidence generated by an experimentation programme is being lost.
A failed hypothesis is still evidence.
Knowing what doesn’t work can improve the next hypothesis, prevent another team repeating the same mistake and challenge assumptions that might otherwise continue influencing decisions.
4. Look for patterns across experiments
Experiments shouldn’t always be treated as isolated events.
Regularly reviewing groups of tests allows teams to identify recurring behaviours and themes. Those patterns can be much more strategically important than the result of any single experiment.
If you’re building these habits into an in-house team, our CRO enablement work is designed to help, and our case studies show what that looks like in practice.
Win rate matters. But so does learning rate.
We shouldn’t stop caring about win rate.
Quite the opposite.
A well-run experimentation programme should continually improve its ability to identify, prioritise and validate opportunities that create measurable commercial impact.
But maximising the number of successful experiments is only one part of the opportunity. The other is making sure the evidence generated by every experiment contributes to what the organisation knows about its customers.
Our experimentation programmes have achieved an average 48% win rate over the last five years. We’re proud of that number, but we’re equally interested in what we can learn from the other 52%.
Because if every experiment improves the evidence behind the next decision, the value of experimentation extends far beyond the individual tests that win. That’s when experimentation becomes part of data driven decision making across the business, rather than simply a way to optimise a website.
If you’d like to see where your own programme stands, take our maturity model assessment or speak to a consultant.
FAQs
Why don't most A/B test results end up influencing business strategy?
Mostly because of what this piece calls the reporting gap, the space between a test concluding and someone with authority over the roadmap hearing about it and changing their thinking. Results tend to get logged in a testing tool’s dashboard for the immediate team only, and rarely get translated into a documented insight that reaches anyone beyond it.
What percentage of A/B tests win?
Optimizely’s analysis of 127,000 experiments puts it at roughly 12% winning on the primary metric, with a 20% average win rate across all experiments. VWO and CXL’s benchmarks land at 25 to 30%, and Microsoft and Google report 10 to 20% in well-optimised products.
Why are so many A/B tests inconclusive?
Roughly 40 to 45% of tests return no measurable effect in either direction. That’s not the same as wasted. An inconclusive result on a strongly held hypothesis is still useful information, but most programmes file it as a non-event rather than writing it up.
Is the barrier to data driven decision making mostly technical or cultural?
Overwhelmingly cultural. NewVantage Partners’ survey of large company data executives found only 24% describe their organisation as data driven, and around 80% say the main barrier is people, process and culture, with technology much further down the list.
What happens to losing or inconclusive tests in most testing programmes?
They typically get no write up at all. Since 80 to 90% of tests fall into the loss or inconclusive bucket, that’s the majority of a programme’s potential organisational learning being discarded by default.
What needs to change for test results to influence strategy?
The piece argues that four things need to change. A shared, searchable evidence library needs to replace results left sitting in individual tool accounts. Each test’s metric needs a clear translation into the business metric stakeholders track. Losing and inconclusive tests need write ups, not just the wins. And there needs to be a regular cadence for reviewing patterns across tests, rather than treating each one in isolation.
Sources
Optimizely, “Top 10 takeaways from running 127,000 experiments”, win rate and category impact benchmarks (2026), optimizely.com
Kohavi and Thomke, “The Surprising Power of Online Experiments”, Harvard Business Review (September to October 2017), Bing example and experimentation culture
Davenport and Bean, “Action and Inaction on Data, Analytics, and AI”, MIT Sloan Management Review (2023), citing the NewVantage Partners survey, data culture and organisational barrier statistics
Gupta, Kohavi et al., “Online Experimentation, Benefits, Operational and Methodological Challenges, and Scaling Guide”, Harvard Data Science Review (2022), 10 to 20% positive effect benchmark and cultural and organisational scaling challenge
Blend Commerce, “A/B Testing Benchmarks for eCommerce”, consolidated industry win rate data (2026)
roast.page, “A/B Testing Statistics 2026”, hypothesis documentation and top performer benchmark data
