
One of the main selling points for AI in conversion rate optimisation (CRO) is that it takes over the analyst’s job. AI tools read customer feedback and analytics data, then build test variants faster than a human dev team.
However, most of what AI takes over is volume work, the reading, sorting and building that used to fill an analyst’s week. Whether a programme makes money depends on judgement calls, such as which problem to solve and whether a winner should ship. Those are the calls AI tools handle worst, so they now make up more of the job. Our view is that judgement is the part of CRO worth paying for, and AI has made that easier to see.
What research says AI is good and bad at
The case for AI has a real basis. In a 2023 field experiment run by Harvard Business School with Boston Consulting Group, 758 consultants worked on 18 realistic tasks. Those with GPT-4 finished 12.2% more tasks, worked 25.1% faster and produced work rated about 40% higher in quality.
The same study set one task the AI couldn’t do well. Consultants had to solve a business problem using interview notes and spreadsheet data, where the right answer depended on a detail in the interviews that the spreadsheet didn’t show. On that task, consultants using AI were 19 percentage points less likely to get the right answer than those working without it. The researchers called this uneven pattern the jagged technological frontier.
That second task looks a lot like a CRO analyst’s normal week, which mixes customer research with analytics data. The risk for CRO teams is analysis that looks finished and turns out to be wrong.
What AI does well in conversion research
Sorting customer feedback at scale suits AI well. A University of Sheffield study in AI & Society compared GPT-4o with human coders on feedback data from a UK charity. It used Cohen’s kappa, a measure of agreement where 1 is perfect and 0 is no better than chance. GPT-4o scored 0.91 to 0.95 on sentiment and 0.61 to 0.65 on sorting comments into themes, but 0.01 or lower when judging whether a comment showed the charity’s work had made a difference.
The data comes from a charity, but the same split applies to CRO research. An AI tool can count how often thousands of reviews mention delivery costs in a few minutes. Whether delivery costs make people leave your checkout is a separate question, one that needs behavioural data and someone who knows the business.
Can AI predict which A/B test will win?
If AI could predict a test result before the test runs, teams could skip the tests likely to lose. Recent research on this is mostly discouraging. A 2025 Marketing Science study by researchers at the University of Washington and the University of Toronto used 17,681 headline A/B tests from the publisher Upworthy. None of the AI-only methods they tried could pick the winning headline with high accuracy, and the best did only slightly better than chance.
Researchers at Spotify looked at the statistics behind replacing real users with AI responses. In a 2026 paper, they concluded that an A/B test run on AI responses is correct only if certain assumptions hold, while a test on real users is correct by design. When they asked an AI model to predict click rates for the Upworthy headlines, its predictions shrank the measured effects towards zero, so changes looked less effective than they were.
A 2026 preprint from Squoosh, a company working on AI test prediction, tested AI judges on 330 real website A/B tests. Different models and prompts agreed closely with each other, with kappa scores of 0.74 to 0.88. Against the real results, the main AI judge scored a kappa of about 0.14, and about 0.11 on tests with significant results. The human experts in the study agreed with each other at 0.53 and did no better than chance against the real results. What stands out to me is that both the AI judges and the experts agreed closely with each other without matching the real results, which is a reason to keep real users in the test.
Results improve when a model learns from real tests. A 2026 NeurIPS paper retrained a small model on each round of Upworthy results as they came in. That loop beat the other methods clearly, and most of the gain arrived with the first round of real results.
Which CRO decisions AI can’t make for you
Once the volume work moves to AI, four decisions stay with the practitioner.
- Choosing the problem means picking the few research themes that are costing conversions.
- A hypothesis has to make a claim about customers that a result could prove wrong.
- Reading a result includes deciding what a flat or losing test rules out.
- Before a winner ships, someone checks it against pricing and engineering plans.
Each of these needs context an AI tool doesn’t have, and someone accountable for the result. So we treat AI output at these steps like a draft from a new analyst, something to question before it goes into a test plan.
Why generic AI gives you the same test ideas as your competitors
Generative AI tends to make ideas better and more alike at the same time. In a 2024 study in Science Advances, Anil Doshi and Oliver Hauser gave some of 300 writers story ideas from GPT-4. Judges rated the AI-assisted stories as more creative, yet the stories were also more alike. The University of Exeter’s summary reports that writers who used one AI idea wrote stories 10.7% more similar to each other than writers with no AI help.
The same thing can happen to test backlogs. If every team asks similar models for ideas drawn from the same best practice, their backlogs will start to look alike. What differs between companies is their own test history, especially the tests that lost, because those show what didn’t work for their customers. Before we ask an AI tool for test ideas, we give it the results of past tests, losses included, and ask it to check new ideas against them.
Does AI make experimentation faster?
Research and building variants have sped up most. Optimizely’s 2026 benchmark covers 173,000 experiments on its own platform. It reports that customers in its average AI usage band ran 70.2% more experiments than before they used its AI tools, and their average test impact rose by 20.6%. Average test impact is Optimizely’s measure of win rate multiplied by the size of the winning uplift.
Optimizely sells these AI tools and describes the comparison as observational, not causal, because teams chose whether to adopt them. The same report found that setup quality was the biggest predictor of win rate. Experiments Optimizely rated as soundly set up won 58% of the time, against 35% for poor ones. Running more tests at once splits your traffic, so each one takes longer to reach a result you can trust. Our article on prioritisation covers how to choose.
Shipping winners hasn’t sped up in the same way. A winning test still has to be rebuilt in the live site, checked against the version that won and released by someone with the authority to do it. Those steps depend on engineering time and agreement between teams, which AI does little to shorten. That’s where much of a programme’s value leaks away.
Should you optimise your site for AI shopping agents?
Some of your visitors now arrive through AI tools. Adobe Digital Insights, which tracks more than one trillion visits to US retail sites, reported that AI-referred traffic to those sites grew 62% year on year in July 2026. Adobe also found those visitors converted at a rate 60% higher than other traffic. That’s vendor data from Adobe’s own platform, covering the US only.
Most of those visitors still buy on your own site. OpenAI launched Instant Checkout in ChatGPT in September 2025 so people could buy without leaving the chat. By March 2026, Forrester reported that OpenAI was replacing it with merchant apps that send shoppers to the merchant’s own checkout. For now, AI tools mostly shape which products a shopper sees, and your site still has to convert them.
How much to spend on product data that machines can read, against the on-site experience, depends on how many of your customers shop this way. Your own analytics can show you.
How to use AI in your CRO process
Work through these steps to hand AI the volume work and keep the judgement with your team.
- Give AI the volume work: use it to summarise reviews, surveys and session notes, and to sort comments into themes you’ve defined.
- Check cause before you test: confirm in behavioural data that a theme is costing conversions, and check any AI segment finding against the original data.
- Feed in your own results: give AI tools your past test results, including losses, before you ask them for ideas.
- Write the hypothesis yourself: state the claim about customers and the result that would prove it wrong before launch.
- Keep real users in the test: use an AI prediction of a winner to decide which test to run first, then run it with real users.
- Measure what ships: make the value of winners that are live on the site your headline figure.
Where this leaves the CRO practitioner
AI tools now handle much of the research and build work, so a team can run more tests than before. That means more decisions about what to test and what to ship, made in less time, and the research above suggests AI can’t make them for you yet. If AI did all the research and build work in your programme from next month, which decisions would be left with your team, and who would be accountable for each one?
If you’d like help working that out, get in touch with our CRO team.
Frequently asked questions
What is AI conversion rate optimisation?
AI conversion rate optimisation is the use of AI tools to raise the share of website visitors who complete a goal, such as a purchase. Teams use it to summarise feedback and build test variants. Deciding which problems to solve and which changes to ship still needs a person who knows the business.
How is AI changing CRO?
AI is changing CRO by taking over high-volume work such as sorting feedback and building test variants. The practitioner’s role shifts towards judgement, including which problem to solve and which winners ship. AI still can’t reliably predict which A/B test will win. A 2025 Marketing Science study of 17,681 Upworthy headline tests found the best AI-only methods were only slightly more accurate than chance.
Will AI replace CRO specialists?
AI is unlikely to replace CRO specialists soon, because the judgement calls that decide a programme’s value are the tasks it handles worst. CRO specialists who hand volume work to AI can spend more of their time on those calls.
Can AI analyse customer research for CRO?
AI can analyse customer research for CRO well at the level of sentiment and themes. In a University of Sheffield study, GPT-4o agreed closely with human coders on sentiment but scored near zero when judging evidence of impact. Use AI to find what customers talk about, then check in behavioural data which issues cost conversions.
Can AI predict A/B test results?
AI can’t yet predict A/B test results reliably enough to replace a real test. In a 2026 preprint, Squoosh tested an AI judge on 330 real website A/B tests and found it matched the real outcomes at a kappa of about 0.14, where 0 is chance, and about 0.11 on tests with significant results. Spotify researchers concluded in 2026 that an A/B test run on AI responses is correct only by assumption, while a test on real users is correct by design.
Sources
- Dell’Acqua et al., Navigating the Jagged Technological Frontier, Harvard Business School Working Paper 24-013, September 2023: https://mitsloan.mit.edu/sites/default/files/2023-10/SSRN-id4573321.pdf
- Wen, Clough, Paton and Middleton, Leveraging large language models for thematic analysis: a case study in the charity sector, AI & Society 41(1), published online August 2025: https://eprints.whiterose.ac.uk/id/eprint/230962
- Ye, Yoganarasimhan and Zheng, LOLA: LLM-Assisted Online Learning Algorithm for Content Experiments, Marketing Science 44(5), September 2025 (17,681 Upworthy headline tests): https://ideas.repec.org/a/inm/ormksc/v44y2025i5p995-1016.html
- Persson, Schultzberg and Ankargren, Statistical Foundations of LLM-based A/B Testing: A Surrogacy Framework for Human Causal Inference, arXiv 2606.17165, June 2026: https://arxiv.org/abs/2606.17165
- Spotify Engineering, When Can LLMs Replace Humans in A/B Tests?, August 2026 (source of the effect-shrinking finding): https://engineering.atspotify.com/2026/8/when-can-llms-replace-humans-in-a-b-tests/
- Dooskin and the Squoosh Technical Staff, The Judge Knows When It Knows: Calibrated Abstention for LLM-Based A/B-Test Prediction, arXiv preprint, 2026 (not peer reviewed, company research; kappa 0.141 on all 330 tests, 0.108 on significant-only labels): https://arxiv.org/pdf/2608.07517
- Predict, Observe, Retrain: Can a Language Model Learn to Pick A/B Test Winners from Its Own Feedback Loop?, NeurIPS 2026: https://neurips.cc/virtual/2026/169211
- Optimizely, Lessons learned from 173K experiments, 2026 (vendor data): https://www.optimizely.com/field-notes/guides/173000-experiments
- Doshi and Hauser, Generative AI enhances individual creativity but reduces the collective diversity of novel content, Science Advances 10(28), July 2024: https://pmc.ncbi.nlm.nih.gov/articles/PMC11244532
- University of Exeter, AI found to boost individual creativity but results in less varied content (summary of Doshi and Hauser, source of the 10.7% similarity figure): https://news.exeter.ac.uk/?p=13749
- Adobe Digital Insights, AI Traffic Trends Report, 19th August 2026 (July 2026 retail figures, vendor data): https://business.adobe.com/assets/pdfs/resources/sdk/ai-traffic-trends-report-august-2026/ai-traffic-trends-report-2026-08-19.pdf
- Forrester, Agentic payments in B2C commerce, where we are now (Instant Checkout replaced by merchant apps, March 2026): https://forrester.com/blogs/agentic-payments-in-b2c-commerce-where-we-are-now