AI in CRO research: what to automate and what not to

More hypotheses don’t mean better experiments.

AI can already audit user journeys, cluster findings and propose tests in minutes. But if most experiments don’t move the metric, the value isn’t in generating more ideas. It’s in deciding better which ones deserve traffic, budget and development hours.

Monday morning. The marketing team proudly presents the output of its new AI agent: sixty test ideas, each with a priority score and cited evidence. A month ago they had twelve.

The problem is that your site only has enough traffic to run a handful of properly powered tests a month. And each one consumes development hours that product, operations and the security team are also asking for.

Sixty ideas aren’t an advantage if you don’t know which five to run first.

This article doesn’t debate whether AI is useful for conversion rate optimization (CRO) research. It is. It looks at where to delegate without losing control of the decision, with evidence you can bring to the conversation with marketing, finance and the board.

Figure	What it means
~1 in 3	Share of experiments at Microsoft that improved the metric they targeted; at Booking.com, Google Ads, Netflix and Bing the figure drops to 10–15%
21.2%	Overlap between the problems found by human experts and by GPT-4o in a heuristic evaluation, although the model found 27 additional problems
30%	Generative AI projects Gartner projected would be abandoned after proof of concept by the end of 2025, due to poor data quality, inadequate risk controls, escalating costs or unclear business value
~6%	Companies McKinsey classifies as AI “high performers”, meaning they report significant business value

The bottleneck was never generating ideas

Start with the most uncomfortable figure in digital experimentation. Ronny Kohavi, who led experimentation at Microsoft, Amazon and Airbnb, has documented that the failure rate was around 85% at Bing and reached 92% at Airbnb. These are organizations with dedicated research teams.

The takeaway for an IT leader is straightforward. Experimentation capacity is a finite resource: traffic, run time, development hours and QA attention. Every poorly chosen test takes a slot a better hypothesis could have used.

An AI that triples the number of ideas without improving their quality doesn’t speed up the program. It clogs it.

The cost of a bad hypothesis isn’t the test that fails. It’s the good test that never ran.

Map the process before choosing the model

In a recent CXL workshop, the organization’s president, Hesh Fekry, and experimentation consultant Ruben de Boer built an AI workflow on top of ResearchXL, a research framework that combines analytics, surveys, user testing and expert review. The first thing they did wasn’t open an agent. It was draw a swimlane diagram with every stage, its owners, tools and timing.

Once the process is visible, the boundary draws itself. Some tasks are operational, repetitive and verifiable; others require judgment. Only the first kind become instructions for the model.

This isn’t a methodological detail. McKinsey found that AI high performers are nearly three times more likely to have fundamentally redesigned their workflows. The tool comes after the redesign, not before.

What to delegate: three tasks with a low cost of error

The workshop automated three chained stages. All three share one trait: if the AI gets it wrong, the error is caught before it costs money.

1. Heuristic review. Auditing user journeys against criteria of relevance, clarity, friction and motivation. The agent reviewed five complete journeys in about 30 minutes; an equivalent human workshop takes hours just to prepare. It’s the method with the least weight in the evidence hierarchy: few people do it, without data volume and with individual bias. That’s what makes it the safest starting point.
2. Thematic analysis. Clustering findings from surveys, analytics, heatmaps, recordings and reviews into themes. It turns hundreds of loose observations into patterns traceable to their sources. When several independent methods point to the same theme, the evidence gets stronger.
3. Idea classification. Translating themes into actions: implement directly, instrument, test, form a hypothesis or research further. Useful as a first draft of the backlog, never as the final backlog.

Thematic analysis has a benefit few people mention: it changes the conversation with the business. Instead of “let’s add testimonials”, the team discusses which customer problem it’s trying to solve. That’s a language the board understands.

The evidence on AI as an evaluator is contradictory, and that’s useful

Two studies published in 2025 reached seemingly opposite conclusions.

In the first, a heuristic evaluation method using multimodal models identified 73% to 77% of the usability problems in two apps, outperforming five experienced human evaluators (57% and 63%). However, it struggled to recognize some interface components and design conventions, and to detect problems that span multiple screens.

In the second, with a different design, GPT-4o matched only a small fraction of the experts’ findings, although it surfaced new problems.

The professional conclusion isn’t to pick one study. It’s to recognize that the result depends on the method, the prompt and the context, and that AI sees different things than a human does. That makes it a valuable additional evaluator, not a substitute. Flow problems, the ones that cut across several screens, are precisely the ones that hurt conversion most in checkout or sign-up processes.

A 42% isn’t a measurement

The workshop agent assigned scores like 42% or 53% to each journey. The decimals suggest measurement. They aren’t: they’re subjective judgments dressed up as rigor.

There’s an irony here. The PXL prioritization framework, created by CXL itself, deliberately uses a binary scale to avoid central tendency bias, and gives lower scores to ideas that come from opinions. A model that returns fine-grained percentages reintroduces exactly the subjectivity those frameworks try to remove.

The second limit runs deeper. AI can check whether a page uses urgency or social proof, because those elements are in the frameworks. It can’t know whether your customer cares about them. A finding that’s correct according to the framework can be irrelevant to conversion.

AI contributes	The team decides
Coverage and consistency in applying a framework	Which problems matter for your business
An initial score to sort by	Whether that order holds up against real evidence
Patterns across many sources	Whether the pattern reflects the customer or the prompt

A score with decimals isn’t more accurate. It’s just harder to challenge in a meeting.

What isn’t delegated: the voice of the customer and the decision

The workshop explicitly ruled out any technique that replaces the opinion of real customers, such as simulated interviews. The evidence backs that line.

Nielsen Norman Group repeated three studies previously run with real participants, this time using synthetic users. The values those simulated users reported were too shallow to act on, and the artificial participants claimed to have finished every course, something no real group does.

There’s also a quieter failure, which De Boer called “research as decoration”: the model lands on a best practice first and then looks for data to support it. The right sequence is the reverse: evidence, finding, hypothesis, test.

The symptom is easy to spot. Generic ideas that would fit any site, with data citations that look bolted on at the end.

Four quality gates

The temptation is to build a single agent, hand it all the information and accept whatever it returns. The professional approach is a pipeline with gates:

1. After the heuristic review. The team adds its own research and the context the model lacks, such as regulatory constraints or recent price changes.
2. After the thematic analysis. Someone checks that each theme makes business sense and has real sources, not just textual coherence.
3. Before generating ideas. Add what the AI missed, especially problems that span screens or channels.
4. When prioritizing. A person sets the final order with an explicit framework and traceable evidence.

This matches what McKinsey sees in organizations that capture value: they’re more likely to have defined processes for deciding when model outputs need human validation. The gate isn’t bureaucracy. It’s the difference between correcting an assumption in minutes and discovering it after a quarter of poorly targeted tests.

What this demands of IT

CRO research usually lives in marketing, but the decisions that determine whether the workflow can be trusted run through your area.

• Personal data. Session recordings, surveys and analytics exports contain customer information. Define what can be sent to an external model, with what anonymization and under what contract, in line with the data protection rules of each country where you operate.
• Instrumentation quality. If event tagging is broken, thematic analysis will cluster noise very eloquently. It’s no coincidence that poor data quality tops Gartner’s list of reasons generative AI projects get abandoned.
• Traceability. Every theme and every idea must be traceable to its source. What can’t be audited shouldn’t reach the board.
• Modular architecture. Separate stages with reviewable outputs are easier to maintain, version and fix than a monolithic agent.
• Delivery capacity. An AI-inflated backlog ends up as demand on your development team. Agreeing on a cap of tests per cycle protects that capacity.

Reasonable objections

“If AI found more problems than the experts in one study, why keep analysts?” Because finding problems was never the expensive part. With experiment success rates that rarely exceed a third, the value lies in choosing what to test. AI frees up operational hours; the analyst reinvests them in prioritization and validation, where the return is won or lost.

“Synthetic users are cheaper. Why not use them at least for filtering?” Use them to prepare interviews or get familiar with a segment’s vocabulary. Not to validate concepts. The available evidence shows overly positive and shallow responses, exactly the kind of bias that leads to launching tests that move nothing.

“This is marketing’s responsibility.” Until someone connects customer data to a model without review, or sixty ideas land in the development sprint. Data governance, instrumentation quality and delivery capacity are shared responsibilities, and IT has the tools to exercise them.

Speed doesn’t replace judgment

AI already reviews more journeys, organizes more research and proposes more tests than any team could produce by hand. It’s worth taking advantage of.

But in experimentation, the constraint was never the number of ideas. It was the quality of the decision about which ones deserve traffic, budget and development hours. Organizations that understand that difference will use AI to decide better. The rest will just fail faster.

Automate the operation. Keep the decision.

Your first step this week: sit down with your CRO lead and draw the swimlane diagram of your research process. Mark which tasks are operational and which require judgment. Pick the heuristic review as a pilot, define the four gates before writing the first prompt, agree on what data can go out to the model and set how many tests per cycle your development team can absorb.

Sources

• Reslan, T. (2026). AI for CRO research: what to automate (and what not to). CXL.
• Zhong, R., McDonald, D. W. and Hsieh, G. (2025). Synthetic Heuristic Evaluation: A Comparison between AI- and Human-Powered Usability Evaluation. arXiv.
• Guerino, G. et al. (2025). Can GPT-4o Evaluate Usability Like Human Experts?. arXiv.
• Kohavi, R. and Thomke, S. (2017). The Surprising Power of Online Experiments. Harvard Business Review. Success-rate figures cited in Product Growth.
• Nielsen Norman Group. Synthetic Users: If, When, and How to Use AI-Generated “Research”. Summary on PM Toolkit.
• Gartner (2024). Gartner Predicts 30% of Generative AI Projects Will Be Abandoned After Proof of Concept By End of 2025.
• McKinsey & Company (2025). The State of AI in 2025. Coverage in IT Brief.
• CXL. PXL: A Better Way to Prioritize Your A/B Tests.
• Speero. ResearchXL.