Maurice, spill the beans for those who missed out on Emerce Fashion. What is A/B testing?

A/B testing is both a scientific research and validation method that allows you to give your new ideas or optimizations the ultimate reality check through an experiment on your website. You have the original page, known as the control (A), and you create a spiced-up variant (B) with tweaks to that same page. Half of your visitors get the A experience, and the other half get the B treatment. The version that outperforms the other is the winner—think better sales, fewer drop-offs on a page, or more newsletter sign-ups. 

Why is A/B testing so crucial?

A/B testing gives you a crystal-clear view of your visitor’s most genuine behaviour. The kicker? The visitor has no clue they’re part of an experiment. They just see a page and do their thing. And let’s not forget, that experiments are happening everywhere—from insurance companies’ phone scripts to the layout of your local grocery store. So, the next time you’re grumbling about where the tomato soup has disappeared, chances are they’re testing out the ideal shelf placement. In the marketing world, experimenting is the name of the game. 

Do other research methods offer less accurate insights?

You should never put all your eggs in one research-method basket. Usually, A/B testing is the cherry on top of various types of research. However, in other studies—especially interviews and user tests—the visitor is acutely aware they’re under the microscope, which can skew their behaviour. 

You shared 7 inspiring examples in your talk; let’s dive into a few of them.

ClickValue changed a client’s button text from ‘Place Order’ to ‘Buy with Card.’ The result? More conversions and increased credit card sales. How do you explain this outcome?

“People often have what I like to call ‘button anxiety.’ What happens when I click that button? Am I committed? Where will it take me? That’s what you see in this test—people were hesitating. What’s going to happen? We also gained this insight through recordings where you can track visitor clicks. Plus, visitors are more suggestible than you might think. Pointing them toward the credit card option nudges them to quickly follow your lead. Small tweaks can yield big wins with your visitors.” 

You stressed the importance of calculating your MDE based on this test. What’s MDE, and why is it so crucial for an experiment?

“MDE stands for Minimum Detectable Effect, and you can calculate this effect based on your traffic, number of conversions, and corresponding conversion rate. The MDE number indicates how significant a change in a variant must be to statistically detect it in an experiment. Don’t underestimate it—with an MDE above 5%, you’ve got yourself a real challenge.” 

“The MDE is vital because it helps shape your testing strategy. Can you focus on small elements like a small change in copy or those infamous buttons, or do you need to go big? With MDEs above 10%, you’ll have to make drastic changes if you even want to test them. That’s where many newbie CRO specialists trip up; they start with the well-known content tests, not realizing that they often aren’t feasible.” 

When I hear you talk about statistics, it makes me wonder, can everyone conduct A/B tests?

“Sure, everyone can eventually (laughs), but A/B testing is essentially the application of a scientific research method. Those who’ve had statistics and/or methodology in school will pick it up quickly. But you’re also dealing with human behaviour that you’re trying to interpret through data. You need specialists like web analysts and researchers to do A/B testing at a high level. I’ve got to be honest about that.” 

Another test you discussed: ClickValue assisted a company in improving its checkout.

A redesign was already in place. The team was all set to implement it, but after testing the new checkout design against the old one, the new design ‘lost’ with an 18% drop in conversions. Why did you think it was so important to highlight this experiment? 

“This is where execution, research, and experiments intersect with the organization you’re working in.” What you unfortunately often see is that “CRO” only gets pulled into redesign projects after something has gone south. The organization had conducted customer interviews, and everyone was over the moon. But these are customers, and for a strong brand at that, who are already thrilled. So, they wanted to get this new checkout live without further research. By pushing for the importance of validation, we eventually won them over and were able to run an experiment. And thank goodness we did—the checkout turned out to be leaky. And not just a little.” 

So, no more research, just A/B testing?

“No, quite the opposite—do more research. But if you can test it, then, test it. It’s risky to base your implementation on just one form of research. Make sure you validate your findings in multiple ways—exit surveys, interviews, data analysis, and A/B testing. The richer your research set, the more you learn. And that was overlooked in this case.” 

Were you able to save the day?

“Absolutely! What many people don’t realize is that we at ClickValue also guide many redesign projects, but we do it data and insight driven. So many mistakes are made in the realm of redesign. Especially when creating something new, you need to continuously seek your customers’ feedback. There’s a good chance you’re missing the mark. By continually researching and testing, we’re able to deliver successful redesigns. And that was the case here, eventually.” 

One last example I found quite interesting: a client noticed that website sales of their old collection were consistently failing.

No surprise—the sales page was tucked away for fear it would overshadow the new collection. The experiment revealed something different: showcasing the new collection alongside sales items boosted sales of the latter, importantly, without cannibalizing the new collection. So, the organization comes into play here as well? 

“Yes, exactly. We sat down with those responsible for the retail strategy. Because, in physical stores, the sales were doing just fine. Their takeaway: mix it up! In stores, products were often sold well when combined. It was different online, but again, the old and new collections didn’t clash. A super useful test to debunk myths in this case.” 

You advocated for as much cross-pollination between departments as possible.

“Yes. Make all your experiment learnings easily accessible to everyone. Ensure that people learn from each other; it’s invigorating. And if possible, occasionally visit organizations outside your industry. Keep your eyes open; you can learn something from everyone.” 

What’s the next step for companies looking to dive into A/B testing?

“I see two groups: those who haven’t started experimenting yet, and those who have. The non-experimenters need to get started ASAP. You wouldn’t believe the money that slips away through poorly designed landing pages, checkouts, and other pages. These companies are often very active with online campaigns. My tip for them: save 10% on your ad spend and invest it in your first experiments. And of course, we at ClickValue are more than happy to help.” 

And for those who have started testing?

“You can test everything. Sustainability is a key concern for many in the fashion industry. We conduct lots of return experiments with our clients. By helping them reduce returns, we contribute to their profitability and a more sustainable world. We’ll soon be hosting a sustainability talk to dive deeper into this.”