Overcome Diminishing Returns of A/B Testing

by IT Manager

Starting a new A/B testing program on your website is an exciting time. Using your web analytics, you identify which pages of your website are underperforming. From there, you put on your proverbial thinking cap and develop ideas to improve page performance. Next, you formulate test ideas and use A/B testing to see what fixes perform better.

When you first start A/B testing, you’ll be overrun with opportunities for improvement and can quickly develop creative solutions. Easily identifiable issues are the easiest to solve, and when you fix glaring weaknesses, you may feel like an optimization all-star.

But what happens after you consume all the low-hanging fruit?


When A/B Testing Stops Working

After you conquer obvious problems and get some great wins, companies tend to develop a huge hunger for testing. However, as you A/B test more and your website improves, it becomes more and more challenging to improve upon your improvements. Your win rate decreases and you end up with too many zero-sum A/B tests.

Credit: Timo Elliott

Your web analytics was great early on because it  helped you figure out where you have problems. However, identifying problem areas isn’t the problem – it’s pinpointing the fixes. That is how an A/B testing program should evolve. It’s only as good as what you feed it.


Move Past the “What” and Get to the “Why”

Big data is not a Rosetta stone – it’s just a tool. Website analytics are archival and tell you what happened, but not why. Analytics aren’t answers but spawn questions.

While evolving your A/B testing program, supplemental information from your users can reveal why some pages underperform. You know “what” from the analytics, but “why” is largely based on your assumptions unless you tap another data source.

If you strategize from misinterpreted information, you’ll waste time and money.

Studying your users breaks through flawed assumptions.


Your Audience Has All the Answers

We are all too close to our products and services. We know too much and have too many biases. This creates blinders that makes it difficult for us to understand why issues exist and would could be done to improve them

Only your audience can tell you that, and User Testing is a very effective, low-cost way to get them to tell you.

User testing shows how your audience interacts with your website, why they do/don’t click, and why some areas of the site fail to engage. You get startlingly relevant insights from studies conducted with an audience that mirrors your ideal customer. These studies move you beyond assumptions that undermine your A/B testing program.

Because user testing observes and records the interaction between your website and its targeted audience, it provides the “why” that you desperately need so you can optimize, drive sales, and deepen customer satisfaction.


User Testing Studies Uncover Hidden Opportunities!

One user test can help you identify many things to fix and opportunities to purse. However one person is rarely a good sample size. Therefore, it’s a good idea to run 5 – 10 user test about one specific journey. This is called a User Testing Study.

By having 5-10 different people in your target audience go through a focused journey, you will get varied points of view that will allow you to develop trends. The strength of these trends will make it possible for you to prioritize what to fix now and test later.

The results of a well-conducted user testing study are almost always startling. You will find issues you never would have thought were issues. Your audience will outright give you some great A/B testing ideas, and through observation, you will develop new A/B test ideas that would have never occurred to you otherwise.

Simply stated, user testing studies supercharge your optimization program.


You Will Need Some Help

The average single user test will run between 20-30 minutes. That means that single study, with 10 participants, will produce between 200 – 300 minutes of video. This is a lot of video review, analyze, and synthesize.

It’s also important to document your study and build a plan around your learnings. These study reports should contain a summary of findings, trends, critical improvements and A/B test ideas that will help you and your organization plan and take action. It’s the equivalent of putting together a focused business performance report from extensive data and analysis.

Without this study report, you will just have a database full of videos that you’ll dread sifting through.


SharpTheory’s UX Learning Solution Can Help

At SharpTheory, we specialize in conducting user testing studies for companies. We offer a service called our UX Learning Solution, which includes a monthly user testing study that includes the following:

  • Analysis and synthesis of up to 300 minutes of user testing videos
  • Summary of the most interesting findings
  • Breakdown of strong and moderate trends identified
  • List of critical fixes that should be made immediately (no A/B testing required)
  • A/B test ideas that should be added to your testing roadmap
  • 3-5 minute highlight real of the most interesting parts of the study that can be easily shared

Every study has a money-back guarantee: if you don’t think a study is valuable, we’ll run another one for free or just give you a refund.

If you’d like to learn more about how it works, what it includes, and what it costs, here’s a short overview.

If you’re ready to start on this journey and supercharge your optimization program, please contact us to schedule an introductory chat.

About Scott

No Comments

Leave a Comment

Overcome Diminishing Returns of A/B Testing