SharpTheorySharpTheory

An introductory guide to the most popular types of A/B Tests

  • 0

by Scott Olivares

How many times have you heard someone use that dreaded blue button vs. yellow button example when explaining the value of A/B testing? Yes, it’s an easy-to-understand example, but it really undersells the power of A/B testing and isn’t very creative.

To add more flavored examplesto our idea bank, this post coversa few standard digital A/B testing experiments and their pros and cons from my perspective. I also welcome my fellow experimenters to comment and contribute with their own examples!

1. Complete redesign tests

I’m going to say it, and I’m not ashamed to say it: I REALLY don’t like site-wide redesigns. I think they often turn into creative expression exercises justified by light research, and they tend to fail. And even when extensive research is conducted, and really talented user experience professionals create them, they still tend to fail. If you’re curious about why they tend to fail or how to avoid failure, check out my other post: “What’s Missing from Your Website Redesign Plan?”

Now that I’ve gotten that out of my system, I’ll admit that sometimes, but not very often, they are necessary. In a redesign test, everything is fair game and can be changed. You can change, move, remove, and add whatever your heart desires in the name improving your digital experience.

Pros: Redesign tests are good for when you have something that you know is a bad experience and needs to be completely replaced. At that point, incremental optimization is not going to cut it. Sometimes what you need is to start over and not to optimize something that is bad. In other words, as my friend Mike King once told me, “you can shine a piece of shit every single day, but at the end of a month you’ll simply have a really shiny piece of shit.”

Cons: When you change too many things at once, it’s practically impossible to learn what helped or hurt your digital experience. You may have used a new hero image that increased conversion by 10%, but changed an important description that decreased conversion by 15%. Since you changed both at once, all you would know is that you had a decrease in conversion of 5%, which could cause you to disregard an awesome hero image. It’s a shotgun approach that I often compare to high-stakes gambling. Redesigns can be great when executed well, but they come with a lot of risk, are difficult to plan around, and you always give up a lot of learning. Use them sparingly.

Tip: If you’re going redesign your entire experience, make sure you maximize this opportunity by doing a lot of due diligence in understanding what your visitors want. Don’t just assume that you can think like a visitor. Have an experienced analyst dive into your web analytics to understand how visitors use your current site and develop some actionable insights. Conduct some usability testing. Go through common use cases. And most importantly, optimize while you slowly ramp your new experience – don’t ramp it to too many people until you know it converts at least as good as your old website.

2. Description tests

Words matter, A LOT. Changing what you say in a page title, photo caption, call to action, or product description can have dramatic effects with practically no creative assets needed.

Pros: Description tests are a great way to incrementally improve a page that you don’t think is that bad. Additionally, since most of these experiments involve testing some kind of HTML text, you’re unlikely to need a designer or heavy web development. They are also pretty easy to implement because changing text usually isn’t too technically complicated.

Cons: Usually, people don’t take the time to read on the Internet, and a common occurrence with description tests is that you may see no conversion difference. In order to make an impact, you need to make sure that you are testing descriptions that people are actually reading.

Tip: Look through your web analytics to identify any keywords that may drive significant traffic to your page or look through your own internal search from that page to identify any popular keywords. You are more likely to get visitors’ attention if you use keywords or phrases that you know people are interested in.

3. Promotion tests

In promotion-centric tests, you can experiment with different prices, promotions, and the way you position promotions to determine what visitors respond to and how much they respond.

Pros: Promotional tests can be very useful in determining an optimal price for your products and how to best position promotions. They give you the freedom to try promotions on a sample population and ensure that you offer the most effective one to the entire population.

Cons: Promotional tests where you offer discounts can be trickier to interpret. In most cases, larger discounts create higher conversion rates, but if your discounts are larger than your conversion increases, you may start eating into your profits. It takes a little more detailed monitoring and analysis to ensure that you are making a good business decision. In addition, you can really upset your potential customers if you’re not careful. Be wary of offering different pricing to different people. Some big companies have received a lot of bad PR for these types of tactics.

Tip: Try the same discounts to everyone, but positioning them differently in your test variations. You may find that your customers respond more to a percentage discount (e.g. 25% off) compared to a flat dollar discount (e.g. get $50 off $200). It’s also possible that visitors are insensitive to $9.99 off but are more likely to convert if you offer $10 off.

4. Image tests

Sometimes just using a different image on your page can make a big difference. If you’re a vacation-planning site, lifestyle images of people having fun may work better than a nice shot of the beach. On the other hand, if you’re selling furniture, visitors may want to see the piece of furniture by itself.

Pros: Imagery is something that people actually pay attention to, and sometimes the right image can really move the needle in right direction.

Cons: It’s easy to get lazy with these experiments and let your subjective judgment dictate the images you test. As with any test hypothesis, maximize the likelihood of a successful test by using research-based insight to determine your imagery.

Tip: Many times, less is more. If you have a page with 3 smaller images, try testing one bigger image that makes an impact. When there are too many things competing for your attention, nothing stands out. Also, there are plenty of simple tools out there such as EyeQuant or Attention Wizard that can help evaluate how your images may impact users’ ability to focus on your key value propositions and CTAs.

5. Design Tests

This is where you try to optimize by testing color, fonts types and sizes, shapes, or layout. These aren’t complete redesigns; rather, it’s changing certain aspects of your design.

Pros: Design-oriented tests not only help improve your conversion, they also allow you to test different design prototypes to ensure they don’t hurt conversion before you release them to the world. Remember that avoiding the implementation of something that hurts conversion is even more important than finding something that improves conversion.

Cons: Design-oriented tests usually require help from a designer, and depending on the complexity, a web developer. This often makes them more resource heavy, political, and slower to produce. Additionally, finding evidence to support that changing things such as button color or font styles will increase conversion may be difficult, so you’re sometimes left with creating less data-driven test variations.

6. Targeting tests

Many will say that targeting is not the same as testing. I agree, but you can test your targeting approach to ensure that you are being as effective as possible. For example, your targeting may be based on a referring keyword, but conversion rate may be higher if your targeting is based on visitors being Mac users.

Pros: Targeting tests can really help you capture hard-to-reach customers by fine-tuning your personalization tactics. These types of tests are the next level, once you have harvested all the low hanging fruit with your other testing.

Cons: Testing different targeting tactics is not as straightforward as testing visual items and requires a pretty deep understanding of your visitors. You will also need to invest in a tool that allows you to target because some of the more basic A/B testing offerings may not have this capability.

Tip: For the most part don’t listen to anyone that says they tried targeting and it didn’t work. Except for rare situations, targeting will help increase conversion if you find a recipe that resonates with users. Like I mentioned in the cons section, good targeting requires a deep understanding of your visitors, but when you figure it out, you can really provide a great experience and improve conversion.

[BONUS]
7. The learning or monitoring campaign

I LOVE learning campaigns. This under-utilized tactic is where you create various visitor segments, track them separately, and show them all the same content. Or perhaps you monitor any patterns based on time of day, day of week, or when your company’s TV commercials air.

Learning campaigns are very helpful because they make it easier to learn about visitor behavior within a very specific page or process that you want to optimize. They can often be helpful in creating a content strategy for your segments and sparking test ideas that you would have never thought about.

Have Any Others to Add?
These are some of the more popular types of tests, but it’s certainly not an exhaustive list. I welcome any of my fellow testing practitioners to add to it or provide their own perspective.
Scott Olivares
About Scott Olivares

No Comments

Leave a Comment

An introductory guide to the most popular types of A/B Tests