April 11

A/B Testing Intro: Why, What, Where, & How to A/B Test


Good A/B Testing can make you more money and get you more conversion for your business. In this video we will cover:
– Why Companies Should Be Doing A/B Testing
– What is A/B Testing?
– Where You Can Run A/B Tests
– What You Can A/B Test
– 10 Steps To Get Started Testing
– Examples of Tests and Case Studies


Hi, I am Rhett and I’ve been doing A/B Testing and multivariate testing for over a decade. I worked at Adobe where I led a large optimization team of testing Consultants. I’ve also started testing agencies where we’ve helped other clients do testing.

I’ve also been in the trenches with a very large corporation helping the company itself learn to do testing. I’ve seen hundreds of companies run thousands of tests.

In this video, we’re going to review why every company should be running A/B tests, what is A/B Testing, where you can run your A/B tests, what you can test, and how you can actually get started doing it. I’ll also show a couple of quick case studies including one at the end of the video that helped Under Armour get a 14 percent lift that actually led to about 3 million dollars in annual revenue for them.

So let’s dive in and get started and I’ll show you why a be testing is so important what it is what you can test how you should do it and where you should do it.

To begin, I want to start out with a case study that comes from Microsoft where specifically for the Bing website one of the ideas they had for that was a simple change that would require a lot of time and effort. But because they were so many ideas in their testing program, this idea got deprioritize by a product manager. So they never actually ran the test this test and the idea languished in their queue for about six months until one day an engineer saw the test and that the test idea would be very easy to accomplish.

So the engineer coded up the test in just a few hours. They launched the test and within just a few hours they started seeing these triggers that there were abnormally high revenues and that something must have been wrong.

However, nothing was wrong. It was an a/b test and this one test variation increased their conversions by 12% which on an annual basis led to more than a hundred million dollars in revenue for Bing. This one test alone was Bing’s see Most valuable revenue-generating idea they’d ever tested. This example shows why a/b testing and multivariate testing are so important. Bing increased their revenue by 12% with a simple A/B test and that’s one of the main benefits of a/b testing. You can actually impact your bottom line and increase your revenue and your conversions by running A/B tests. Another benefit that this example illustrates is the ability to do rapid iteration.

You can Implement a new experience and whether it wins or loses you can do it quickly. So you win quickly and get gains to your business or you lose quickly and you learn what doesn’t work for your visitors. A/B Testing is also powerful because you’re using your actual visitors to give you the result. You’re not looking at just random data sets. You’re not just going out and asking people what they think. Your actual visitors who are coming to your digital properties are telling you what works and what doesn’t work.

You’re letting your data and your visitor’s guide your decisions your investment decisions your production decisions and the things that you put out to the world. These are just a few reasons why a/b testing is so powerful and why every company should be doing it.


So now you know why it’s so important. Let’s talk about what A/B Testing is. You may have heard this called by different names you may have heard of split testing or conversion rate optimization or multivariate testing or landing page optimization. You may have heard it called digital optimization or online experimentation or even growth hacking. These are all just different ways of saying you’re running tests. You’re learning about your visitors with actual data on your actual site.

Let me give you an example of what A/B Testing looks like in real time. You can see from this visual that there are four variations we are testing and they’re allowing visitors to get each of the variations. Visitors are coming and going and cycling through experiences. Some of them are good and experienced. Some of them are bad and overall you get a result that tells you the value and impact of that test you can see that the image on the right has the highest impact once the test is finished because that’s the one that visitors likes overall. It’s the one that impacted their visitor experience the most.

One of the nice things about doing A/B Testing is that your visitors don’t know they’re in a test. They are randomly assigned and they’re in your real life real time experience. They’re not in some dev or staging environment. They’re not in some focus group. They’re actually going through your digital experiences experiencing that you have and they don’t know that they’re being tested. This is really important because it gives an unbiased view of the experience and by using data directly from your visitors, you’re able to eliminate biases and guesses and opinions that go into building digital experiences.

So now that you know what testing is, let’s talk a little bit about where you can actually do those tests. You can run your A/B tests and multivariate tests on your homepage, your landing pages, your site content, or product pages, your conversion flows, and signups your checkouts.

You can also do your tests on your paid advertising. You might be running paid ads to get more people to your site. That’s a great opportunity to do A/B Testing with your paid ads. Do you have the right headlines? Do you have the right titles? Do you have the right images and thumbnails?

You can also test in your mobile apps. There’s an infinite number of things you can test on your site and in your mobile apps.

One of the things and places you can test is your marketing campaigns. You put out marketing content and you might want to test and see how well does that marketing content do relative to your other campaigns and other content.

And finally, another good place to test is in your emails. Basically, every time you send out an email and every visitor touchpoint of an email inside any digital experience that your visitors are where they’re engaging with you is a great opportunity to do A/B Testing.

So you can see there are all these visitor touch points where you can test but what can you test in those visitor touch points?

Well, there’s a lot you can test. Pages and flow pricing, headlines, videos and testimonials, social proof. You can test how much content you put above the fold. What kind of elements they see in each experience.

But you can go beyond just testing things on the page or where you put stuff. You can test your business model. For example, you might try testing if we did free shipping and we offered that through an a/b test.

By putting that in an a/b test you would see hey how much more revenue we make by offering free shipping and what was the tradeoff by comparing that to the control.

You can also test back-end algorithm changes. For example, Amazon might test when do we recommend these different types of products or when should we target a certain person or genre with this information or this product? You might also test if you’re adding new products or new services to the market. How do these do if we introduce them as part of a test by introducing them as part of a test you see the value and impact of that product or service and that allows you to evaluate the success in the short term and long term rho past that product or service so you can see there’s a lot of things you can test across a lot of places but let’s talk about how do you actually do that? I’m going to give you a quick overview of how do you get started? What are the basic steps you would need if you want to start doing A/B Testing step. Number one, you need a testing tool and there’s lots of tools out there see some of my other videos to see some reviews on those tools. There are some of them are free. Some of them are paid and depending on the size of your business and how much money you have and how big a team you have and see some of my other videos for more information on teams and growing a testing culture, but the tool you get may depend on where you’re at. The company but to do testing you need a testing tool. So that’s the first step. You need to get a testing tool step. Number two, you need to define success in order to do a good test. You have to know what success metrics are trying to influence. If you want to increase your Revenue per visitor because you’re selling products. That’s the main success.

If you want to increase your conversion rates and get more visitors signing up for a newsletter. Well, that’s your main success. But you have to choose one success you’re trying to influence so that you have a clear answer to each test. So that’s the second step to find your success metric the third step in doing A/B test is to decide what you want to test and I usually recommend beginning with a high-level question. You don’t want to start with something too. Granular. You want to start with something higher level and then once you find something works you kind of zoom in to get a result from there. Okay. That’s the third step decide what you want to test. What idea you want to try first usually start with something simple. Don’t go crazy big but just choose what you want to test the fourth step is to now create the variations based on the question or the thing you want to learn about.

Ideally you’re creating your variations.

Ethically first so you’re thinking here’s my hell of a question. And here are the three or four variations the map. They would answer that question.

You want to be very clear about the variations and ask what will I learn if this duration wins?

What will I learn with if this variation wins you need to know before you even start the test your code the test.

What will you learn? And what will you get if something wins or loses that way even if there’s a small lift or a big lift? You’re still learning about your site. So that’s the first step Define your variations and the fifth step is to now code those variations. So you might need to go to your creative people and have them do a design. You may need to get your front end developer involved so we can change the site experiences.

Now this the fifth step is to actually create the variation so that you can start the test step six is actively the test. So now that you’ve had a coated you have the designer work on it. You’re ready to go live. You want to make sure you have the measurements measurements in place.

Remember you want to see how does this test influence the main success measure and you want to make sure the measurements is tracking that so when your test result is very clear and easy to answer so that step 6.

The test make sure your measurements are in place step 7 is to analyze the results. Once your test is complete you’re going to go in and look at the results and see how each variation did relative to the others. Now. There’s a lot that goes into results analysis.

So you might want to check out some other videos on that. But for now just know that you’re analyzing the results to see which one had the biggest impact step 8 is an important part of the process the some people lump into the previous step and analyzing results, but I want to call it as a separate step because it’s Debbie. What you’re going to do is you’re going to document what you learn and what the recommended actions were. This is an important separate step. If you don’t document your learnings and evangelize those learning so that people know about it. You’re testing efforts become they become wasted people forget. It’s we have short-term memories.

So step 8 is critical you have to document what you learn and let the company and your everyone know about the results of the test. Step number nine is tied to step number eight. Once you’ve learned something. This usually leads you to more questions. Well this happened and so now we wonder about X y&z ideation is Step number nine. After you have some learnings and some recommendations from your previous test you now take those and you learn you create new ideas from those previous learnings.

So step number nine is ideation create ideas based on what you learned from the test and that leads us to step number ten Step Entertainment simple. You just repeat the process with your new ideas and you can go back to the beginning create a good business question. Make sure you have the right things in place and start a new testing Series.

So those are the quick and dirty 10 steps on how to get testing.

You want to get a testing tool you want to find success you want to begin with a high-level question and then create strategic variations to answer that question you then create your test with your by by having designers make things and developers code things like in the right measurements in place. When you activate the test, then you analyze the results you learn something from the test and make recommendations.

And then you get ideas for follow-up tests based on the previous results. And then you repeat the process and do it again to finish off. I want to show you a real life case study of a test that I hope perform when I Looking at Adobe as a consultant.

One of my clients was Under Armour.com and we help them run a test that led to a 12 percent lift, which created a massive game 3 million dollar annualized gain for them. The nice thing about this test is it works? So well you this was several years ago, but that is still the test results and learning are still implemented on their site today. So let me show you what that looks like. So this was the control as you can see, we’re just the regular home page. They have some products. They were showing that if you sub components and in this was the bottom of their page down here add a few items down here and the variation and I’ll show you these side-by-sides and see the difference to the variation. We add a simple recommendation Zone on their homepage.

The business question was how does adding recommendations to the homepage impact visitors revenue and their business their main success metric was Revenue per visitor.

So that’s all we did was a simple test. We didn’t do a lot of variations here because we want to test it the the model itself.

So these are the two variation side by side. You can see that the control and the homepage and the variation exactly the same. The only difference is the recommendations own insert in between the last couple components of the page.

And again, we’ve already talked about this there was a 14 percent lift in Revenue which led to about 3 million dollars in annualized income or revenue for Under Armour.

This was a huge gain and an experience that they still do today. In fact, I’ll show you that right now. Let me just go to Under Armour.com.

And if their own bed, you can see that somebody has changed a lot. Hopefully they’ve done lots of testing since then but as you come to the bottom of the page, you can see this a similar recommended product zone now.

I haven’t shot in her armor in a while and that’s why it is recommended. They must have like a most popular product algorithm going now, but this test was so successful. It’s something they still do on their side today from this example, you can see there’s a huge potential for doing A/B testing the right way. It can impact your bottom line. It can make you more money. You can bring you new clients customers.

You can help you improve your business business and your business models and your strategies.

Everyone should be doing A/B Testing now, I want to hear from you in the comments below if you can let me know why you want to do A/B testing or what if you’re already doing A/B Testing. What is your favorite part about doing a/b testing?

Also, if you found this video helpful, please hit the like button just like the video so other people can find as well. If you’re interested in more videos on a be testing an advanced strategies like a multivariate testing other testing techniques.

Go ahead subscribe. To the channel, so you’ll be the first to know when these new videos come out. I post new videos every Thursday. So go and hit the Subscribe button now also you can go to testing three.com or you can subscribe to my newsletter where you can learn more about a/b testing and other Advanced strategies about multivariate testing personalization and other topics related to digital optimization.

Thanks for joining me today testing theory is where marketers come to do better A/B testing and get more conversions.

You may also like

2600 Year Old AB Test

2600 Year Old AB Test

Prospect Theory, Loss Aversion, and A/B Testing

Prospect Theory, Loss Aversion, and A/B Testing
{"email":"Email address invalid","url":"Website address invalid","required":"Required field missing"}

Ready take your optimization to the next level?

The optimization master class by Rhett will teach you everything you need to know to master the experience optimization. Take your A/B testing to new levels as Rhett teaches you the foundational, intermediate, and advanced principles of conversion rate optimization.