Zoom Zoom A/B Testing Strategy
The Zoom Zoom A/B testing strategy will help you get better testing results by helping you focus on the things that matter most. Too many split testers test things that are so granular that they don’t even matter. This results in tests that have low lifts or no lift at all.
By following the Zoom-Zoom testing strategy you will be able to identify what matters to your visitors and get bigger gains faster because you will know where the value is.
In this video, you will learn several things to improve your testing efforts including:
- How to avoid testing things that are too granular
- What it means to zoom out with your business question
- What a zoom zoom sandwich is
- The 4 steps to following the Zoom-Zoom Testing Strategy
Testing Theory is where professional testers turn to do better A/B testing and get more conversions.
Creating a Culture of Optimization
Good A/B testing isn’t about just running split tests. Conversion Rate Optimization is about creating a data-driven culture inside the company. This video will cover 6 ways to build a data-driven culture of optimization. As you do each of these things you will create a stronger optimization practice.
Here are the six steps and the things that we will cover.
1. Get Test Results
2. Evangelize Results Broadly
3. Ideation From The Company
4. Create Infrastructure
5. Integrate Testing Widely
6. Cross-Functional Team
Testing Theory is where professional testers turn to do better A/B testing and get more conversions.
WebsiteBox Existence Testing
This A/B testing example shows a case study from WebsiteBox.com. The very first two tests we did with them helped us learn how to prioritize our testing to make future tests even more valuable.
These tests show how to do a back to back existence knockout punch that would be great for the first two tests of any site.
This video will help you learn a ton about foundational tests to jumpstart your split testing.
In this video, you will learn several things to improve your testing efforts including:
- The efficiency of existence testing current pages and content
- Learning over lift approach
- Learning regardless of loss in lift
- How the relative impact of existence testing helps prioritize efforts
- How to do a reverse existence test
- Combination effect of elements
- Redesign based on the discovered value of elements relative to each other
Testing Theory is where professional testers turn to do better A/B testing and get more conversions.
3 Reasons to A/B Test the “Impossible”
As you are doing conversion rate optimization, you will get push back from stakeholders, technical people, and even team members. They will tell you the same thing, that an A/B test can’t or shouldn’t be done. “We can’t build that test,” or “So and so doesn’t like that split test,” or “That A/B test is too complex.”
This video will address 3 reasons why you should A/B test the impossible.
In this video, you will learn several things to improve your testing efforts including:
- The three benefits of testing things that people say you shouldn’t
- How to test something that is impossible
- Fake it before you make it strategy
- Cautions in testing the impossible
Testing Theory is where professional testers turn to do better A/B testing and get more conversions.
What is the Optimal Number of Variations Per A/B Test?
Did you know there is an optimal number of variations to run per A/B test? Some simple math can help you identify what that sweet spot is so you get more gains with your split testing. More gains per test amount to better visitor experiences and more conversions for your company.
The Math of More Variations
In this video, you will learn several things to improve your testing efforts including:
- How more variations increases your success rate
- The math behind choosing the optimal number of variations
- The diminishing returns of adding more variations
- The constraints to consider before building multiple variations
Testing Theory is where professional testers turn to do better A/B testing and get more conversions.
Their Tests Are Killing Your A/B Testing Program
A/B Testers want to see the successes and failures of other people’s tests so they don’t make the same mistakes and so they can capitalize on the successful tests. Unfortunately, there are 4 dangers of using other peoples test results.
In this video, I will walk through the four things to avoid as you look at other’s test results. I will also teach you the three ways you should be using other people’s test results.
The Four Dangers Include:
1. Lack of Context
2. Theirs are Theirs
3. Overgeneralization
4. “We Already Know” mentality
The Three Ways You Should Use Test Results
1. Drive Ideation
2. Principles and Patterns
3. No Traffic or Testing Constraints
Testing Theory is where professional testers turn to do better A/B testing and get more conversions.
Transcript provided for your reading convenience:
As testers, we all like good test results. We like to see our successes, and we like to see other people’s successes. Today I want to talk to you about some cautions you should use when you’re looking at other people’s test results; four things you need to avoid, and three ways to actually use people’s test results.
Many of you may remember that old website WhichTestOne. That was one of the first in the industry to showcase lots of test results. Most of the major companies who have tools for AB testing also have their own case studies they’re putting out. Case studies from them, case studies from other companies, and as testers, again, we love to see these, but we have to be cautious when we use them.
We’ve all been in the meeting where someone says, “Hey, XYZ company is doing this, and we should do that, too.” As testers, we know that’s not a good approach because we don’t know if that company’s tested anything. But when you bring some data to the table and someone says, “Hey, look, I ran this test and I got this lift,” for some reason we feel compelled that we need to take that into account and run our own tests on that same thing, or just take their advice for it and use those learnings and apply it to our site.
There are several problems, though, by taking other people’s test results and applying them to our site or even just running the test from that. I want to highlight what those four things are.
The first reason you should be cautious is because most test results show a lack of context. Context is king, and without the proper context, you don’t truly know if that lift was a good lift. For example, there are several things that make up good context. Let’s talk about those. Sometimes with test results, you don’t see the sample size. They may have a 500% lift, but if it was only on five visitors, that’s not a good enough sample size. Often when people share case studies they don’t always share all the details, for good reason. Some of that data may be sensitive. Even myself, when I share a case study with you guys, I’m not going to share all the data, because some of that data is sensitive, some of it’s proprietary, some of it we don’t want the world to know all the details. But I do want to share with you the learnings from those tests in case you can apply the principles of the test. So the first area where there’s lack of context is with the sample size. If you have a 500% lift but it’s only on five visitors, well, that’s not quite good enough.
Another thing that you won’t always see with any test result is the natural variance. They might have a lift, but what if the variance is just as high as that lift? Again, see my video on doing a variant study, but the natural variance is part of your test results, and you won’t always know what that is just by looking at those results. Another thing to be cautious of is the right metrics. When people are trying to promote themselves or their testing, often they’ll choose the metrics that look the best. If they’re using metrics to inflate the value of their test, you may not be able to know for sure if those are the right metrics to make an impact on the business. The other thing that we rarely see with test results is a comparison of other tests. Just because you see a test result, you don’t necessarily know that’s valuable because it’s a comparison of tests that truly show value.
Finally, most test results are never talking about the bottom line impact of the test on the business, or the lifetime value of the customer. All these things are the context that we should be considering if we’re truly data-driven and disciplined that we usually don’t get with other case studies and test results, for good reason. Sometimes it’s sensitive like I said, and sometimes it’s just too much to share. But for the most part, there’s a lack of context with tests that are being shared.
The second reason to be cautious of other people’s test results is because of a very important principle, and that is that these test results are their test results, and their test results apply to their visitors and their sites. Even if it’s a competitor of you in the same industry, those visitors coming to your site are there for a reason, and from what I’ve seen, whenever we’ve tried to replicate something that’s working somewhere else, it never works as good as just trying to find what works on your site for your visitors. So their results are theirs, and we need to be cautious when we just take those at face value, knowing that it’s a different audience, a different site, perhaps even a different industry.
The third reason that you should be cautious when you’re looking at other people’s test results is that in most times, test results and the learnings will be overgeneralized. For example, I saw a test result recently where someone had tested using logos of companies on a testimonial, rather than just saying the name or having an image of the person. They put a logo of the company. The takeaway from that test result was, so, if you want to have better results, you need to use company logos instead of face shots. That overgeneralization is something we need to be very cautious of that, yes, that may have worked on that test, but to overgeneralize and say that this works for everyone, man, we really have got to be cautious with that. We need to find out if that works for us and our business and our visitors. So that’s the third reason, be cautious of overgeneralization.
The fourth and most dangerous reason why we need to be cautious of other people’s test results is that often when we test results or show them to other people, they get this mentality like, “Great, now we know the answer. Now we don’t even need to test it.” That mentality of “we don’t need to test it now that we know”, is very dangerous.
So those are the four reasons to be cautious of other people’s test results. First, there’s the lack of context. Second, their results are their results. Third, there’s overgeneralization. And fourth, people have a tendency to want to not test when they’ve seen other people’s data.
Okay, so let’s talk about how you should use other people’s test results. Again, I’m a fan of learning from other people’s tests. I share test results myself. So how should we use them? There are three ways that I want to show you that you should use test results.
The first and most important is to drive ideation. Testing is about good ideas. When we see someone have success somewhere, naturally we want to take that idea and assimilate it and try it for ourselves. The important thing is that those test results are informing your ideas and they become tests for you as well. Don’t take them at face value and just implement it. The other thing that I would suggest as you’re using other people’s test ideas to drive ideation is to assimilate the idea, but then add more ideas to it. Don’t just say, “Wow, that worked more than a test on our site,” but say, “Hey, this worked, let’s test it, and then let’s create some more variations to see how we can push those boundaries of something that may have worked for someone else.
The second way to use other people’s test results is to see how what they did might apply to a broader principle. This is different from overgeneralization. Overgeneralization says that you should do this because it works. Everyone does it. A principle’s a pattern, and so if you’ve run 20 tests that have a certain pattern that is being exhibited with your site, and you see someone else’s test results that follow that same pattern, you might say, “Hey, that pattern matches what we’re already seeing. Because of that, we might want to test on our side as well.” Testing is looking for patterns with your site and your visitors, and you can apply those principles when you see that matching pattern.
The third and final way in which you might want to use other people’s test results is the least valuable and the one I would caution you against, but sometimes there’s a use case for it. When you’re a new site or a new business or a small business and you don’t have a lot of traffic and you’re trying to make the best site you can but you don’t actually have enough visitors to test with, then in that case looking at other people’s test results and saying, “Hey, this worked for them, I’m going to try with me because I know we’re not going to be testing,” that is a good use case. Again, first and foremost we should always be testing when and where possible, but there are sometimes where it doesn’t make a lot of sense to test if you don’t have the traffic, or if there are other constraints that don’t allow you to test for some reason. Then, yeah, you should apply as much knowledge as possible to the best pages possible, and if you have low traffic that’s a great time to say, “Hey, this worked for them, I’ll try it on my site as well.”
So should we use other people’s test results? The answer is yes. But there’s a right way to do it and a wrong way to do it. I hope this video’s been helpful in helping you see the ways that you should be doing it. I’ll also be posting my own successes and my own case studies with these videos. Now you know how to apply the principles in the right way and use case studies in the right way.
And if you found this video helpful, please hit the like button, subscribe to my channel, and you can also visit me at TestingTheory.com where I have courses that go in depth in some of these testing strategies and will get you better training so you can get better testing and more conversions.
Now, it’s your turn. I want to hear from you. When did you see an amazing test result? Share it in the comments below. I also want to make a quick shout out to Debra at GuessTheTest.com. If you’re looking for more test results that you like to see on a regular basis, GuessTheTest.com might be a perfect opportunity for you to sign up and see what people are submitting their test results and inform how that can help you and your business. If you’re interested in going to Guess The Test and seeing what they have to offer, I’ll put a link down below in the description.
Intelligent Risk Taking
Many marketers and executives wonder if changing the site with A/B Testing is risky. They wonder about visitors seeing a lower performing experience during a split test. They question if we are moving people’s cheese with tests.
All of these concerns are related to risk. I will show you the strategies and concepts to use to help people overcome their misconceptions about risk.
At the end of the video, we will review a Norwegian Cruise Lines case study that was worth millions in revenue to them and increased revenue per visitor by 13%.
Better testing by understanding positive risk.
In this video, you will learn several things to improve your testing efforts including:
- How risk is both positive and negative
- How testing maximizes positive risk and minimizes negative risk
- How testing has actual risks and perceived risks
- 5 of the ways that split testing is intelligent risk taking
- An example of how taking positive risks can drive a lot of revenue and conversions for your business
Testing Theory is where professional testers turn to do better A/B testing and get more conversions.