June 13

Their Tests Are Killing Your A/B Testing Program


A/B Testers want to see the successes and failures of other people’s tests so they don’t make the same mistakes and so they can capitalize on the successful tests. Unfortunately, there are 4 dangers of using other peoples test results.

In this video, I will walk through the four things to avoid as you look at other’s test results. I will also teach you the three ways you should be using other people’s test results.

The Four Dangers Include:
1. Lack of Context
2. Theirs are Theirs
3. Overgeneralization
4. “We Already Know” mentality

The Three Ways You Should Use Test Results
1. Drive Ideation
2. Principles and Patterns
3. No Traffic or Testing Constraints

Testing Theory is where professional testers turn to do better A/B testing and get more conversions.


Transcript provided for your reading convenience:

As testers, we all like good test results. We like to see our successes, and we like to see other people’s successes. Today I want to talk to you about some cautions you should use when you’re looking at other people’s test results; four things you need to avoid, and three ways to actually use people’s test results.

Many of you may remember that old website WhichTestOne. That was one of the first in the industry to showcase lots of test results. Most of the major companies who have tools for AB testing also have their own case studies they’re putting out. Case studies from them, case studies from other companies, and as testers, again, we love to see these, but we have to be cautious when we use them.

We’ve all been in the meeting where someone says, “Hey, XYZ company is doing this, and we should do that, too.” As testers, we know that’s not a good approach because we don’t know if that company’s tested anything. But when you bring some data to the table and someone says, “Hey, look, I ran this test and I got this lift,” for some reason we feel compelled that we need to take that into account and run our own tests on that same thing, or just take their advice for it and use those learnings and apply it to our site.

There are several problems, though, by taking other people’s test results and applying them to our site or even just running the test from that. I want to highlight what those four things are.

The first reason you should be cautious is because most test results show a lack of context. Context is king, and without the proper context, you don’t truly know if that lift was a good lift. For example, there are several things that make up good context. Let’s talk about those. Sometimes with test results, you don’t see the sample size. They may have a 500% lift, but if it was only on five visitors, that’s not a good enough sample size. Often when people share case studies they don’t always share all the details, for good reason. Some of that data may be sensitive. Even myself, when I share a case study with you guys, I’m not going to share all the data, because some of that data is sensitive, some of it’s proprietary, some of it we don’t want the world to know all the details. But I do want to share with you the learnings from those tests in case you can apply the principles of the test. So the first area where there’s lack of context is with the sample size. If you have a 500% lift but it’s only on five visitors, well, that’s not quite good enough.

Another thing that you won’t always see with any test result is the natural variance. They might have a lift, but what if the variance is just as high as that lift? Again, see my video on doing a variant study, but the natural variance is part of your test results, and you won’t always know what that is just by looking at those results. Another thing to be cautious of is the right metrics. When people are trying to promote themselves or their testing, often they’ll choose the metrics that look the best. If they’re using metrics to inflate the value of their test, you may not be able to know for sure if those are the right metrics to make an impact on the business. The other thing that we rarely see with test results is a comparison of other tests. Just because you see a test result, you don’t necessarily know that’s valuable because it’s a comparison of tests that truly show value.

Finally, most test results are never talking about the bottom line impact of the test on the business, or the lifetime value of the customer. All these things are the context that we should be considering if we’re truly data-driven and disciplined that we usually don’t get with other case studies and test results, for good reason. Sometimes it’s sensitive like I said, and sometimes it’s just too much to share. But for the most part, there’s a lack of context with tests that are being shared.

The second reason to be cautious of other people’s test results is because of a very important principle, and that is that these test results are their test results, and their test results apply to their visitors and their sites. Even if it’s a competitor of you in the same industry, those visitors coming to your site are there for a reason, and from what I’ve seen, whenever we’ve tried to replicate something that’s working somewhere else, it never works as good as just trying to find what works on your site for your visitors. So their results are theirs, and we need to be cautious when we just take those at face value, knowing that it’s a different audience, a different site, perhaps even a different industry.

The third reason that you should be cautious when you’re looking at other people’s test results is that in most times, test results and the learnings will be overgeneralized. For example, I saw a test result recently where someone had tested using logos of companies on a testimonial, rather than just saying the name or having an image of the person. They put a logo of the company. The takeaway from that test result was, so, if you want to have better results, you need to use company logos instead of face shots. That overgeneralization is something we need to be very cautious of that, yes, that may have worked on that test, but to overgeneralize and say that this works for everyone, man, we really have got to be cautious with that. We need to find out if that works for us and our business and our visitors. So that’s the third reason, be cautious of overgeneralization.

The fourth and most dangerous reason why we need to be cautious of other people’s test results is that often when we test results or show them to other people, they get this mentality like, “Great, now we know the answer. Now we don’t even need to test it.” That mentality of “we don’t need to test it now that we know”, is very dangerous.

So those are the four reasons to be cautious of other people’s test results. First, there’s the lack of context. Second, their results are their results. Third, there’s overgeneralization. And fourth, people have a tendency to want to not test when they’ve seen other people’s data.

Okay, so let’s talk about how you should use other people’s test results. Again, I’m a fan of learning from other people’s tests. I share test results myself. So how should we use them? There are three ways that I want to show you that you should use test results.

The first and most important is to drive ideation. Testing is about good ideas. When we see someone have success somewhere, naturally we want to take that idea and assimilate it and try it for ourselves. The important thing is that those test results are informing your ideas and they become tests for you as well. Don’t take them at face value and just implement it. The other thing that I would suggest as you’re using other people’s test ideas to drive ideation is to assimilate the idea, but then add more ideas to it. Don’t just say, “Wow, that worked more than a test on our site,” but say, “Hey, this worked, let’s test it, and then let’s create some more variations to see how we can push those boundaries of something that may have worked for someone else.

The second way to use other people’s test results is to see how what they did might apply to a broader principle. This is different from overgeneralization. Overgeneralization says that you should do this because it works. Everyone does it. A principle’s a pattern, and so if you’ve run 20 tests that have a certain pattern that is being exhibited with your site, and you see someone else’s test results that follow that same pattern, you might say, “Hey, that pattern matches what we’re already seeing. Because of that, we might want to test on our side as well.” Testing is looking for patterns with your site and your visitors, and you can apply those principles when you see that matching pattern.

The third and final way in which you might want to use other people’s test results is the least valuable and the one I would caution you against, but sometimes there’s a use case for it. When you’re a new site or a new business or a small business and you don’t have a lot of traffic and you’re trying to make the best site you can but you don’t actually have enough visitors to test with, then in that case looking at other people’s test results and saying, “Hey, this worked for them, I’m going to try with me because I know we’re not going to be testing,” that is a good use case. Again, first and foremost we should always be testing when and where possible, but there are sometimes where it doesn’t make a lot of sense to test if you don’t have the traffic, or if there are other constraints that don’t allow you to test for some reason. Then, yeah, you should apply as much knowledge as possible to the best pages possible, and if you have low traffic that’s a great time to say, “Hey, this worked for them, I’ll try it on my site as well.”

So should we use other people’s test results? The answer is yes. But there’s a right way to do it and a wrong way to do it. I hope this video’s been helpful in helping you see the ways that you should be doing it. I’ll also be posting my own successes and my own case studies with these videos. Now you know how to apply the principles in the right way and use case studies in the right way.

And if you found this video helpful, please hit the like button, subscribe to my channel, and you can also visit me at TestingTheory.com where I have courses that go in depth in some of these testing strategies and will get you better training so you can get better testing and more conversions.

Now, it’s your turn. I want to hear from you. When did you see an amazing test result? Share it in the comments below. I also want to make a quick shout out to Debra at GuessTheTest.com. If you’re looking for more test results that you like to see on a regular basis, GuessTheTest.com might be a perfect opportunity for you to sign up and see what people are submitting their test results and inform how that can help you and your business. If you’re interested in going to Guess The Test and seeing what they have to offer, I’ll put a link down below in the description.

You may also like

2600 Year Old AB Test

2600 Year Old AB Test

Prospect Theory, Loss Aversion, and A/B Testing

Prospect Theory, Loss Aversion, and A/B Testing
{"email":"Email address invalid","url":"Website address invalid","required":"Required field missing"}

Ready take your optimization to the next level?

The optimization master class by Rhett will teach you everything you need to know to master the experience optimization. Take your A/B testing to new levels as Rhett teaches you the foundational, intermediate, and advanced principles of conversion rate optimization.