There are many ways to do website personalization wrong and a lot of what the CRO industry pitches can lead you astray. In this video, I will teach you the AL-Personalization Method to help you avoid personalization mistakes and show you 6 steps that will keep you from personalizing the wrong way.
This simple process and method will help you increase site conversions and avoid personalization mistakes.
Today we’re going to talk about the number one mistake that people make when they do website personalization and I’ll also show you the six steps you can take to avoid that mistake.
We’ll also use a couple of examples and a case study to walk through and help you illustrate how to do this. I’ll also show you the AL of personalization method which is your key to not making personalization mistakes.
Website personalization is when a business will show content or experience to a visitor because they think that experience is better. They have information about a visitor and so they show them something thinking that will improve their customer experience and hopefully give them more conversions.
The challenge with this is there’s a lot of things built into showing those assumptions and the whole industry has a lot of assumptions built into doing website personalization. We’ve all had that experience where someone knows something about us and so they show us some piece of content and often times it feels creepy, we’re like, “whoa hey you’re personalizing in a way that’s a little too much for me back off,”
You have to be aware that the industry is pitching this personalization thing as, something that will solve everything because you know this visitor but in reality, it has to be done strategically.
Here’s an example of why this is difficult, the industry says, personalization allows you to show the right content to the right person at the right time and you have these people here like hey that’s good for me.
In theory that sounds great the challenge is it is very hard to do in practice when I was a consultant at Adobe one of my clients was staples.com and the CEO had mandated it that they should do website personalization and so their optimization team their analytics team, their marking team, they all got together and they came up with a plan for how they should personalize.
Despite what we said we were the consultants they were paying us to consult them on and do the right way to give them the best practices and guide them through this experience. The route they chose was to just show content to visitors without testing it, without getting data and despite all of our recommendations to the to the contrary they went ahead with this plan because the CEO had asked them to do it and so we begged, we pleaded, we taught, we tried to educate, we helped them see that what they were doing wasn’t going to give them the value they wanted in the long run.
They were under the charges of the CEO and so, they just did it well. As it turns out two years later they had done all this personalization over two years and the CEO was like well, what do we have to show for it. What value has this provided? What impact has this had? The truth was they didn’t know and so they stopped their personalization program after two years of putting extensive resources, time, and energy into creating some personalization campaigns and doing something that ended up not having an impact.
So the moral of the story here is they made the staples made the number one personalization mistake they assume they know what to personalize to their visitors and because they assume they know what works they don’t measure or try alternative experiences.
Here’s another example of a company that says website personalization is easy to do and you know something about your visitors so you might as well personalize to them.
Visitor 4003 found your website by searching the internet for the term big airship for sale. He’s about to land on your homepage which displays the same generic content you show to everyone he finds your website but hold on a minute don’t we already know something about visitor 4003? Sure we do. We know he’s looking for an airship so, let’s focus the page on that. It’s also clear that size matters to this visitor so let’s list them in size order. Now when visitor 4003 arrives at your site he’s welcomed by content tailored to his specific interest in aims. What an impact that would have.
This is building all kinds of assumptions into what experience people want. This person is searching for a big airship, why don’t we just show them airships. We know they want a big one, why don’t we filter by size, but that’s with industry pitches.
You know something about this person you might as well just show them what you think they should see, based on what you know about them. It’s a very very limited view personalization, maybe it’s true maybe the visitor does want a big airship, however, assuming that we just give them the airship without their alternative products may be the budgets. The problem is that they want a big airship. Maybe we need to show them the smallest price first. Maybe, it’s not about the airship itself but it’s about the qualities and the features.
So, anytime that we just assume something will work because we have one little piece of data about a visitor that’s when we get challenges, that’s when we potentially set ourselves up to make mistakes.
You may have heard of the Darwin Awards. Those are the awards they give to people who do really stupid things that end up getting themselves killed. Basically the theory that they are self-limiting from the population.
For example, you may have heard the person that was in a nasty divorce and they were going to lose their house to their spouse. So, instead of giving them the house they decided they would go in and burn the house down. But they didn’t give themselves time to get out of the house and so they ended up dying in the burning house that they set on fire themselves.
Website personalization is kind of like that sometimes we think we have a great plan, that will be an amazing experience and it ends up just hurting ourselves.
So, let me illustrate why this doesn’t work. If you’re gonna personalize you have to have three things. You have to know who you want to personalize too, you have to have an audience, you have to know what you want to show them, and you have to know where you want to show it.
The problem is there’s an infinite number of combinations this leads to. The who. Like do I show first-time visitors, or new visitors, or people with an account, or people who came from Google or, people who come when they’re in the Firefox browser, or the Chrome browser, or people who have some certain cookies, or visit other websites, or came from a paid campaign the who is endless it goes on and on and then you have the what do you show. Well is it a message, is it a call-to-action, is that a promotion, is that a discount, is it an offer, is it some special thing that they should sign up for.
You also have to know where. The where is in this as well. You have sites and you have apps do you do it on your home page, on your landing page, on your product page, or your signup page, do you do it in the top navigation, or the hero banner if you do it below the fold do you as they’re checking out. There are all these places where you could do it.
The what, the where, and the who all combined to create an infinite number of combinations and because of the infinite amount of combinations anytime, we assume we know the right place, the right person, and the right thing to show to personalize, that’s a pretty big assumption. We’re good we’re just basing that on tons of guesses because of the infinite possibilities that are out there. The other problem with this number one mistake that people make, is they don’t measure alternative experiences when they just assume what works. The cost trade-off of personalizing to a small audience become very large now I’ll give an example of why that’s all so bad.
Basically, if you have a one hundred percent your population suppose you had 1,000 visitors that come to your site monthly. Just for the ease of math, and suppose you’re running a test and you get a 20% lift say your standard conversion rates 10% and you the 20% lift sets to lift that conversion rate to 12% so now your 20% lift gives you a hundred 20 conversions per month because you have a thousand visitors per month. Now suppose that you’re going to personalize to an audience. Suppose that the audience is fairly large, it’s some 40% of your total audience. Now of the 1,000 monthly site visitors you’re going to personalize to 400 of those site visitors to get that same value as testing to the entire audience. Your impact would have to be much larger so the 20% lift led to 20 extra conversions. To get that same 120 conversions with 400 members of your audience, you have to have like a 200% lift to get that same value. When you reduce the audience you have to increase the lift to make up for the benefit of a reduced audience. That’s another reason why without measuring anytime you assume an audience that’s smaller than the total population that you could test with. You actually increase the amount of lift that you have to get to improve the experience, which makes it very difficult. That’s the second issue with just personalizing by assuming you know the audiences.
The infinite combinations of who, what, and where, then the reduction in audience size makes it much more difficult to get the same value as if you’ve just optimized to your entire audience.
I want to walk you through an example of someone that did it right. About a month ago a client that I worked with came to me and they said we want to personalize on our homepage because we have people who have created an account and our members, and have logged in, and with Pete visitors who come to the site and they’re not members, they haven’t created an account, they haven’t logged in, and the hypothesis that they have, the business question was, we think that those people who aren’t logged in who haven’t created an account should have a different experience than people who have created an account.
On the surface this sounds like a great idea right, we know something about this visitor, some are members, some are not members. We know that they have an account and some don’t. You’d say that’s probably true. We probably should personalize or give them a new experience and so the product manager came to us and said let’s personalize. We have two variations that we’ve designed, ones for members, ones for those that are not members, ones that have not created an account. We want to show these two experiences and on the one on the variation that was for members. You had tools specific to them that they can only use behind the login. On the one that was for people who haven’t create an account, you had things like, are you new here, maybe you want to chat or maybe you want to learn more about some basic information and you had these icons that we’re trying to point them in the right direction based on who they were.
Again, on the surface that sounds like a pretty good test. But I want to walk you through what we actually ended up doing because this product manager is very wise. We talked about it and we said, we can show this experience to these people and target them so that the members get one thing and the non-members get another thing.
But the challenge is without comparing two alternatives we’ll never really know if that worked. I want to talk to you about the six steps that we need to do. That was step one. Step one is to as my friend James Alder says check yourself before you wreck yourself. What I mean by that is number one, check your assumptions, what are we assuming if we run this test. Well, we’re assuming the members and people who don’t have an account need different experiences.
Again, it’s a good hypothesis but it’s still an assumption at this point we haven’t proven that through testing. Step number two is to check your audience size. As we talked about if you have a small audience size they’re going to personalize to, the gain that you have to create to compensate for the smaller audience size is much larger and so before you just personalize to an audience you want to make sure that the audience is very large. You start with a larger audience first. You have to compare that and measure that against the value of just optimizing to their populations as a whole.
So, step number three is to design your variations for the audience’s that you think will work. What I mean by this is that the product manager actually did this in an example we were talking about. He had designed a variation for those that didn’t have an account and he had a design a variation for those that did have an account. That’s actually what you want to do. You want to create a new variation for those visitors and one of them should be the generic one, and one of them should be the personalized one and one personalized for each audience. Then once you have that you get to step four. Step 4 is the key part. This is the critical step that most people may miss when they’re doing website personalization. Step 4 is allowing all your visitors to see all your variations and you may be like, “that doesn’t make sense.” This is how you check your assumptions, you have to make sure that each audience responds positively to that variation.
The only way you can check your assumption is if you allow other audiences to also see that variation. Now, this is the exciting part! If you’re right in your assumption, in your original idea that personalization or this personalized experience is good for this audience, what you’ll see in the test results is you’ll see that audience doing very well with that variation. If that’s the case great, you succeeded, you were right, keep optimizing you keep moving on. However what usually happens is the audience you thought would win with that variation often doesn’t, or you might see a different audience respond so you allow all visitors to see all variations and then use segment after the fact to see which audience segment did the best, with each variation. That’s step four, allow all audiences to see all variations and see which audience does the best.
Step five is really interpreting results based on the pattern that the audience has had. So, in this example again we showed a member and a non-member experience and what we found was that the member and the people that didn’t have accounts, that were just regular visitors, their pattern of behavior was exactly the same. You can see this kind of arching pattern in the results. What that means is that we didn’t influence their behavior by giving them this custom experience. Now that’s what you want to see. To disprove the idea. If we have seen something different if we had seen one audience respond differently than the other then that would have been the case for personalization, but we saw was the exact same pattern and so by seeing the same pattern you’re able to prove or disprove the value of this personalization experience. That’s step 5, look at the pattern and see if the pattern is the same, or different between the audiences in question.
Step 6 is to evaluate what you learned in this case. We saw the pattern was exactly the same. We did some more analysis on it, we looked at the heat maps, we saw that people weren’t actually engaging in this content. Some people might say your personalization experience failed and with the transfers no it didn’t. We learned what didn’t work, and whenever you learn what doesn’t work, you learn something valuable and you get some next steps. So, as a takeaway, we realize that we need to create some drastically different experiences to personalize and we’re doing that now we’re working on some next steps and we’re gonna create some new variations to continue this experience, to see what does make this non-account member this new visitor to the site.
What does help them respond positively? We’re gonna figure that out! So, that we will be able to personalize to them and give them a custom experience that meets their needs. In the meantime though we learned a lot with this test, we learned that what we did didn’t work, we learned that we need to try something else, we had data now the back sis and we’re not going on assumptions and gut checks but we have something tangible that the organization can use to make informed decisions moving forward and that’s why this product manager did a great job. He didn’t just rely on his guesses, and opinions, and his desires. He allowed himself to follow a data-driven approach and he got results back. They’re now helping take really good action steps moving forward.
So, those are the six steps I just want to summarize real quick because they’re all very important. Step number one and two, to check yourself before you wreck yourself.
Step number one check your assumptions to make sure you’re not assuming things where you don’t have data.
Step number two check your audience size to make sure that the personalized audience will be large enough. If it doesn’t make sense.
Step number three is to actually design variations that you think would be good for that personalized audience.
Step number four, most important, show each of those variations to all audiences so, you can segment after the fact, which is step number five.
Segment after the fact and see if the pattern of behavior is the same or different. If it’s the same then you realize the personalization didn’t make sense. In this case, if it’s different you now have a case for personalization.
Finally, step number six, take what you learned. Make a plan of action and use what you learn by following this disciplined data-driven approach to make a new plan of action to get closer towards personalization.
Now I want to hear from you. When have you been personalized in a way that it made sense, and what did you do about it. Also, if you liked this video please hit the like button. Hopefully, this is valuable. If you learned something new, or if you had an insight, or an aha moment hit the like button. I also make videos every Thursday. I post them every Thursday’s so please subscribe so, you can get the future videos coming up. Then finally if you want to visit me at testingtheory.com you can sign up for my newsletter. you can sign up to get insider strategies and I’d love to see you there. Anyway thanks for joining me. Remember testing theory is the place where businesses come to get better a/b testing and higher conversions.
AUTO-GENERATED TRANSCRIPT PROVIDED FOR YOUR CONVENIENCE
Hi, I am Rhett and I’ve been doing A/B Testing and multivariate testing for over a decade. I worked at Adobe where I led a large optimization team of testing Consultants. I’ve also started testing agencies where we’ve helped other clients do testing.
I’ve also been in the trenches with a very large corporation helping the company itself learn to do testing. I’ve seen hundreds of companies run thousands of tests.
In this video, we’re going to review why every company should be running A/B tests, what is A/B Testing, where you can run your A/B tests, what you can test, and how you can actually get started doing it. I’ll also show a couple of quick case studies including one at the end of the video that helped Under Armour get a 14 percent lift that actually led to about 3 million dollars in annual revenue for them.
So let’s dive in and get started and I’ll show you why a be testing is so important what it is what you can test how you should do it and where you should do it.
To begin, I want to start out with a case study that comes from Microsoft where specifically for the Bing website one of the ideas they had for that was a simple change that would require a lot of time and effort. But because they were so many ideas in their testing program, this idea got deprioritize by a product manager. So they never actually ran the test this test and the idea languished in their queue for about six months until one day an engineer saw the test and that the test idea would be very easy to accomplish.
So the engineer coded up the test in just a few hours. They launched the test and within just a few hours they started seeing these triggers that there were abnormally high revenues and that something must have been wrong.
However, nothing was wrong. It was an a/b test and this one test variation increased their conversions by 12% which on an annual basis led to more than a hundred million dollars in revenue for Bing. This one test alone was Bing’s see Most valuable revenue-generating idea they’d ever tested. This example shows why a/b testing and multivariate testing are so important. Bing increased their revenue by 12% with a simple A/B test and that’s one of the main benefits of a/b testing. You can actually impact your bottom line and increase your revenue and your conversions by running A/B tests. Another benefit that this example illustrates is the ability to do rapid iteration.
You can Implement a new experience and whether it wins or loses you can do it quickly. So you win quickly and get gains to your business or you lose quickly and you learn what doesn’t work for your visitors. A/B Testing is also powerful because you’re using your actual visitors to give you the result. You’re not looking at just random data sets. You’re not just going out and asking people what they think. Your actual visitors who are coming to your digital properties are telling you what works and what doesn’t work.
You’re letting your data and your visitor’s guide your decisions your investment decisions your production decisions and the things that you put out to the world. These are just a few reasons why a/b testing is so powerful and why every company should be doing it.
So now you know why it’s so important. Let’s talk about what A/B Testing is. You may have heard this called by different names you may have heard of split testing or conversion rate optimization or multivariate testing or landing page optimization. You may have heard it called digital optimization or online experimentation or even growth hacking. These are all just different ways of saying you’re running tests. You’re learning about your visitors with actual data on your actual site.
Let me give you an example of what A/B Testing looks like in real time. You can see from this visual that there are four variations we are testing and they’re allowing visitors to get each of the variations. Visitors are coming and going and cycling through experiences. Some of them are good and experienced. Some of them are bad and overall you get a result that tells you the value and impact of that test you can see that the image on the right has the highest impact once the test is finished because that’s the one that visitors likes overall. It’s the one that impacted their visitor experience the most.
One of the nice things about doing A/B Testing is that your visitors don’t know they’re in a test. They are randomly assigned and they’re in your real life real time experience. They’re not in some dev or staging environment. They’re not in some focus group. They’re actually going through your digital experiences experiencing that you have and they don’t know that they’re being tested. This is really important because it gives an unbiased view of the experience and by using data directly from your visitors, you’re able to eliminate biases and guesses and opinions that go into building digital experiences.
So now that you know what testing is, let’s talk a little bit about where you can actually do those tests. You can run your A/B tests and multivariate tests on your homepage, your landing pages, your site content, or product pages, your conversion flows, and signups your checkouts.
You can also do your tests on your paid advertising. You might be running paid ads to get more people to your site. That’s a great opportunity to do A/B Testing with your paid ads. Do you have the right headlines? Do you have the right titles? Do you have the right images and thumbnails?
You can also test in your mobile apps. There’s an infinite number of things you can test on your site and in your mobile apps.
One of the things and places you can test is your marketing campaigns. You put out marketing content and you might want to test and see how well does that marketing content do relative to your other campaigns and other content.
And finally, another good place to test is in your emails. Basically, every time you send out an email and every visitor touchpoint of an email inside any digital experience that your visitors are where they’re engaging with you is a great opportunity to do A/B Testing.
So you can see there are all these visitor touch points where you can test but what can you test in those visitor touch points?
Well, there’s a lot you can test. Pages and flow pricing, headlines, videos and testimonials, social proof. You can test how much content you put above the fold. What kind of elements they see in each experience.
But you can go beyond just testing things on the page or where you put stuff. You can test your business model. For example, you might try testing if we did free shipping and we offered that through an a/b test.
By putting that in an a/b test you would see hey how much more revenue we make by offering free shipping and what was the tradeoff by comparing that to the control.
You can also test back-end algorithm changes. For example, Amazon might test when do we recommend these different types of products or when should we target a certain person or genre with this information or this product? You might also test if you’re adding new products or new services to the market. How do these do if we introduce them as part of a test by introducing them as part of a test you see the value and impact of that product or service and that allows you to evaluate the success in the short term and long term rho past that product or service so you can see there’s a lot of things you can test across a lot of places but let’s talk about how do you actually do that? I’m going to give you a quick overview of how do you get started? What are the basic steps you would need if you want to start doing A/B Testing step. Number one, you need a testing tool and there’s lots of tools out there see some of my other videos to see some reviews on those tools. There are some of them are free. Some of them are paid and depending on the size of your business and how much money you have and how big a team you have and see some of my other videos for more information on teams and growing a testing culture, but the tool you get may depend on where you’re at. The company but to do testing you need a testing tool. So that’s the first step. You need to get a testing tool step. Number two, you need to define success in order to do a good test. You have to know what success metrics are trying to influence. If you want to increase your Revenue per visitor because you’re selling products. That’s the main success.
If you want to increase your conversion rates and get more visitors signing up for a newsletter. Well, that’s your main success. But you have to choose one success you’re trying to influence so that you have a clear answer to each test. So that’s the second step to find your success metric the third step in doing A/B test is to decide what you want to test and I usually recommend beginning with a high-level question. You don’t want to start with something too. Granular. You want to start with something higher level and then once you find something works you kind of zoom in to get a result from there. Okay. That’s the third step decide what you want to test. What idea you want to try first usually start with something simple. Don’t go crazy big but just choose what you want to test the fourth step is to now create the variations based on the question or the thing you want to learn about.
Ideally you’re creating your variations.
Ethically first so you’re thinking here’s my hell of a question. And here are the three or four variations the map. They would answer that question.
You want to be very clear about the variations and ask what will I learn if this duration wins?
What will I learn with if this variation wins you need to know before you even start the test your code the test.
What will you learn? And what will you get if something wins or loses that way even if there’s a small lift or a big lift? You’re still learning about your site. So that’s the first step Define your variations and the fifth step is to now code those variations. So you might need to go to your creative people and have them do a design. You may need to get your front end developer involved so we can change the site experiences.
Now this the fifth step is to actually create the variation so that you can start the test step six is actively the test. So now that you’ve had a coated you have the designer work on it. You’re ready to go live. You want to make sure you have the measurements measurements in place.
Remember you want to see how does this test influence the main success measure and you want to make sure the measurements is tracking that so when your test result is very clear and easy to answer so that step 6.
The test make sure your measurements are in place step 7 is to analyze the results. Once your test is complete you’re going to go in and look at the results and see how each variation did relative to the others. Now. There’s a lot that goes into results analysis.
So you might want to check out some other videos on that. But for now just know that you’re analyzing the results to see which one had the biggest impact step 8 is an important part of the process the some people lump into the previous step and analyzing results, but I want to call it as a separate step because it’s Debbie. What you’re going to do is you’re going to document what you learn and what the recommended actions were. This is an important separate step. If you don’t document your learnings and evangelize those learning so that people know about it. You’re testing efforts become they become wasted people forget. It’s we have short-term memories.
So step 8 is critical you have to document what you learn and let the company and your everyone know about the results of the test. Step number nine is tied to step number eight. Once you’ve learned something. This usually leads you to more questions. Well this happened and so now we wonder about X y&z ideation is Step number nine. After you have some learnings and some recommendations from your previous test you now take those and you learn you create new ideas from those previous learnings.
So step number nine is ideation create ideas based on what you learned from the test and that leads us to step number ten Step Entertainment simple. You just repeat the process with your new ideas and you can go back to the beginning create a good business question. Make sure you have the right things in place and start a new testing Series.
So those are the quick and dirty 10 steps on how to get testing.
You want to get a testing tool you want to find success you want to begin with a high-level question and then create strategic variations to answer that question you then create your test with your by by having designers make things and developers code things like in the right measurements in place. When you activate the test, then you analyze the results you learn something from the test and make recommendations.
And then you get ideas for follow-up tests based on the previous results. And then you repeat the process and do it again to finish off. I want to show you a real life case study of a test that I hope perform when I Looking at Adobe as a consultant.
One of my clients was Under Armour.com and we help them run a test that led to a 12 percent lift, which created a massive game 3 million dollar annualized gain for them. The nice thing about this test is it works? So well you this was several years ago, but that is still the test results and learning are still implemented on their site today. So let me show you what that looks like. So this was the control as you can see, we’re just the regular home page. They have some products. They were showing that if you sub components and in this was the bottom of their page down here add a few items down here and the variation and I’ll show you these side-by-sides and see the difference to the variation. We add a simple recommendation Zone on their homepage.
The business question was how does adding recommendations to the homepage impact visitors revenue and their business their main success metric was Revenue per visitor.
So that’s all we did was a simple test. We didn’t do a lot of variations here because we want to test it the the model itself.
So these are the two variation side by side. You can see that the control and the homepage and the variation exactly the same. The only difference is the recommendations own insert in between the last couple components of the page.
And again, we’ve already talked about this there was a 14 percent lift in Revenue which led to about 3 million dollars in annualized income or revenue for Under Armour.
This was a huge gain and an experience that they still do today. In fact, I’ll show you that right now. Let me just go to Under Armour.com.
And if their own bed, you can see that somebody has changed a lot. Hopefully they’ve done lots of testing since then but as you come to the bottom of the page, you can see this a similar recommended product zone now.
I haven’t shot in her armor in a while and that’s why it is recommended. They must have like a most popular product algorithm going now, but this test was so successful. It’s something they still do on their side today from this example, you can see there’s a huge potential for doing A/B testing the right way. It can impact your bottom line. It can make you more money. You can bring you new clients customers.
You can help you improve your business business and your business models and your strategies.
Everyone should be doing A/B Testing now, I want to hear from you in the comments below if you can let me know why you want to do A/B testing or what if you’re already doing A/B Testing. What is your favorite part about doing a/b testing?
Also, if you found this video helpful, please hit the like button just like the video so other people can find as well. If you’re interested in more videos on a be testing an advanced strategies like a multivariate testing other testing techniques.
Go ahead subscribe. To the channel, so you’ll be the first to know when these new videos come out. I post new videos every Thursday. So go and hit the Subscribe button now also you can go to testing three.com or you can subscribe to my newsletter where you can learn more about a/b testing and other Advanced strategies about multivariate testing personalization and other topics related to digital optimization.
Thanks for joining me today testing theory is where marketers come to do better A/B testing and get more conversions.