April 11

#1 Website Personalization Mistake and 6 Steps to Avoid It


There are many ways to do website personalization wrong and a lot of what the CRO industry pitches can lead you astray. In this video, I will teach you the AL-Personalization Method to help you avoid personalization mistakes and show you 6 steps that will keep you from personalizing the wrong way.

This simple process and method will help you increase site conversions and avoid personalization mistakes.

Transcript provided for your convenience.

Today we’re going to talk about the number one mistake that people make when they do website personalization and I’ll also show you the six steps you can take to avoid that mistake.

We’ll also use a couple of examples and a case study to walk through and help you illustrate how to do this. I’ll also show you the AL of personalization method which is your key to not making personalization mistakes.

Website personalization is when a business will show content or experience to a visitor because they think that experience is better. They have information about a visitor and so they show them something thinking that will improve their customer experience and hopefully give them more conversions.

The challenge with this is there’s a lot of things built into showing those assumptions and the whole industry has a lot of assumptions built into doing website personalization. We’ve all had that experience where someone knows something about us and so they show us some piece of content and often times it feels creepy, we’re like, “whoa hey you’re personalizing in a way that’s a little too much for me back off,”

You have to be aware that the industry is pitching this personalization thing as, something that will solve everything because you know this visitor but in reality, it has to be done strategically.

Here’s an example of why this is difficult, the industry says, personalization allows you to show the right content to the right person at the right time and you have these people here like hey that’s good for me.

In theory that sounds great the challenge is it is very hard to do in practice when I was a consultant at Adobe one of my clients was staples.com and the CEO had mandated it that they should do website personalization and so their optimization team their analytics team, their marking team, they all got together and they came up with a plan for how they should personalize.

Despite what we said we were the consultants they were paying us to consult them on and do the right way to give them the best practices and guide them through this experience. The route they chose was to just show content to visitors without testing it, without getting data and despite all of our recommendations to the to the contrary they went ahead with this plan because the CEO had asked them to do it and so we begged, we pleaded, we taught, we tried to educate, we helped them see that what they were doing wasn’t going to give them the value they wanted in the long run.

They were under the charges of the CEO and so, they just did it well. As it turns out two years later they had done all this personalization over two years and the CEO was like well, what do we have to show for it. What value has this provided? What impact has this had? The truth was they didn’t know and so they stopped their personalization program after two years of putting extensive resources, time, and energy into creating some personalization campaigns and doing something that ended up not having an impact.

So the moral of the story here is they made the staples made the number one personalization mistake they assume they know what to personalize to their visitors and because they assume they know what works they don’t measure or try alternative experiences.

Here’s another example of a company that says website personalization is easy to do and you know something about your visitors so you might as well personalize to them.

Visitor 4003 found your website by searching the internet for the term big airship for sale. He’s about to land on your homepage which displays the same generic content you show to everyone he finds your website but hold on a minute don’t we already know something about visitor 4003? Sure we do. We know he’s looking for an airship so, let’s focus the page on that. It’s also clear that size matters to this visitor so let’s list them in size order. Now when visitor 4003 arrives at your site he’s welcomed by content tailored to his specific interest in aims. What an impact that would have.

This is building all kinds of assumptions into what experience people want. This person is searching for a big airship, why don’t we just show them airships. We know they want a big one, why don’t we filter by size, but that’s with industry pitches.

You know something about this person you might as well just show them what you think they should see, based on what you know about them. It’s a very very limited view personalization, maybe it’s true maybe the visitor does want a big airship, however, assuming that we just give them the airship without their alternative products may be the budgets. The problem is that they want a big airship. Maybe we need to show them the smallest price first. Maybe, it’s not about the airship itself but it’s about the qualities and the features.

So, anytime that we just assume something will work because we have one little piece of data about a visitor that’s when we get challenges, that’s when we potentially set ourselves up to make mistakes.

You may have heard of the Darwin Awards. Those are the awards they give to people who do really stupid things that end up getting themselves killed. Basically the theory that they are self-limiting from the population.

For example, you may have heard the person that was in a nasty divorce and they were going to lose their house to their spouse. So, instead of giving them the house they decided they would go in and burn the house down. But they didn’t give themselves time to get out of the house and so they ended up dying in the burning house that they set on fire themselves.

Website personalization is kind of like that sometimes we think we have a great plan, that will be an amazing experience and it ends up just hurting ourselves.

So, let me illustrate why this doesn’t work. If you’re gonna personalize you have to have three things. You have to know who you want to personalize too, you have to have an audience, you have to know what you want to show them, and you have to know where you want to show it.

The problem is there’s an infinite number of combinations this leads to. The who. Like do I show first-time visitors, or new visitors, or people with an account, or people who came from Google or, people who come when they’re in the Firefox browser, or the Chrome browser, or people who have some certain cookies, or visit other websites, or came from a paid campaign the who is endless it goes on and on and then you have the what do you show. Well is it a message, is it a call-to-action, is that a promotion, is that a discount, is it an offer, is it some special thing that they should sign up for.

You also have to know where. The where is in this as well. You have sites and you have apps do you do it on your home page, on your landing page, on your product page, or your signup page, do you do it in the top navigation, or the hero banner if you do it below the fold do you as they’re checking out. There are all these places where you could do it.

The what, the where, and the who all combined to create an infinite number of combinations and because of the infinite amount of combinations anytime, we assume we know the right place, the right person, and the right thing to show to personalize, that’s a pretty big assumption. We’re good we’re just basing that on tons of guesses because of the infinite possibilities that are out there. The other problem with this number one mistake that people make, is they don’t measure alternative experiences when they just assume what works. The cost trade-off of personalizing to a small audience become very large now I’ll give an example of why that’s all so bad.

Basically, if you have a one hundred percent your population suppose you had 1,000 visitors that come to your site monthly. Just for the ease of math, and suppose you’re running a test and you get a 20% lift say your standard conversion rates 10% and you the 20% lift sets to lift that conversion rate to 12% so now your 20% lift gives you a hundred 20 conversions per month because you have a thousand visitors per month. Now suppose that you’re going to personalize to an audience. Suppose that the audience is fairly large, it’s some 40% of your total audience. Now of the 1,000 monthly site visitors you’re going to personalize to 400 of those site visitors to get that same value as testing to the entire audience. Your impact would have to be much larger so the 20% lift led to 20 extra conversions. To get that same 120 conversions with 400 members of your audience, you have to have like a 200% lift to get that same value. When you reduce the audience you have to increase the lift to make up for the benefit of a reduced audience. That’s another reason why without measuring anytime you assume an audience that’s smaller than the total population that you could test with. You actually increase the amount of lift that you have to get to improve the experience, which makes it very difficult. That’s the second issue with just personalizing by assuming you know the audiences.

The infinite combinations of who, what, and where, then the reduction in audience size makes it much more difficult to get the same value as if you’ve just optimized to your entire audience.

I want to walk you through an example of someone that did it right. About a month ago a client that I worked with came to me and they said we want to personalize on our homepage because we have people who have created an account and our members, and have logged in, and with Pete visitors who come to the site and they’re not members, they haven’t created an account, they haven’t logged in, and the hypothesis that they have, the business question was, we think that those people who aren’t logged in who haven’t created an account should have a different experience than people who have created an account.

On the surface this sounds like a great idea right, we know something about this visitor, some are members, some are not members. We know that they have an account and some don’t. You’d say that’s probably true. We probably should personalize or give them a new experience and so the product manager came to us and said let’s personalize. We have two variations that we’ve designed, ones for members, ones for those that are not members, ones that have not created an account. We want to show these two experiences and on the one on the variation that was for members. You had tools specific to them that they can only use behind the login. On the one that was for people who haven’t create an account, you had things like, are you new here, maybe you want to chat or maybe you want to learn more about some basic information and you had these icons that we’re trying to point them in the right direction based on who they were.

Again, on the surface that sounds like a pretty good test. But I want to walk you through what we actually ended up doing because this product manager is very wise. We talked about it and we said, we can show this experience to these people and target them so that the members get one thing and the non-members get another thing.

But the challenge is without comparing two alternatives we’ll never really know if that worked. I want to talk to you about the six steps that we need to do. That was step one. Step one is to as my friend James Alder says check yourself before you wreck yourself. What I mean by that is number one, check your assumptions, what are we assuming if we run this test. Well, we’re assuming the members and people who don’t have an account need different experiences.

Again, it’s a good hypothesis but it’s still an assumption at this point we haven’t proven that through testing. Step number two is to check your audience size. As we talked about if you have a small audience size they’re going to personalize to, the gain that you have to create to compensate for the smaller audience size is much larger and so before you just personalize to an audience you want to make sure that the audience is very large. You start with a larger audience first. You have to compare that and measure that against the value of just optimizing to their populations as a whole.

So, step number three is to design your variations for the audience’s that you think will work. What I mean by this is that the product manager actually did this in an example we were talking about. He had designed a variation for those that didn’t have an account and he had a design a variation for those that did have an account. That’s actually what you want to do. You want to create a new variation for those visitors and one of them should be the generic one, and one of them should be the personalized one and one personalized for each audience. Then once you have that you get to step four. Step 4 is the key part. This is the critical step that most people may miss when they’re doing website personalization. Step 4 is allowing all your visitors to see all your variations and you may be like, “that doesn’t make sense.” This is how you check your assumptions, you have to make sure that each audience responds positively to that variation.

The only way you can check your assumption is if you allow other audiences to also see that variation. Now, this is the exciting part! If you’re right in your assumption, in your original idea that personalization or this personalized experience is good for this audience, what you’ll see in the test results is you’ll see that audience doing very well with that variation. If that’s the case great, you succeeded, you were right, keep optimizing you keep moving on. However what usually happens is the audience you thought would win with that variation often doesn’t, or you might see a different audience respond so you allow all visitors to see all variations and then use segment after the fact to see which audience segment did the best, with each variation. That’s step four, allow all audiences to see all variations and see which audience does the best.

Step five is really interpreting results based on the pattern that the audience has had. So, in this example again we showed a member and a non-member experience and what we found was that the member and the people that didn’t have accounts, that were just regular visitors, their pattern of behavior was exactly the same. You can see this kind of arching pattern in the results. What that means is that we didn’t influence their behavior by giving them this custom experience. Now that’s what you want to see. To disprove the idea. If we have seen something different if we had seen one audience respond differently than the other then that would have been the case for personalization, but we saw was the exact same pattern and so by seeing the same pattern you’re able to prove or disprove the value of this personalization experience. That’s step 5, look at the pattern and see if the pattern is the same, or different between the audiences in question.

Step 6 is to evaluate what you learned in this case. We saw the pattern was exactly the same. We did some more analysis on it, we looked at the heat maps, we saw that people weren’t actually engaging in this content. Some people might say your personalization experience failed and with the transfers no it didn’t. We learned what didn’t work, and whenever you learn what doesn’t work, you learn something valuable and you get some next steps. So, as a takeaway, we realize that we need to create some drastically different experiences to personalize and we’re doing that now we’re working on some next steps and we’re gonna create some new variations to continue this experience, to see what does make this non-account member this new visitor to the site.

What does help them respond positively? We’re gonna figure that out! So, that we will be able to personalize to them and give them a custom experience that meets their needs. In the meantime though we learned a lot with this test, we learned that what we did didn’t work, we learned that we need to try something else, we had data now the back sis and we’re not going on assumptions and gut checks but we have something tangible that the organization can use to make informed decisions moving forward and that’s why this product manager did a great job. He didn’t just rely on his guesses, and opinions, and his desires. He allowed himself to follow a data-driven approach and he got results back. They’re now helping take really good action steps moving forward.

So, those are the six steps I just want to summarize real quick because they’re all very important. Step number one and two, to check yourself before you wreck yourself.

Step number one check your assumptions to make sure you’re not assuming things where you don’t have data.

Step number two check your audience size to make sure that the personalized audience will be large enough. If it doesn’t make sense.

Step number three is to actually design variations that you think would be good for that personalized audience.

Step number four, most important, show each of those variations to all audiences so, you can segment after the fact, which is step number five.

Segment after the fact and see if the pattern of behavior is the same or different. If it’s the same then you realize the personalization didn’t make sense. In this case, if it’s different you now have a case for personalization.

Finally, step number six, take what you learned. Make a plan of action and use what you learn by following this disciplined data-driven approach to make a new plan of action to get closer towards personalization.

Now I want to hear from you. When have you been personalized in a way that it made sense, and what did you do about it. Also, if you liked this video please hit the like button. Hopefully, this is valuable. If you learned something new, or if you had an insight, or an aha moment hit the like button. I also make videos every Thursday. I post them every Thursday’s so please subscribe so, you can get the future videos coming up. Then finally if you want to visit me at testingtheory.com you can sign up for my newsletter. you can sign up to get insider strategies and I’d love to see you there. Anyway thanks for joining me. Remember testing theory is the place where businesses come to get better a/b testing and higher conversions.

You may also like

2600 Year Old AB Test

2600 Year Old AB Test

Prospect Theory, Loss Aversion, and A/B Testing

Prospect Theory, Loss Aversion, and A/B Testing
{"email":"Email address invalid","url":"Website address invalid","required":"Required field missing"}

Ready take your optimization to the next level?

The optimization master class by Rhett will teach you everything you need to know to master the experience optimization. Take your A/B testing to new levels as Rhett teaches you the foundational, intermediate, and advanced principles of conversion rate optimization.