PPC Insights 28: A/B Testing: 3 Ways Marketers Fool Themselves

00:00:00: Herzlich willkommen zum AdStrive PPC Insights Podcast.

00:00:04: Hi everyone, welcome back to a new episode of our PPC Insights podcast!

00:00:09: Today is our second episode where I am joined by Nora so you might know her from last time which is why we are not gonna do introductions anymore.

00:00:18: if you don't know who she is just go back to the last episode and you will find out.

00:00:23: And today we are not gonna talk about dashboards and quantitative and qualitative data, but we are gonna talk about A/B testing.

00:00:31: Which is probably kind of the next step forward, when we actually found out what's wrong and what might be the cause

00:00:38: So let's say, we are moving to A/B testing today and Nora is gonna tell us about I think three different ways where marketers fool themselves.

00:00:49: and let's find out about 

00:00:51: it.

00:00:51: Yeah, it's one of my favorite topics or I have to say my favorite topic is A/B testing, because you can reach so good results with so surprising changes

00:01:03: So yeah, I urge everyone to try it.

00:01:06: And in the end you get more sales and more leads for kind of the same price right

00:01:10: if you can improve the conversions on your website

00:01:13: yeah

00:01:14: like

00:01:14: this is very important even especially important in our days I think with the economic crisis with the prices increasing

00:01:24: in even paid ads and you know everything that you do in marketing so you have to

00:01:30: like take with every coin and get the most out of your website or app 

00:01:39: Very true.

00:01:39: So, today we're going to talk about traps and maybe you can give us some more background info.

00:01:45: What is the first trap?

00:01:47: And how are we fooling

00:01:48: ourselves.

00:01:50: Yes, because I have seen it many times that teams implemented tests, run the test, announce a winner, they implemente it on the site and then they see nothing.

00:02:06: the revenue didn't change or even declined and you know that's because A/B testing or doing changes is tempting but if you are too impatient - as human beings we are - you might call a victory too early 

00:02:27: you know you start running an A/B test on Monday and then you check out the data on Wednesday

00:02:35: you can see that your variation is better than the original version with I don't know twenty percent

00:02:42: and then you stopp the test and implement the change

00:02:46: that's not a good process, because you are actually calling the winner too early 

00:02:53: you didn't leave enough time

00:02:55: the sample size was probably too small...and there was a statistical noise 

00:03:03: it's like you know flipping a coin

00:03:05: if you flipp a coin five times and you get four heads then you know you cannot say that the coin is rigged or manipulated to give you four heads it's just you didn't flipp it enough time 

00:03:21: in order to reach the valid fifty-fifty percent chance to have both sides of the coin. So

00:03:32: I say that defining sample size and the time frame when you are running a test is very important. And you have to 

00:03:45: take into consideration the business cycle that is 

00:03:49: valid for your business. For some businesses it might be a week but for some other 

00:03:55: it might be two weeks. You know there is a seasonality here also even between weekdays and weekends.

00:04:02: So, that's why we used to say that for running a test, running a test for at least two weeks is the minimum and the best way to use it and you can

00:04:17: trust in the A/B testing tools, because it really has a thorough statistical knowledge behind it

00:04:25: and it will announce a winner when it reaches the statistical significance.

00:04:30: So it's better to wait for it than to run into early conclusions.

00:04:35: Yeah, I'm a very impatient person I have to say.

00:04:38: I think all of my colleagues would agree.

00:04:40: So, I need to work on myself when it comes to A/B testing I see.

00:04:44: And I will try to be a little bit more calm and wait for an outcome

00:04:49: So.

00:04:50: You can watch the results meanwhile, you know.

00:04:52: I'm impatient as well!

00:04:54: I love data, but I just wanna see that it says that yes your version two is the winner.

00:05:01: (laughing)

00:05:02: It's a little bit like watching a movie or a race, right, so where you see two ... So, that is the first one.

00:05:10: Now we are a kind of moving on to the second trap.

00:05:14: So what is that?

00:05:15: Yes!

00:05:16: So the second most common trap that I see is optimizing for a proxy metric instead of a real business metric or KPI. So what is a proxy metric?

00:05:29: It's something which might be a micro conversion but maybe even less like a micro conversion can be an add to cart event in your web-shop but something which even less is just like a button-click you know.

00:05:44: But your business KPI might be a purchase or for a B2B-business a form submission.

00:05:54: So, it's very important to run the test not only towards a proxy metric but also at least as a secondary conversion to a business KPI

00:06:08: for example if there is a B2B-site and they wanna improve form submissions and they change the button which says "request a quote" from light blue color to vivid red color

00:06:29: and they even change the text on it for let's say "save your money"

00:06:35: they might see a spike in the clicks on the button, but it's not necessarily mean that the form submissions will improve as well and increase

00:06:47: So they might see the clicks-skyrocket for the button

00:06:50: but actually the sales team doesn't necessarily get more qualified leads

00:06:56: so it can be misleading!

00:06:59: And you might optimize one step but you broke the journey.

00:07:04: That's why I say, having multiple goals for a test including business metrics as well is 

00:07:14: very important.

00:07:14: I think so. It makes a lot of sense.

00:07:15: I have kind of an example from the real world, where one of our clients said ... I think it was something like "try our software for free 

00:07:26: no credit card info required"

00:07:27: But there actually was credit card info required.

00:07:30: So obviously they had like a lot of clicks, but people were kind of fooled

00:07:35: so as you mentioned just like the click cannot always tell the whole

00:07:40: story.

00:07:40: Yeah, it can be harmful, because you can cause extra work for your sales team.

00:07:45: Ja, very true!

00:07:47: That's also like trowing out money on window...

00:07:49: Definitely, ja.

00:07:50: So, keep that in mind!

00:07:52: We do have a great reminder.

00:07:54: and then we are moving on to our third and our fourth trap.

00:07:59: Ahm,

00:08:00: so the third one is testing without a clear hypothesis.

00:08:07: As we said we are impatient.

00:08:09: So sometimes marketing teams want to do like a whole redesign of a homepage. They at the same time change the headline, change the hero-image, probably move the pricing-table and shorten the form.

00:08:28: That's like four different changes.

00:08:31: And you know even when they do a test on this and they see that their version is winning - I don't know by 

00:08:40: twenty percent over the original one

00:08:42: they do not know, why was it better?

00:08:47: Was it because of the better headline? Was it because of the more compelling hero-image?

00:08:54: Was it because the price-table was easily reachable? Or you know, was it because the form was shorter?

00:09:03: And basically, they didn't learn anything about their users.

00:09:06: So, you know, you have to be very clear, because based on these results you will not be able to optimize your next campaign.

00:09:20: So, when we say for example that you know, we have to change the headline, because we have to address the pricing objections, because we think that people are leaving because or bouncing

00:09:40: when they see the costs, that's a clear thing, a clear hypothesis.

00:09:45: and you solely focussing on that message and that item

00:09:50: and that's way more useful than changing everything and not knowing why is it better than the previous one.

00:09:59: Yeah, that makes a lot of sense.

00:10:02: And I think we are now kind of moving on to our last question, which would be, if our team is willing to test to launch a test tomorrow or maybe you are planning to launch a test tomorrow, 

00:10:15: what is kind of the final piece of advice that you would give us?

00:10:20: Well if you are like virgins in testing let's say then print out a test card or write it on a board In your office.

00:10:34: Just three sentences or three questions.

00:10:37: like we are testing x-thing, like we are testing the headline or we are testing the button

00:10:46: Know what you are testing

00:10:49: and be more specific and focus on one thing.

00:11:00: The next sentence that you can write on this board is like we are expecting an impact in x, y, z business metric, whether it's in purchases, in revenue, in form submissions, as you mentioned.

00:11:11: But one very important business KPI.

00:11:15: And the third one is, that you have to say that you are not looking or we are not looking at the results until at least this deadline and keep in mind your business cycle. If you follow this simple advice then probably you will avoid like ninety percent of the pitfalls that we just discussed 

00:11:44: And that could actually create a competitive advantage like comparing to your competitors.

00:11:48: Yeah, that makes a lot of sense.

00:11:49: So we actually need like a clear plan and like a clear idea, what we wanna test, how long we wanna test it 

00:11:56: and we need a little bit of patience

00:11:58: Yeah.

00:11:59: Which I don't have, but I'm working on it.

00:12:01: We are both working on it

00:12:01: I'm cheering for you!

00:12:05: Thank you for listening, until next time!

00:12:07: And thank you Nora for sharing your piece of advice about A/B testing.

00:12:13: Thanks for having

00:12:13: me.

00:12:15: Das waren die PPC Insights der AdStrive Podcast rund um das Thema Performance Marketing.