VEXNOMICS OFFICE – THE DAY BEFORE A PITCH

It’s the day before pitching a new client, and the owner of the agency wants to know if they’ll be a good fit. The proposal requires 3 tests per week to satisfy what the client wants, but do they have enough traffic to support that level of testing?

I’ve got a quick task for you
We’re pitching 3 tests per month to a new potential client, but I’m concerned that they might not have enough traffic to support that.
Can you run a statistical significance calculation to figure it out for me?
They get about 8,000 people to the site every week, and conversion rate is 5%

Statistical significance is a function of the size of the impact of a test, and how many observations each variation has seen. You don’t know the size of the impact ahead of time, but you do know how much traffic you have. So it’s possible to calculate how many weeks worth of traffic it would take to detect a reasonable outcome from your test. From there you can work out how many tests you can run per month. It’s preferable to use weekly estimates because you shouldn’t cut off a test mid week, or day of week trends will affect your results (some tests perform better at weekends).
The actual maths can be complicated and is rarely used by actual practitioners. Usually they use test duration calculators, many of which can be found online for free. Typically they will ask for how much traffic you get on average, how many conversions you normally would expect from that traffic, and how many test variations you’re running. Then for a given MDE (Minimum Detectable Effect) – i.e. the expected uplift from the test variation over the control – you’ll know how many weeks it’ll take to run the test. The higher the MDE, the fewer the number of weeks the test will need to run for.
There are other parameters that affect the calculation, for example the confidence level (typically 95%), and the statistical power (usually 80%). The confidence level represents the probability of getting the same or better result if you repeated the test, so a 1 in 20 chance of being wrong. The statistical power is the probability of not accidentally rejecting a potential winning variation. It’s best to use the defaults for these unless you know what you’re doing, or you might find you get a lot of spurious results.

Test Duration

Hey, here's how to do pre-test analysis. So test duration calculations. So we're looking at the test calculated by CXL. This is my personal favorite. Although there are lots of free ones on the internet. This is just the easiest to use and I like the way that it lays out the minimum detectable effect.
Okay. So first thing you need to input is the weekly traffic. Let's say we have 10,000. Sessions or users coming to our website every week. And then how many conversions to normally get? So let's say it's like 300, so we have a conversion rate of three. Now we need to know the number of variants and that's including the control.
So two variants who had been AB test is the control. And B is the test that we're running. If we wanted to run, say, you know three tests a month presuming one would be a control, then we'd have four at variances. Okay. So with our 3% conversion rate and then the default confidence level as statistical power has descriptions here with those are I just use the defaults because you know, it's not worth messing with although it does change the calculus.
Then what it gives you is the number of weeks running the test in order to find this minimum detectable effect. And that minimum detectable effect is basically the impact of the test will have, that can be reliably detected. So in order to detect. If there's a 75% difference. So if you have like one test variation, which is 75% better than everything else then you'll be able to detect that in just one week, because it's such a big difference.
Then statistically, you know, the it's going to reach significance much faster. Because significance is a factor of not just how much traffic you have but also how big the differences. Now, if you have a much smaller difference say, you know 30% difference in, in conversion. Then you're going to need a lot more visitors preventing, and that translates to a lot more weeks to run the test.
So this is going to be five weeks in order to detect a 30% difference. Now you can play around with this say you wanted a 20% difference because that's the average. Effect that you normally get from your experiments or that's conservatively, what you would expect. Then you can drop the number of variants here and see, okay know, every five weeks I can get 21%.
You know, detectable effect, you know, 24% detectable effects. If I run, you know just two tests per month. So that's gonna be, you know, more realistic for me to achieve. And obviously this changes over time. So if you know, you grow your traffic and then, you know, next month you know, you get.
Yeah, 12,000 session. So week you know, then that this changes slightly. So you know, the other thing is if your conversion rate goes up, so now maybe you're getting like, you know, it's like 400. Conversion. So now you're paying 3.3% conversion rate instead of 3%, then are going to be able to reach significance much faster.
So that gives you a good sense of what these numbers mean. And just remember it's a function of conversion rate and also the size of the effect as well as how much traffic. So those are the three things you have to play play with. When you're figuring out how many tests you can run per month.

Assuming weekly traffic of 8,000 and 400 conversions, what’s the MDE for 4 weeks of running 3 test variations plus a control?

The average effect size for all the tests across the agency is 20%. Are you likely to get 4 significant tests per month for this client on that basis?

What implications does an assumed 20% MDE have for our recommendation regarding the client pitch?

The client’s add to cart conversion rate is significantly higher at 12.5% than their purchase conversion rate. Could we run 4 tests per month using this rate instead?

How many tests can we run

Before you take on a new client as a conversion optimization consultant, your boss wants to know if they’ll be a good fit. Do they have enough traffic to run a minimum of 3 tests per week? There’s a way to calculate that. 

Vexpower Learning Certificate of Completion

Congratulations brave-cybernetic-58 💪