How many tests can we run?

Before you take on a new client as a conversion optimization consultant, your boss wants to know if they’ll be a good fit. Do they have enough traffic to run a minimum of 3 tests per week? There’s a way to calculate that.


Test Duration

In order to reach statistical significance, a conversion test needs to run for a certain period of time, based on how much traffic you have...More


Mike Taylor

Built a 50-person growth agency.
💪 Useful 0
😓 Difficult 0
🎉 Fun 0
😴 Boring 0
🚨 Errors 0
😕 Confusing 0
🤓 Interesting 0
Premium subscription required.
Excel experience recommended.
1. Scenario
It’s the day before pitching a new client, and the owner of the agency wants to know if they’ll be a good fit. The proposal requires 3 tests per week to satisfy what the client wants, but do they have enough traffic to support that level of testing?
Charlotte Cook
at Vexnomics

I’ve got a quick task for you

We’re pitching 3 tests per month to a new potential client, but I’m concerned that they might not have enough traffic to support that.

Can you run a statistical significance calculation to figure it out for me?

They get about 8,000 people to the site every week, and conversion rate is 5%

This course is a work of fiction. Unless otherwise indicated, all the names, characters, businesses, data, places, events and incidents in this course are either the product of the author's imagination or used in a fictitious manner. Any resemblance to actual persons, living or dead, or actual events is purely coincidental.

2. Brief

Statistical significance is a function of the size of the impact of a test, and how many observations each variation has seen. You don’t know the size of the impact ahead of time, but you do know how much traffic you have. So it’s possible to calculate how many weeks worth of traffic it would take to detect a reasonable outcome from your test. From there you can work out how many tests you can run per month. It’s preferable to use weekly estimates because you shouldn’t cut off a test mid week, or day of week trends will affect your results (some tests perform better at weekends).

The actual maths can be complicated and is rarely used by actual practitioners. Usually they use test duration calculators, many of which can be found online for free. Typically they will ask for how much traffic you get on average, how many conversions you normally would expect from that traffic, and how many test variations you’re running. Then for a given MDE (Minimum Detectable Effect) – i.e. the expected uplift from the test variation over the control – you’ll know how many weeks it’ll take to run the test. The higher the MDE, the fewer the number of weeks the test will need to run for.

There are other parameters that affect the calculation, for example the confidence level (typically 95%), and the statistical power (usually 80%). The confidence level represents the probability of getting the same or better result if you repeated the test, so a 1 in 20 chance of being wrong. The statistical power is the probability of not accidentally rejecting a potential winning variation. It’s best to use the defaults for these unless you know what you’re doing, or you might find you get a lot of spurious results.

3. Tutorial

Hey, here's how to do pre-test analysis. So test duration calculations. So we're looking at the test calculated by CXL. This is my personal favorite. Although there are lots of free ones on the internet. This is just the easiest to use and I like the way that it lays out the minimum detectable effect.

AB+ Test Calculator by CXL
4. Exercises
5. Certificate

Share This Course