GoolyBib Offices – Website Planning Meeting

Your team has a number of test ideas you want to run, but you don’t have enough time, traffic, or resources to run them all. You need some framework or way of designing the experiment so as to get results quickly, without missing a winning combination.

Have you heard of orthogonal arrays?
It’s something I learned about in my MBA. From this guy called Taguchi. It’s a way of designing experiments so that you can get results quickly, but without leaving a potential winner on the table.
They work by allowing you to test a variety of different hypotheses at the same time. That way, you can find out which ones are most likely to be true quickly, and then focus on those.
Take a look at what you can find online, and see if you can figure out how to apply it.


In marketing we typically have more variables than we have time or money to test. We can decrease the scope of our testing program, but that may mean missing out on potential winning variations. Orthogonal arrays offer a solution to design tests in a way that you cover all of the major pairings of tests, but without having to run every single combination exhaustively. Orthogonal arrays were first developed by the statistician R. A. Fisher in the early 1900s. Fisher's work was motivated by the need for efficient designs for sampling from discrete distributions. He showed that certain types of matrices, now known as orthogonal arrays, could be used to construct efficient designs. Orthogonal arrays have since been used in a wide variety of applications, including statistical design and analysis, computer science, cryptography, and communication theory. More recently they were popularized by the work of Genichi Taguchi in the Ford Motor Company with Bob Moesta, the co-creator of the “jobs to be done” framework.
An orthogonal array is a matrix that allows you to map out all of your variables, and options for each variable, and then pick the least number of combinations to test. For example, if you were testing three variables, with two options for each variable, you would use an orthogonal array with three columns (for the three variables), and six rows (two options for each variable). If you were testing four variables, with three options for each variable, you would use an orthogonal array with four columns (for the four variables), and 12 rows (three options for each variable). And so on. In decreasing the number of experiments that need to be performed, this test design technique gets you to better results quicker, but without the risk of missing a potential winner. If a single value for a variable is performing badly, that’ll show up in the data wherever that value is present. If two values are interacting badly with each other that’ll also show up in that unique pairing (each pair only appears once). If more than two variables are interacting you’ll still see what region this issue shows up, and that gives you an idea for where to drill down for further testing.

Test Design

Hey, let's walk through how to use orthogonal arrays to decrease the number of tests you need to run without potentially missing any winners.
So these are the experiments we wanted to run. We have our hero text we have our description, and we have our call to action.
This is for our website. We have three different options for each. We don't know which combination of options is gonna work, so maybe next generation smart babe works really well with, you know, why is everything else in your house smart?
And that, and, and that combined with buy now works well. But, you know, we wanna test all these combinations and we, you know, don't want to potentially miss out on a winner.
So to test all these combinations, we'd have, you know, three different options. It's a power of three different variables, so that's 27 different variations.
So that would be quite a big test if we ran that all side by side. And most of those variations switch fail, right?
So we don't wanna run too many experiments. Were waste traffic take a lot longer to get results. And maybe the test would even be insignificant because, you know, we don't have enough traffic to test that.
So how do we cut this number of variations down? Well there's an article here which you can take a look at, which is linked on experiment design.
Kind of walks you through how to do this with a different example. There's also a link here to this lookup table.
So that's what I'm gonna show you is this lookup table here. So what, these are orthogonal arrays. And basically they're just a combination of pairs.
So you can see like, you know, we were three to the power three. This is three to the power forwards, the closest one that we have.
So just gonna copy and paste that across so can make that into a new grid. And then actually before you paste you wanna maybe just copy, click on that, and then go format as plain text so we don't lose the zeros.
There we go. So now that's paced across and we have all the different zeros. So the reason you can see it's pairs is you can see you know, we have a combination of zero and zero, zero and one, zero and two, one and zero, one and one, one and 2, 2, 0, 2, 1, 2, 2.
So like every potential pair, but it's not like every single combination. Like if you had three to the, you know, three to the power of four that would be 81 combinations, right?
So that's what this square is coming in down to the, just the most important nine combinations. And you know, we, we don't need three to the power four.
We need three to the power three. So we can actually just chop off the one on the end. So the way I kind of get this array out here is if I, if you go you know, this is 1, 2, 3, 4 and then I'm just gonna use the mid formula.
So we'll take a string and we're starting at one in this case, and we're just extracting a length of one.
And there's a couple things we need to do. We need to lock this with the dollar sign so that it, when you drag it across, it doesn't look for a different string.
And then we need to lock this with the dollar sign, just the number one.  So that we always should look at the top column here for this lookup.
Cool. So it's doing autofill here. So you can see it's extracted. Just the first thing here. 0, 0, 0, 1, 1, 1, 2, 2, 2. Cool. And if we drag this across, then it should split everything out for you.
Cool. So that seems to have worked. Now like we said, we don't need the actual last values here because we just have three variables.
And then all we need to do really is, you know, copy and then control or command shift v just to get the values here.
And now we have the specific values for the grid. Cool. So now we just need to replace our names to map our experiments.
So we have option A, B, and C for the hero text description is cta. So this is the hier text, this is the description, and then this, the cta.
Okay? And now we have all these like ones and zeros. So how do we, how do we replace those? So we're just gonna create a new grid here.
I'm just gonna say replace. And then if it's that or it's actually, maybe it's substitute be better. Yeah, so text to search, search for actually not this be easier with the V look up.
So it's just gonna v look up this. So yeah, there's usually multiple different methods you can, you can use to do replacements and stuff.
But I'm just gonna use this one. I think this will be easier. So index is one. So what am I doing here?
I'm taking this I'm gonna actually move this across, say like, I'll put my array here. Don't worry. I'll fill that in in a second.
I'm gonna press F four to put dollar signs in front of each of that. Cool. So so what this is, oh, this should be two.
So what this is saying is take this value of b2, look it up in this box over here, and then give me the second value or the second column in that box when it matches.
So right now there's nothing in that box but we're gonna put 0 1, 2 and then A, B, C and that means when there's a zero, we're gonna replace it with a, when there's one replace with B, when there's a two replace with C.
Cool. So we should be able to just drag that down and then drag that across. And there we go. So now we have A for zero and then C for two, B for one.
So we have that. So now we know which options we need. And you can just, you know, draw the over that.
If, if you want or in our new array maybe we just, you know, this is our array here. And then this is gonna variation, And This is 1, 2, 3.
Then just track that down. Cool. So now with nine tests, so nine test variations. Now we can test all these different variables and and, and we're gonna, you know, not gonna test every combination every, you know, of all 27.
But we're gonna get a good idea of what region shows the most promise. So for example, let's say we've run this experiment and let's say in the results you know, this is 0.3.
So this the 30% or yeah, 30% conversion. Really, this is 20%. Say it's all 30% on average. And then this one has 10%,  On average.
So you'd be able to see that you know, this looks like a really bad variation, right? This value of c is bringing down the average.
So that's something that you might wanna look at, right? Because that is like an issue, right? So, you know, whenever anything would match with value, see for the hear text it performed badly.
So you can look back and say, Okay, this doesn't, maybe an mention anti choking device in the copy is, is what's performing badly.
So that would be like a, a single variable anomaly, right? The second variable anomaly or, you know, pair-wise anomaly two mode anomaly, whatever you wanna call it, would be if you know, one of these pairs ends up performing badly or, or really well whichever, whichever way it goes.
So say for example maybe this a and B performed, you know, 50%, right? So that's a really good sign, right?
We have this one that, you know, performed really well. And if you look, you know, this is, this is the only combination where it's, you know, this is the only combination where the, that pair appeared, right?
Because you know, it's only pairs, right? So we have, we only have b value for hero text here. The only around three tests for hero text.
And this was the only one where that was paired with a now it's also the only one that was paired with C.
So one limitation here is we don't know if it's the B and if, you know, if it's the hero text and the description interacting here, or if it's the hero text and the CTA interacting or all three.
So that's one limitation of this result. But now we have an area that we can drill down. We know that this combination of you know, of bac seems to be the most fruitful.
And this is like the best performance. We know that we can probably rule out value C for the here text in the future.
And we know that we need to drill down into this. So what we could do is now that we know that it's you know, this, this combination is working really well we can start to kind of run all the, all the other potential combinations here as well.
So we can try B and A with A, B, and C, and we can try, you know the hear text B with you know, the three different descriptions and then C for the cta.
And so we can drill down and just kinda hold those constant and figure out what actually, was it the combination?
Was it the combination of being the hear text description B or is it like the CTA in hero text or whatever, what have you.
But you know, we don't have to worry about any of these tests. We don't have to worry about any of these tests.
So we've ruled out quite a few of the 27 different tests that we would normally have to run. And we have a variation that is performing better already with only running nine experiments.
So this is really helpful technique. You know, it's it's just one of many kind of test design, test test design and like prioritization techniques you can use.
But I found it really, really helpful, especially when there are lots of different variables that all might interact in different ways.
Okay, Hopefully that was helpful.


If you had 5 different variables you wanted to test, with 4 different options each, how many variations would that be?

How many tests could you run if you were using an orthogonal array for 5 variables and 4 options each?

If you have 3 options for 2 variables, you have 9 potential variations. Why can’t orthogonal array testing decrease that number?

What’s the benefit of using orthogonal array testing?

Can you think of any downsides to orthogonal testing?

You always have more ideas than you could test. You can brainstorm and come up with 9 good ideas in an hour, but that could be 3 ^ 3 = 27 different variations, and you're lucky to have the traffic to test maybe half that. It's frustrating because you don't want to leave potential winners on the table.

Vexpower Learning Certificate of Completion

Congratulations envious-robot-49 💪