We've got too many test ideas

You always have more ideas than you could test. You can brainstorm and come up with 9 good ideas in an hour, but that could be 3 ^ 3 = 27 different variations, and you're lucky to have the traffic to test maybe half that. It's frustrating because you don't want to leave potential winners on the table.


Test Design

When you have more test variations than you have the time or resources to implement, orthogonal arrays are a magic trick that lets you still cover the same ground, but with far fewer experiments...More


Mike Taylor

Built a 50-person growth agency.
💪 Useful 0
😓 Difficult 0
🎉 Fun 0
😴 Boring 0
🚨 Errors 0
😕 Confusing 0
🤓 Interesting 0
Premium subscription required.
Excel experience recommended.
1. Scenario
GoolyBib Offices – Website Planning Meeting
Your team has a number of test ideas you want to run, but you don’t have enough time, traffic, or resources to run them all. You need some framework or way of designing the experiment so as to get results quickly, without missing a winning combination.
Gustav Gieger
at GoolyBib

Have you heard of orthogonal arrays?

It’s something I learned about in my MBA. From this guy called Taguchi. It’s a way of designing experiments so that you can get results quickly, but without leaving a potential winner on the table.

They work by allowing you to test a variety of different hypotheses at the same time. That way, you can find out which ones are most likely to be true quickly, and then focus on those.

Take a look at what you can find online, and see if you can figure out how to apply it.

This course is a work of fiction. Unless otherwise indicated, all the names, characters, businesses, data, places, events and incidents in this course are either the product of the author's imagination or used in a fictitious manner. Any resemblance to actual persons, living or dead, or actual events is purely coincidental.

2. Brief

In marketing we typically have more variables than we have time or money to test. We can decrease the scope of our testing program, but that may mean missing out on potential winning variations. Orthogonal arrays offer a solution to design tests in a way that you cover all of the major pairings of tests, but without having to run every single combination exhaustively. Orthogonal arrays were first developed by the statistician R. A. Fisher in the early 1900s. Fisher's work was motivated by the need for efficient designs for sampling from discrete distributions. He showed that certain types of matrices, now known as orthogonal arrays, could be used to construct efficient designs. Orthogonal arrays have since been used in a wide variety of applications, including statistical design and analysis, computer science, cryptography, and communication theory. More recently they were popularized by the work of Genichi Taguchi in the Ford Motor Company with Bob Moesta, the co-creator of the “jobs to be done” framework.

An orthogonal array is a matrix that allows you to map out all of your variables, and options for each variable, and then pick the least number of combinations to test. For example, if you were testing three variables, with two options for each variable, you would use an orthogonal array with three columns (for the three variables), and six rows (two options for each variable). If you were testing four variables, with three options for each variable, you would use an orthogonal array with four columns (for the four variables), and 12 rows (three options for each variable). And so on. In decreasing the number of experiments that need to be performed, this test design technique gets you to better results quicker, but without the risk of missing a potential winner. If a single value for a variable is performing badly, that’ll show up in the data wherever that value is present. If two values are interacting badly with each other that’ll also show up in that unique pairing (each pair only appears once). If more than two variables are interacting you’ll still see what region this issue shows up, and that gives you an idea for where to drill down for further testing.

3. Tutorial

Hey, let's walk through how to use orthogonal arrays to decrease the number of tests you need to run without potentially missing any winners.

GoolyBib Testing Matrix
4. Exercises
5. Certificate

Share This Course