"There's no such thing as a bad idea You spent some time with the team to generate different ideas for ads, and now you worry you might have too many! How do you prioritize?
Bank of Yu Offices - Friday Afternoon
You've just been doing a big creative refresh project for the FinTech app where you work, and the new designs and copy are due back today.
Bank of Yu You're working as a media buyer in a promising FinTech company called Bank of Yu. The app offers multiple features, for example tracking your financial goals, a brightly colored bank card and all the features you'd expect from a startup bank. The main value proposition however is the ability to loan money to friends in return for interest. You have been seeing success with Facebook ads, which is your main marketing channel. The creative has been getting a little bit stale of late, so the Head of Marketing commissioned new creative for you to use in testing. He wants to make sure that we maximize our learning from these new ads, and get them deployed as quickly as possible to improve performance.
FW: Creative Assets\nHey,\nWe've got the creative back - see attached.\nLet me know your plan for testing them.
Creative Testing Creative is the largest driver of performance in advertising, responsible for a whopping 47% of the impact of your campaigns. Despite that, it's hard to know in advance what will work: predicting human behavior is much harder than rocket science! There is always a power law distribution to ad performance: your best 20% of ads will generate 80% of your profits, and the rest will fail. These facts make creative testing the most impactful thing a marketer can do - if you aren't testing and learning what works, you're decreasing your odds of success.\nHowever it's not obvious how to approach creative testing: how many ads do you test at once, and which ones? It's common for marketers to enter into a testing schedule and see some performance gains, but then a few months later realize they haven't really learned anything, because they weren't systematic with their approach. Or the opposite problem: rolling out new creative is happening too slowly because the team is being overly scientific. Getting this balance right is key to success in marketing.
A simple but useful framework for creative testing is Eric Seufert's \"Concept - Theme - Variant\". Sorting your tests into larger concepts and smaller variations helps you figure out where to be scientific, and when to just let the algorithm decide. Choosing between Concepts is a big strategic decision that's harder to change, whereas iterating on border color or button text will only have a small effect and doesn't need to be A/B tested.
Some decisions are consequential and irreversible or nearly irreversible – one-way doors – and these decisions must be made methodically, carefully, slowly, with great deliberation and consultation... But most decisions aren’t like that – they are changeable, reversible... [they] can and should be made quickly by high judgment individuals or small groups.
How do we categorize our creative tests? With a number of ideas and potential for more, you need to categorize your creative into concepts, themes and variants.
Bank of Yu Office - Later that day
You have two tasks ahead of you: categorize your creative ideas into concepts, themes and variations, and then see how many of these combinations you can realistically test and still reach statistical significance.
A concept is the broader idea behind a campaign, the theme is a design that fits with the concept, and a variation is an iteration within the concept and theme. So for example the concept for an ecommerce store might be Luxury, or Eco-Friendly. The theme within Eco-Friendly might be showing the product being worn, or showing where the product is manufactured. Then there are hundreds of potential variations of adcopy or smaller design changes that fit the Eco-Friendly Concept + Place of Manufacture Theme. For example different angles of the factory, different borders on the image or different quotes from the factory workers.
What are the main themes (designs) present in the ads we received for Bank of Yu?
Take a look at the Meal ads: what one concept ties these three lines of copy together?
Without differentiation you have no brand.
Which Concept + Theme combination would you choose to differentiate the Bank of Yu brand on, and why?
How many tests can reach significance? You need a good solution for deciding how many to test at a time, and still reach statistical significance.
Bank of Yu Office - The next day
In order to know how many tests you can run, you need to understand how much it'll cost to reach statistical significance on your test. Luckily a friend has a template.
Yes I have a template for you\nOne of the geeks in IT put it together for me\nYou just input your CPM and CTR and it tells you what effect size you can test for, and how much that'll cost\nWe use it when plotting out our creative testing roadmaps with clients\nHope it helps!
Statistical Significance Statistical significance refers to the claim that a result from data generated by testing or experimentation is not likely to occur randomly or by chance but is instead likely to be attributable to a specific cause.
If at first you don't succeed, try two more times so that your failure is statistically significant
Test Duration Before you run an experiment, you should know how long you need to run it for. Statistical significance is a function of both effect size (how big off a difference you expect your test to make) and sample size (how many people will be exposed to the test). If you spend a lot on advertising and have a lot of traffic or impressions, then you can run multiple tests over a short period of time. However if you don't have a large budget and the tests you're running don't make a big difference individually, it can take months to reach significance. Knowing this tradeoff up front by calculating test duration can really help your testing strategy, because you can change your plans to ensure a better chance at a successful test. For example you can decrease the number of variations, increase the budget or go for bigger design or copy changes in the ads you're running.
Test Duration Hey I'm just going to use this test calculator to give you some intuition around the tradeoffs when you're running experiments\nSo if were running an A/B test - this is just two variations, the control and the test variation and this is a creative test on Facebook ads, I would expect the typical cost on Facebook ads to be $10 CPM and would expect the CTR to be 1%\nBased on these assumptions we can figure out how big of a difference we can detect\nIf we have a very small difference between the ads, like one has one word changed or a slightly different button text, then we'd expect the difference to be very small\nSmall differences take a lot longer to prove - we'd need 25m impressions and that might not even be possible with the audience\nIf we made a really big difference, for example changing the price of a product, we'd expect a 50% impact from the ad, and therefore we need a much smaller sample size: it would cost only $206 for this test\nOf course we don't know in advance what the effect size really is we're just estimating this in order to tell how long to run the experiment\nThe other thing you can do is flex these assumptions, so if we were running 6 variations, all the assumptions for budgets are going to change\nWe need $4000 to find a 20% impact for 6 variations\nIf the CTR is lower or if the CPM is higher, that will also mean we will have to spend a lot more money.\nFinal thing I'm going to show you is that a lot of the clients you talk to will ask you to optimize to conversions not clicks\nIt's very very expensive to test based on conversions not clicks - if we have a CTR of 1% and multiply it by a 20% conversion rate we get a full funnel conversion rate of impressions to conversions, and then the calculator works the same way. Now it costs us $20,000 to run this same experiment\nAdjust the numbers here to get to a reasonable estimate of how big an effect size you can detect, and how much budget you can use for the experiment.
We have 9 ad variations (1 concept x 1 theme x 3 variants = 9 ads) to test. Using the Test calculator, how much would it cost us in budget to detect a 20% uplift in CTR?
I was thinking\nWe shouldn't optimize to CTR - I don't care what ad gets clicks\nI need to know what gets conversions after the click, or we can't justify putting more budget into it\nCan you redo the plan to take that into account?\nLet me know when it's done
If you optimize to conversions instead of clicks, your test will take a lot longer and cost more budget. This is because you must factor in both the CTR and the conversion rate (CVR), meaning you have less observations for reaching significance. This might be worthwhile if it's very important to determine which ad drives more sales, but usually the cost and time difference is so large it usually makes sense to start with a CTR-based test anyway.
We can expect a 35% conversion rate on our ads. Factor this in to the click through rate (multiply them together) to recalculate the plan. How much budget do we need now with 9 variations to reach a 20% effect size?
What's the maximum number of variations we can afford to test with a budget of under $10k and an effect size of 20%, assuming we want to test to conversions not clicks?
Point taken!\nLet's start with the $5.8k budget for a click-based test of 9 variants\nThen we can decide if we want to secure more budget to confirm the winner works to drive conversions\nMaybe we can drop a few variants if they do badly on clicks\nLet's get these tests live
" 
