You’ve been given budget to launch a new game, but what do you build? It’s hard to know what will resonate with the audience, and you’re worried about making the wrong decision.

The good news is we’ve been given some budget to experiment with a fantasy games concept, like we asked for
I really think this is a gap in the market, because there just aren’t that many good fantasy games
We can also use AI to generate the game once we decide on a concept, keeping costs low
However we need to provide some data to management that proves we’re building the right thing
The devs are ready and waiting, so let’s roll out some tests to see what resonates
I’m thinking maybe ogres? I know you were interested in a dragon game…

Performance branding is a marketing strategy that combines performance marketing principles with brand strategy. It uses in-market testing to prove which creative and messaging combinations work, before incorporating them into the wider brand as distinctive brand assets. This approach takes the risk out of the design and copywriting process, and gives teams solid justification for the normally ‘fluffy’ decisions they need to make in branding.
Tim Ferris’s book, “The 4-Hour Work Week” is a powerful example of how testing can lead to success. He A/B tested 6 different title options and ended up with the one that most resonated with his target audience. When planning a performance branding experiment, the first step is to come up with the ideas you want to test. The template provided can help teams make data-driven decisions, and calculate the budget required for the test.
Estimating the expected effect size is tricky, as by definition you don’t know how the test will perform until you run it. For most tests, 20% is used as the standard, as that was the average across 8,000 experiments. A case study from Money Dashboard revealed an unexpected insight that the best ad for one audience was the best ad for all. This counter-intuitive insight was only achieved through a robust creative testing process like performance branding.
In conclusion, performance branding offers a powerful solution to the brand vs performance debate. It takes the risk out of the design and copywriting process, and gives teams solid justification for the normally ‘fluffy’ decisions they need to make in branding. The template provided can help teams make data-driven decisions, and calculate the budget required for the test. Through a robust creative testing process like performance branding, teams can discover counter-intuitive insights that lead to success.

Performance Branding

0:01 Hey, let me show you how to use this template. So this is a creative testing template, and in here we input the creatives that we want to test, so the designs or images, and then the messaging as well.
0:14 So I'll explain what that means. But what we get out is the number of variations. So if we multiply every piece of copy against every image or creative then these are the different combinations we'll have, and we have six of them.
0:30 That's really important because the more variations we have, the more expensive the test is to run. So let's before we dive into how we do that calculation let me kind of walk through this example here.
0:43 So I'm using this example from Eric Schiffer's blog post on creative testing, which is one of the best out there.
0:51 It's kind of an homage. So he talks about concept, theme, and variant this creative testing framework. And he's saying that it's really important to split test at the top level, the really existential decisions, like if you're creating a game you want to know whether people want ogres
1:09 in the game or dragons, and it's gonna be a very different game if you choose one or the other. And you might not even be able to change back once you've made that decision.
1:18 That's normally an opinion led decision, but you can test it and you can see what the market wants. And you know, furthermore, at the conceptual level, once you decide it's ogres that they, that's what they want.
1:30 You can figure out different themes within you know, within that concept. So it could be a realistic ogre or a cartoon ogre.
1:39 So these are the types of decisions that you should test. But the types of decisions you shouldn't test are like the smaller changes, the small tweaks and variations like in this example, the border color.
1:55 You know, or it could be like the button text or whatever just like, you know, Facebook or TikTok or Google decide it might experience or you know, put your faith in high trust individuals like the designers or copywriters you work with.
2:10 So I've put that example in here with Ogas versus Dragons. One thing that Eric's post didn't mention is the copy.
2:18 So yes, Ogas might work on average, but but only paired with specific types of copy. So we wanna find those combinations.
2:28 And the three examples that in here, typical for gaming. So adventure is one big draw. Join the adventure. You know, you could talk in lots of different ways about adventure, but you wanna know conceptually does adventure you know, is that the draw to this game?
2:45 Equally, you know, some, everyone loves a challenge. If you talk about the game being difficult to complete or challenging then that would attract different types of people.
2:54 And we wanna know how many, right? Like, does adventure work better than challenge?  Equally friends, right? So if your game has social dynamics you wanna play with your friends, like maybe playing you know, dragging characters is more interesting with friends than with os.
3:12 So you wanna see how these things pair, and that's what's happening over here. So every time we add a creative or message, it pops into here.
3:21 So let me add one. You know, let's talk about rewards, right? So maybe you could earn money while's playing this game.
3:30 And you can see that we've popped an extra variation at the bottom. So this is message four. And then we have the combination.
3:37 So message four, against creative one and message four against creative two. These naming conventions are important because we found that my agency, when we were doing a lot of this testing if you don't get the naming conventions right in the URL parameters or in the campaign names or
3:54 ad names then you lost your information in terms of which campaigns were performing and what was actually being tested in those ads.
4:04 So you know, now you have a creative ID and a message id, and it's saved in this spreadsheet. You know, you shouldn't ever really have an issue when you go back to report on this.
4:14 Cool. So now getting into the meat of it the reason why this template exists is because everyone always wants to test more things than they actually can, right?
4:23 So, you know, maybe we add in fairies here and as well as rewards you know, potentially you know, there's another thing that's like, you know, on a beat, they get to the top of the ranking, whatever.
4:38 So everyone always has lots of good creative ideas, and they treat them as non-negotiable. But look what's happening to the budget.
4:45 It's costing 38 grand to run this test <laugh>, right? And how do we figure this out?  So we have 15 variations, that's just every creative times, every message.
4:56 And then we have a few other assumptions that we need to put in. So one is the conversion rate, and this is from, this is, you know, specifically for advertising although you could adjust this for anything.
5:06 It's the, you know, how many people convert or take like a cl you know, it could be optimizing to clicks or installs or purchases how many people convert divided by the number of impressions.
5:17 And, and if you haven't actually run any campaigns for this game then you can put in benchmarks for the industry.
5:24 So this would be the impression to install rate, and then the cost per conversion. In this case the, you know, the, the typical cost per install might be $4, right?
5:35 And changing these things actually flexes how much the cost is, right? So if you think you can get installed for $2, it's gonna significantly bring down the cost.
5:46 So I'm gonna change that back there. The final thing to talk about is the effect size. And this is why it's so important to only test big conceptual level changes and to not test Too many things because you know, the average expected impact of, of running a successful
6:05 test is about 20%. We ran 8,000 tests at ladder over six years, and we found that was the average, about 20% uplift if you, if you do get a successful result.
6:16 And only about a third of tests I think were successful. But you know, like you could actually adjust this assumption.
6:24 So if you expect lower than average results so you only expected 10% improvement because you're testing small things then your budget's 150 gram, nobody can afford that.
6:36 Yeah, unless you're a really big company the the other thing you could do is go the other way. So if you're testing really big things like oga versus Dragon versus ferries then you might reasonably expect to have a much bigger difference between the variations, right?
6:54 So it might be a 50% difference. That's probably the max I would go to. In terms of assumptions. I usually use 20% just because it gives me honest and helps me you know, prioritize these these variations because you know, typically you have too many ideas to test and
7:13 you need to, you know, decide which are the ones that are really existential to the business. So we're deciding, you know, we're not gonna care about ranking.
7:21 It was kinda like a fifth choice. We're not gonna talk about rewards because of course we wanna make money from playing games and we're gonna drop the ferries because we think that you know, we made the creative decision.
7:33 We think ogres versus dragons is the really important thing, and therefore the cost is, you know, 15 grand based on those assumptions.
7:42 And the reason that you have this template is really to make these prioritization decisions so you can work with your client or with your boss or the rest of your team, and set realistic expectations about how much it's gonna cost to run this test.
7:55 Once you have the result of the test then, you know, you use a different calculator, statistical significance calculator to check whether one actually performed better than the other.
8:05 And you can figure out which combination worked best. Then you're off to the races and you have some validated market data and you've chosen your brand based on what's actually likely to work.
8:17 And that's, that's really key and, and important. We call that process performance branding. So, you know, I like obviously heavily influenced by Eric's post here.
8:29 But you know, feel free to check this out. It's linked to in the template. It's based on my experience here.
8:35 And you know, essentially what a lot of smart people do in this space, which is actually kind of use testing to drive what, what works into the brand.
8:46 So you're not you know, you're not just making opinion driven decisions about what creative and what messaging to, to, to use, right?
8:54 Just kinda walks you through a few example cases talks you through the template like I just did, and show you a case study as well.
9:02 Okay? Hopefully that was useful. And feel free to use this template. Just give attribution.

What is the average expected impact of running a successful test?

What is the main purpose of the creative testing template?

What is the maximum expected difference you should assume when testing big creative concepts?

What is the recommended naming convention for campaigns and ads when doing creative testing?

Most creative decisions are made by opinion, but when it’s an existentially important topic, you really should be testing it in market first to see what works.

Vexpower Learning Certificate of Completion

Congratulations Abbie Sharman 💪