"Email from the CEO – subject: TV? As the Head of Growth of GoolyBib, you just bet big on TV as your next growth channel... but how do you measure the impact?
GoolyBib Office – Morning
You arrive at the GoolyBib office and take a look at your inbox. One email in particular jumps out at you – It's Gus, the CEO. You should probably check that message out first. What is it this time...
TV?\nHey,\nDo we have evidence on TV moving the needle??\nNeed an answer ASAP: prepping investor deck.
B2C2B: Wearable Tech for Babies As the Head of Growth at GoolyBib, a fast-growing DTC startup selling smart bibs, you started running a TV campaign 2 months ago. The wearable device for babies market is ultra-competitive, and potential customers need to be educated on your innovative new product: a smart bib that alerts you if your baby is choking on their food.\nAfter some success with Affiliates and Facebook ads, you bet big on TV as the next major growth channel for GoolyBib. However leveling up from purely digital campaigns to TV gets you out of your comfort zone: you've always used last click attribution and haven't had time to explore other methods.\nNow the CEO needs to see the numbers – this is critical for the future success of the company, and also your career! So let's get started...
Half the money I spend on advertising is wasted. The trouble is I don't know which half.
You text a friend to ask for advice
Your friend Dushyant runs performance marketing for Ultra, a fast-growing male skincare brand. They spend a lot on TV so you're hoping he can shed some light on how to get the numbers your CEO needs to see.
Hey\nHonestly I'd just start with a before / after analysis\nIf numbers are up compared to the same date range prior, I'd say that was probably due to your TV campaign!\nWe do surveys and a few other things, but it takes time\nJust keep it simple for now
All models are wrong, some are more useful than others You knew last-click attribution was far from perfect, but in this case it's useless: when there isn't a 'click', you're going to need another approach.
GoolyBib Office – Afternoon
On your friend's recommendation you decided to run a before and after analysis. In preparation you've pulled revenue data from your analytics platform, with a wide enough date range to make your comparison.
Before & After Analysis Let's take a look at what our analytics is telling us. The CSV has data from Google Analytics telling us our revenue by channel.\nOf course there is no 'TV' line item, because sales from TV will come through unattributed, or via other channels like an increase in brand term searches on Google.\nYour task is to do some basic analysis of the data to understand how channels have changed before TV and after the campaign launched.
How much revenue total did we generate from all sources?
The TV campaign began on the 23rd of August, 2020. How much did revenue change (in %) in the period after TV started running versus before it started running? Explain your method.
The TV campaign spent approximately $500k. Did it break even?
What 2 sources had the biggest percent change in revenue during the TV campaign?
Facebook Ads also saw an increase in revenue – what could explain this?
Matching spikes and dips to events and actions The before and after method bought you some time, but now you need to learn how the professionals do it: Marketing Mix Modeling.
GoolyBib Office – The Next Day
Unsatisfied with the simplicity of the before and after approach, you decide to dig deeper and find out how attribution is done by more sophisticated TV advertisers. You message your friend for more details.
Yeah sure, happy to help\nThe main thing we do is Marketing Mix Modeling\n[https://en.wikipedia.org/wiki/Marketing_mix_modeling](https://en.wikipedia.org/wiki/Marketing_mix_modeling)\nIt's basically just a linear regression model... the trick is to figure out what should go into the model.\nIt's like a 1-2 week exercise, we do it in-house once a quarter
Marketing Mix Modeling (MMM) Marketing Mix Modeling was introduced in the 1980s to match spikes and dips in sales to actions taken in marketing (and external factors). It can tell you the incremental impact of each part of your marketing strategy, so you can make data-driven decisions on where to allocate budget. Previously used mostly by Fortune 500 advertisers spending millions on SuperBowl ads, MMM is now the only future proof way to measure marketing ROI across channels, thanks to Apple, GDPR/CCPA and adblockers limiting marketers' ability to track users.\nAs a probabilistic attribution method, it can only give you top down insights on what channels are working, not user-level data on who bought what like more deterministic methods such as last click. It's also a more manual and technical process, not something you can easily automate or pull off the shelf. To master MMM effectively, you're going to have to change the way you think about marketing attribution.
Marketing mix modeling (MMM) is a time-tested method for measuring the impact of your marketing.
At Home – On Your Laptop
Finally after work you get back and open up your laptop... ready for more work. You never get the time to do the necessary research for your job in actual work hours, so you make it up at home. Nevermind getting ahead – it's the only way to keep up! Tonight's mission is to understand Marketing Mix Modeling you're researching. You find a useful looking blog post...
What marketing attribution questions can MMM answer?
Unlike user-level attribution which builds up from individual user behavior, marketing mix modeling works from the top down, associating events and actions with spikes and dips in sales. This allows you to still measure ROI and make strategic decisions even when theres no user level data, but you usually don't have enough data to tell how individual tactics, day-to-day optimizations or other smaller changes are working.
In the Econometric Model for Pie Sales, what variable had the biggest impact on sales (positive or negative), apart from the baseline?
Explain what the Adstock effect is, and what it's used for
A friend of the CEO – hiring a consultant MMM is a black box, and you don't have time to look inside – maybe it's time to call in a favor and bring in some outside help.
GoolyBib Office – Morning
Soon after you arrive at the office, you feel a tap on your shoulder. It's the CEO, inviting you to a 'quick' meeting on the TV performance. You share your opinion with Gus that MMM could be the way forward.
Sounds like the right path\nInvestors will want something substantial if we're going to score a Series A\nAn old friend of mine from college does this sort of thing\nI'll get in touch\nLeave it with me
Three days later
Marketing Mix Modeling\nHi there,\nGus connected us as he said you needed a marketing mix model built to explain the impact of TV?\nThis is a pretty common exercise for me, so I'm confident I'll be able to help!\nLet me pull together a list of the data you'll need and we can go from there.\nBest,
Data scientists spend 60% of their time on cleaning and organizing data. Collecting data sets comes second at 19% of their time, meaning data scientists spend around 80% of their time on preparing and managing data for analysis.
Data Request Checklist Data collection and cleaning are not only the most time consuming jobs, but also the tasks that Data Scientists, Statisticians and Econometricians enjoy the least! With Marketing Mix Modeling projects, at least some of this burden falls on the client (that means you!).\nA good vendor will usually send you a data checklist – an example of which you'll find below. This goes through each of the 4 P's of the Marketing Mix (Product, Price, Place, Promotion) and lists all the usual suspects in terms of data sources and activities that are likely to show up as significant in your model.\nMMM is a holistic exercise so you'll need to provide a lot of data on almost everything you can think of that might have driven sales, in order to give your vendor the best chance of building an accurate model. As a vendor will be unfamiliar with your business and what data you have accessible, you'll need to give guidance on what data is available, what variables you think are likely to drive sales, as well as help them get the data in the right format.
Getting ready: data collection & cleaning You didn't realize the modeling process would be so all-encompassing... you need a lot of data and you want to get it into the right format, so you don't waste the consultants' time.
GoolyBib Office – The Following Monday
Luckily you didn't have to do the hard work to pull the data – a friendly member of the analyst team did that for you. But you do have to get it into the right format for modeling...
Unstacking Long Format Data Take a look at the Google Analytics data in the Data Room folder, labelled 'GoolyBib-trafficsources-all-traffic'. Right now it's in 'stacked' or 'long' format, and we need to 'unstack' it for modeling. Each Source/Medium should be a new column like in the image above ('unstacked' or 'wide' format). How would you accomplish this (without just copy/pasting)?
You should try to 'unstack' the Google Analytics data in the Data Room (GoolyBib-trafficsources-all-traffic), converting it from long format to wide (each Source/Medium has its's own column. How do you accomplish that?
Data Requests\nHey,\nJust checking if you had the TV data in daily format instead of weekly for the model?\nIf not we're going to have to transpose it.\nBest,
Interpolating and Transposing Data Now we need to look at the TV data in the Data Room folder, labelled 'GoolyBib-TV-Expenditure-2020'. The problem with this data, is that it's weekly data when our marketing mix modeling vendor has requested daily!\nLuckily there's a technique called 'Interpolation' which let's us fill in the gaps to turn weekly into daily data. Each day of the week for the date range should divide the weekly number we have and divide it by 7. We also need the data to be 'transposed', meaning one date per row, rather than per column.
Now try to Transpose and Interpolate the TV data in the Data Room (GoolyBib-TV-Expenditure-2020), turning it from weekly to daily and making sure there's one date per row (rather than horizontally across columns). List the steps you took to do this.
RE: Data Requests\nThanks so much!\nIt looks like everything I need for my model is in here.\nI appreciate you putting it into the right format - the transposing saved me some time!\nLook out for my finished model next week\nBest,
We've got the numbers, but can we trust them? This isn't like the attribution methods you're used to – how do you make decisions and take actions based on the model?
The Next Week
Finished Model\nHi there,\nAs promised, attached are the slides for the presentation.\nTop level summary:\n- The model could explain 75% of the movements in revenue, and was only out by 5% on a daily average across the period\n- TV returned $0.40 cents for every dollar invested, but this doesn't capture long term effects on the brand (likely to be substantial)\n- Affiliates doesn't look to be driving incremental revenue. You should investigate their behavior further\n- Facebook looks to be driving a high return at $4.20 ROAS, but we should validate this with an incrementality test\n- There are improvements to be made with the model, for example the numbers for Facebook are likely inflated\n- According to this model, with $10,000 spent on Facebook and $5,000 on TV per day, we'd average $57k revenue\nPlease let me know if you have any questions – let's hop on a call to discuss.\nBest,
Interpreting the Model Even if this isn't the first Marketing Mix Model you've seen, there's a lot going on here to interpret. First and foremost is the accuracy – our model could explain 75% of the movements in revenue (an R squared of 0.75) and was 5% out on any given day (a mean absolute percentage error of 5%). This is within reasonable levels for decision-making, so it's worth digging into the model further.\nNext we look at the coefficients and see if they make sense. Facebook Ads drove the majority of revenue due to its high return of $4.20... but this seems too high. Maybe in the model it's stealing credit from elsewhere. Affiliates is driving no significant revenue, which may very well be true: as far as you could tell they were mostly targeting existing customers who would have purchased anyway. TV is less good news, as we're losing money at 40 cents on the dollar.\nNow the statistics – if we look at the models spreadsheet we can see a number of additional information above and below the coefficients: Omnibus, Skew, Kurtosis, Durbin-Watson, Jarque-Bera and Cond No. None of these likely mean anything to you, so that's what we'll investigate next. Read the following article for more info.
After Some Light Reading...
After reading that article, you see that the model failed the Omnibus, Jarque-Bera and Cond. No statistical tests. You message the vendor about this to get their explanation.
Hey, thanks for messaging to follow up\nYou're right on the money – the model still has some flaws.\nSpecifically this means that the errors aren't evenly distributed, and the variables may not be independent.\nIt's rare for a model done in a short time on limited data to pass all tests, though in this case we're not in a bad position\nThe variable giving us trouble is Facebook... the numbers have been pretty constant but then rose at the same time as TV.\nThat's the reason one of my recommendations was to pump Facebook up in step changes to get a clearer readout.\nHappy to help you spec this test out!
RE: Finished Model\nThanks Charlie,\nAppreciate the quick turnaround.\nI think we have enough for now, but Facebook def too high. I'm fine with TV numbers: this isn't capturing long term branding impact.\nWe'll discuss and circle back.\nThanks,
Congratulations! You've made it all the way to the end of the simulation. You learned about Marketing Mix Modeling and how it can help you tell the impact of TV.\nThis free course is focused on a Marketing Manager's role in working with a vendor for Marketing Mix Modeling. It's meant to give you a taste of what the deliverables look like and what to expect in terms of process.\nIf you want to keep learning and go more in depth, we'll soon have a library of content to train you or your team in Marketing Mix Modeling and Marketing Attribution, available for paying customers.
" 
