" Not indicative of future results
Off the back of a successful modeling exercise for website sales, your CEO wants you to model sales on Amazon. But you only have 15 weeks data... is that enough?
GoolyBib Office – Morning
You've completed a successful marketing mix model project, but there's no time for rest: the CEO has already emailed you about the next project.
Amazon Sales Model\nHey,\nYou did a great job on modeling website sales: but we're in the dark on Amazon.\nCan you use the same technique? We've only been live 15 weeks, is that enough data?\nTy,
GoolyBib Office – Afternoon
You've built a marketing mix model for Amazon sales, but there's something nagging: you always were told to have at least 7-10 observations per variable... in this model you only have 5.
Number of Observations
According to Chan & Perry 2017 (Challenges and Opportunities in Media Mix Modeling), a rule of thumb for a minimum number of data points for a stable linear regression is 7-10 per parameter. In our model we have 15 observations (one data point for each week) and 4 variables (parameters) including the constant. This is a common issue with MMM, and it can limit our model's ability to predict accurately.\nIn our case the model looks like it is doing well: the R squared is 0.94 out of 1, and the Normalized Root Mean Square Error (NRMSE) is 8.83%, both of which are good. However it's telling us that our advertising variable isn't statistically significant, neither is category rank, both of which we would assume are important to sales! If we wanted more granularity in the data to see if it fixed these issues, we could potentially go back and pull daily data. However that might not always be available, and can introduce a lot of noise into the model due to day of week trends. You could stop here, but shouldn't you make sure the lack of observations isn't a problem?
It is never crowded along the extra mile
It's important to understand that accuracy metrics like R squared and NRMSE are just estimates. We don't know and we won't ever know what the 'truth' is: all models are just a simplification of reality. It's possible for a model to fit the data well, but be utterly useless when it comes to predicting new data. So it helps to dig a little deeper when it comes to model selection, and remember that doing well on these metrics doesn't necessarily mean you've got the right model. A model that tells you advertising is statistically insignificant, isn't one that's likely to earn you a promotion!
It’s all about location, location, location
A friend comes to you with a bright idea: breaking your data down by region to give your model more data points with greater variance. How do you interpret a model based on geo region?
GoolBib Office – The next morning
You asked a friend for advice on how to deal with a lack of data. After a few hours you get a notification: you just received a text from them...
Have you tried a geo model?\nLess noise than breaking it down by day\nI'd keep it to fairly large regions to start and see what it does to the accuracy
Breaking your data down into different geographical regions can improve the accuracy of your model. There are more likely to be differences day to day between variables at the regional level, which get smoothed out at the national level. So by predicting regional sales instead, the model has more variance to go on. For example if you happened to spend more in the midwest on some days than you did in other regions, the model will be able to pick up that signal to paint a better picture of the impact of your spend. Unlike breaking the weekly data down to daily, you're not likely to introduce any day of week patterns, and keeping the regions relatively large (north, south, midwest, west) you can avoid adding a lot of noise to the data set.
GoolyBib Office – Later that day
You've pulled the data broken down by region, and have rebuilt the model... but how do you interpret it? We're now predicting regional sales not national, so we need to roll the data back up for predictions.
Roll Up Geo to National
The geo model is done, but we can't use it to make predictions until we roll the data up to the national level. Remember our coefficients in this case pertain to the impact of a variable on regional sales, not on national sales, so we need to take the contributions at the regional level and sum them to get the national data we need. Once you fill in the national data at the bottom (starting row 97) the charts should update with the new predictions and errors.
Describe your process for rolling up geo regions to the national level.
What metrics got worse in the new geo model compared to the old national model?
Less accurate, but more correct
Your geo model gives you unexpected results. You start to question what we really mean when we say a model is more or less 'accurate'. A valuable lesson is learned.
GoolBib Office – Later that afternoon
You've built your geo model, but it's not what you expected. Accuracy actually went down? How do we explain this to the CEO?
Ok not to worry, it's not as bad as it looks\nYes the 'accuracy' went down, but that doesn't mean it's a worse model\nTake a look at what the variables are telling you\nAccuracy isn't everything
What metrics got better in the new geo model compared to the old national model?
What two metrics had the biggest improvements in terms of Margin of Error (MoE %)
Can you explain what it means when we see a better margin of error and more statistically significant variables?
R squared isn't the only way to measure a model, most people know that, but even accuracy measures like Normalized Root Mean Square Error aren't a perfect fit. In this case we generated this data before adding some noise, so we know what the ground truth is: the actual coefficient for each variable. If we take a look we can see that the coefficients are pretty close for both the national and regional models, it's just that the regional model is much more sure of each one, as evidenced by the improved Margin of Errors.\nIn reality we don't know what the ground truth is - and will never know! The model is just an estimation of what the truth could be, but there's no way to truly measure ourselves in an objective way. So we use the different tools in our tool box, like NRMSE, R2 and MoE to pick up on how robust the model looks. One source of frustration for practicioners is that when you select your model on any one or more of these metrics, particularly if you automate that selection (for example by testing thousands of models) you're often left with garbage!\nAll models are wrong in some way, so it's important that we choose one that's useful. This is why marketing mix modeling is still a relatively manual process, in an era where many other analytical tasks have been automated. There are lots of tools that can automate parts of the job, but there's no substitute for domain expertise. Marketing mix modeling is both an art and a science, and you need both statistics and a knowledge of the business to be effective.
When you're building a model, don't just blindly look at the R-squared value or other accuracy metrics. Do the coefficients make sense? Has it excluded a variable you think is important? These are the sorts of questions that a good modeler will ask. The trick is getting to a plausible model, without crossing the line into politics. Sometimes marketing campaigns geniunely don't work, but analysts will keep stirring the pot until the model gives them the results they want. Avoid this at all costs!
What if the answers are wrong? Just stir the pile until they start looking right"