GOOLYBIB OFFICES – 6 months into a marketing mix modeling project

You’ve been building marketing mix models for the past six months, but an uncomfortable question just came up in a meeting. The CEO came to the realization that our MMM assumes marketing performance doesn’t improve! What’s more when we update the model with new date ranges the coefficients are wildly different. Your credibility is in question, and they might decide to abandon MMM entirely (but what else is there?).

I just don’t get it
Is everyone that’s using MMM making this assumption?
Of course marketing performance changes over time! Otherwise we could all just retire early instead of working on optimizing the accounts…
Let’s search for a way to let marketing performance flex up and down
If we can’t do that, I’m not sure I can condone using MMM, it just won’t be something I can get behind. What if one of our investors makes the same ‘discovery’?
This is top priority

One of the big assumptions everyone is making when doing marketing mix modeling, is that marketing performance doesn’t change over time. Most aren’t even aware that’s the assumption they’re making. Of course MMM usually accounts for changing in spend over time with diminishing returns and adstocks, but the baseline coefficients don’t change in most models. If you shorten the date range or update the model, you tend to find that the coefficients vary significantly from time period to time period. This is because now the model doesn’t “know” about those previous time periods and can’t remain consistent with them. For most models making the assumption that marketing performance doesn’t change very much isn’t as bad as it seems. For simple models and small businesses, that assumption largely holds. It’s the red queen effect: we do constantly optimize our accounts, but so do our competitors, so the net result is usually pretty stable performance over time.
However this hidden assumption is a ticking timebomb. It sounds so obviously wrong, that when someone non-technical like a CEO discovers this insight, it can ruin the credibility of marketing mix modeling in general as a method. The solution to this problem is an advanced, but growing technique, called “Bayesian Time-Varying Coefficients” (BTVC): used by companies such as Recast and Uber. One unique benefit of using Bayesian methods, is that you can use priors to allow marketing coefficients to change over time in a stable way. Today’s performance is likely to be like yesterday’s, but if the model has strong evidence today is different, it’ll be allowed to evolve away from yesterday’s results. The end result of this model is that coefficients for each variable change over time, letting marketing performance increase or decrease with seasonality, optimization and competition.

Time-Varying Coefficients

Hey, let's walk through how to use Uber orbit, which is their time series forecasting tool, or we're using it for marketing mix modeling.
So specifically what it's for other it's one of these cases it supports. So we're going to follow along with this tutorial and this is very high quality.
I really enjoyed this guys content Mario feeler from a category of Grandmaster. So really interesting approach that he has here.
I'm just going to kind of walk you through it. I've replicated what he did are using the data. But I've also added a couple of things as well.
So first thing we need to do is installed the libraries. One thing you can do in, in Google CoLab notebooks, and you can make a copy of this is you can do percentage PIP and that kind of lets you run the PIP commands like PIP install or ML to the actual kernel of this computer that this has kind posted on.
So first you saw, I finished pressed a shift and enter here. First you install or an ML and then there's also never grad, which I use for budget optimization.
It's the same one that Facebook Robin uses. And then we're also doing this fix. This is, this might have been fixed by the time you you know, try this tutorial.
But essentially I found that there was some error where if we didn't use this specific version of Mac club lib then it would break.
So you can see here that it's sort of everything you just want to hit restart runtime and click. Yes. So it can use those libraries and you can see here, it says initializing.
So that means area. So now we've got the library it was already forgot the runtime and it should work. Okay, cool.
So first thing we're gonna do after that is important, the different libraries we're using, so we're using orbit or it models KTR, this is the this is the time varying model.
The one that we're most interested in where the coefficients can change over time and you'll see pretty quickly what that means.
So here authorized if you haven't authorized your Google account this is gonna pop up with a login to your Google account prompt, and then you just authenticate and then this is going to let you pull the data from from the spreadsheet.
So what I've done is I've used the same data or that he points out here and just going to click through to his other tutorial where he gets the data from Kaggle.
So I just downloaded the data set and unpacked it.  One thing I did find is that the raw data has some errors in the dates.
So this was like something I ran into. Let me see if I can find the example. Oh, here we go.
Yeah. So see some of the data in this format instead of in this format. So it's important to fix those.
I fix them manually. So if you look at my data set that's all clean. There's none of its issues. So that's the one that I've linked to here, and this is open you know, for you to use as well.
Cool. Okay. So what this code does is when you're authenticated, it looks at this spreadsheet and then pulls in and then shows the dataset as you can see here.
One thing that we need to do, I found is convert the date column to a date time format.  And then this is just telling you, what format is it in right now, which is the short month of percentage be the date and then the year after a common course.
So now let's look at this, you can see that the spend and sales objects are objects, they should be floats.
And the reason why they're like that is they have dollar signs in commerce in them. So this is something else I had to do that you know, it wasn't mentioned the tutorial quite often, you have to do data cleaning.
So I just have, you know, taking those columns and then just apply these functions to remove the the dollar sign and then applied the same function.
But using, you know, the calmer here to replace the calmer and then I converted it into a float and you can see now the D types, two types of flits instead of objects.
Cool. So now that's done I'm going to run this code here, which is from institutory again. What it does is kind of group the data and kind of pivot and staff here.
So now it's in this format, which is you know, the format we're going to use for the actual marketing mix modeling.
And this is where we're setting up the writing with following. So we're kind of taking the different columns that we're using.
We're using sales, the aggressor columns. This is that, you know, X variables. These are the marketing campaigns that you can see with you choosing a split point of 28 days.
So it splits at the end of the data. And that's the holdout group. That's going to be how we predict you know, future data, right?
So that the model hasn't seen, so we're going to hold back the last 28 days.  And then the model is going to try and predict that after training on the rest of the data and if, if you predicted, you know, 28 days, hasn't seen then it should be able to predict the next 28 days for you as well.
Cool. So we have the split point here, which is kind of splitting into the train and the test data sets, and then we have to scale the data.
So this is pretty common for any machine learning or data science. You have to scale the data. So it's all, you know, you don't have like some variables that are like in the thousands of, so rails are in the tens.
What this does is it takes it divides by the the highest absolute value. So, you know, don't worry too much about this.
This is how it works. This is fit transformed thing here. And then and then we have a different scaler for the Y variable.
So all the X variables are the same, and then we have the Y variable, which is different. Okay. So this is actually, sorry, I'm going to just run that.
And chairman pressing just like shift and to run it, but you can also hit play here. And then this is the actual model.
So I run this model and then I'm going to talk about what it's doing. So we're feeding it, the response column as the day to COA what's our date column, the regresses we can actually incorporate this which will hurt the accuracy, but it would make sure that we don't have any negative coefficients, so that might be worth doing, or at least we're testing.
If for example, you expect that your marketing spend is never going to drive negative sales. So that's quite useful. We also have here at the seasonality.
And so this is like how many like what the, what the pattern is. We, we do a seven day pattern here because we have daily day tests.
This is kind of a weekly trend. You could also do. If you had a weekly data, could put 52 here, we'd have to have at least 52 weeks in the training dataset, then we have an estimator and then this is related to the Beijing estimation stuff.
So we're going to fit the data and then you get the prediction. This is runs pretty quickly in my experience and just sort of finished that course.
So once we have estimates, then we can just plot the regression coefficients, and this is the money shot. This is what we, you know, we're here to see.
So what we're showing is reach one of our channels. How did the coefficients change over time? So the blue line in the middle is the coefficient that we'd normally get.
The blue conflate blue shaded area is uncertainty. So you can see as quite, you know, fairly wide range here. And then you can see that it changes over time.
So some coefficients are pretty stable. Other coefficients like change over time, but this one's kind of style low and then got high and then down again.
So you know, this one's had like started with more uncertainty and ended less so you can see some interesting patterns here.
And typically what you're going to see is like, you know, some channels are more stable than others. Of course. So that's that's the really interesting thing about this library as the time Varian coefficients.
It means that you know, when you, you know, like when you do a normal mock team-based model the assumption is actually the coefficient be the same for all time periods, which isn't true.
Like isn't likely to be true. You worked hard on improving your marketing over time, right. And optimizing you shouldn't expect your marketing to perform the same as it did last year.
It might actually perform worse. You know, if because of COVID or something like that. So this is a really great way of dealing with that.
Cool. So so now that that's run, we just need to set up this we did it a full data frame and that's, that's how we're going to work out the contribution.
So that's basically just taking the scaler and kind of getting the values that each basically multiplying the coefficients over, over time by the inputs over time.
And then we're going to get the total contribution of each channel, and we can plot that here. So this is how many sales drove.
Like we have some channels here, like at your one brand here at your one that drove like what a sales, we have some that are negative which is, you know, unlikely, right?
Like these are all marketing channels. So you know, we wouldn't expect them to be negative. If we wanted to rerun this we actually could.
So I'm going to do that maybe at the end just to you see the difference, but for now, let's just see how this model works.
So so we look at ROI and then ROI, you can see that like a lot of them have no ROI, right, because they're negative.
And then somehow a lot of ROI, which is, you know, maybe a sign that the model hasn't gotten it. Correct.
You know, I would look a little bit more into this, if this was something I was presenting to a client, because they're probably not going to believe that, you know, one ad group drove like 400 ROI Airforce, $400 for every dollar invested in some drove zero or negative.
Cool. So the next thing we're going to do is optimize the media budget. So I'm not going to go too much into that because it's covered elsewhere, but this is using this is basically creating an optimization function to to shift the budget around within limits.
And then we're using never ground here to actually kind of do that recommendation. So you just pass it as optimization function and then it should give you here we go.
Yes. So this is how much you should spend in each one of the channels. So this channel, you should spend 3000, this channel 212.
So yeah, I'm not going to go deep into that. This is something that you don't, it's not specific to Uber or wit this is using never grad, which is what Facebook uses to optimize budget and Uber doesn't provide its own, which is why we've covered that, why you covered that in the tutorial.
Cool. So a few of the things you didn't cover in the tutorial, one thing which I think is interesting is seeing the trendline every time.
So because this is kind of a time series prediction library. It has like something which is kind of cool. It shows you know, what the, what the trend was.
So this is the underlying kind of organic trend absent all the marketing spend. And you can see it's actually a pretty stable, like flat over time, but then certainty is getting bigger and bigger.
So that is maybe an indication that we're not capturing everything in our watching, like small we've got you know, different levels of spend, et cetera, but maybe we've caused some maybe we have, like, if we have kind of increasing uncertainty over time that's a patent that we don't want to see.
Maybe, maybe there's something we can prove to them are all like, we're missing a variable. We want to add it.
Cool. So the other one is the actual kind of regression. So this is everything like separate from the trend. So that's an underlying trend and then this is the impact of the marketing marketing spend.
Cool. So the other thing that I wanted to add in here was part of the predictive data. I just think this looks really good.
It's like a great way to see how accurate you think the you know, how accurate you think the model is.
It's just kind of shares the data over time, and then what the prediction is for the future. You can also overlay on here.
I didn't get a chance to, but you could overlay on here what the actual as were. So yeah, that, that, that kind of tells you like what it thinks like if you, if you were actually predicting, if you didn't have the actuals and you were predicting what cells are going to be next month then you could use that you know, to predict that and then you could kind of then try different scenarios and say, Hey, what if we doubled our marketing spend?
You know, where would we get to and how much it certainty is there around that? Cool. So the final thing we want to do, you can see this is still scaled.
So is between 1.4 and zero. So is not a real number, right. It's not the real sales number, so we have to unscaled it.
And we want to get the actual kind of percentage error. So we want to see how accurate this is. So everybody uses <inaudible>, which is like symmetric mean absolute percentage error.
Facebook uses me which is just a normal <inaudible> but the way, you know, we don't have to worry about how that's implemented.
They provide that function, maybe function that we inputted earlier. But what we do need to do is on scale of the predictions.
So we take our scalar Y and we, then we do inverse transform, and we pass in the predictive values and reshape them.
So they're in the right format. Then we'll get back the unscaled prediction and then we just need the actual, so we're going to take the rural media data.
We're going to split it by the split point to get the test data only and just the Response column. So just sales and then values and reshape, same thing again.
And then we just pass those both in Smith. We get 13%. So on average, you know, on a given day this model is incorrect by about 13%.
And I would say below 10% for me is really good. So this model is like pretty good for where it's at and you can make directional decisions based off it.
Okay. Hopefully that's helpful.

What is the main unique feature of Uber Orbit?

What could be the business benefit for having time-varying coefficients in your model?

Why might negative coefficients be a problem?

Forcing coefficients to be positive will do what to accuracy (SMAPE)?

Can we try Uber Orbit?

I’ve built 20+ Marketing Mix Models, and it’s always the same thing that blows the model’s credibility with decision-makers. The assumption that marketing performance doesn’t change! Nobody believes all their hard work was for nothing. Uber’s ‘Orbit’ model solves that problem.

Vexpower Learning Certificate of Completion

Congratulations Annie Jarvis 💪