Your boss wants to try the latest experimental tool released by Facebook... but you don't know how to code! Here goes nothing...

Your boss sends you an email. She wants you to check out the new automated marketing mix modeling solution from Facebook.

Facebook Robyn
Hey I saw this Twitter thread on Facebook getting into the MMM space.
Can you check out their solution and see what all the fuss is about?nTY,

Project Robyn is an open-source and automated marketing mix modeling tool built by the Facebook Marketing Science team. It uses modern techniques to reduce human bias by automating the modeling process. For example it uses Ridge Regression to penalize insignificant variables, which means less manual feature selection needed. It offers ground truth calibration, meaning you can feed it the results of your incrementality tests to ensure it arrives at a result consistent with what you already know about attribution. It also employs other Facebook-built open-source libraries, like Nevergrad, an evolutionary optimization algorithm for testing thousands of models automatically, and Prophet, a forecasting tool, to build seasonality and holidays into the model. After running the model Robyn displays a one pager of charts for the model showing the impact of each channel, for example a waterfall breakdown of channel contribution, the response curves for each channel (how they behave at each spend level) and the predicted vs actual results over time.

Facebook Robyn

You don't always need to know how to code to run scripts. Sometimes if you can suffer through a little setup and lots of Googling, you can run code (that you trust) without knowing what it does. Then if you find it solves a problem for you, as you get more technical you can come back to make changes and customize the script to how you want it.

Internet Meme

I just keep Googling stuff and it keeps working

New shiny object syndrome

Learn how developers learn on the job: Googling the problem and reading obscure Stack Overflow comments. What's the worst that could happen?

You don't know how to code, but you won't let that stop you. You start searching for solutions online and reading obscure stack overflow comments...

Robyn is (so far) only available in R, a programming language primarily used by statisticians. Our best bet is to use R-Studio, which is an IDE (Integrated Development Environment) that the Robyn team recommends. We first need to install R on our computer using our local server (CRAN Mirror), then we can install R-Studio and see if it works.

Setup R-Studio

R is an open-source programming language, hosted simulataneously by multiple websites around the world, called CRAN Mirrors. The R community asks that you pick the location nearest to you, so as to distribute bandwidth costs. All of these downloads are identical (hence the name 'mirrors'), so you'll be fine whichever one you pick.

R-Studio is an IDE that lets you run R code, and has lots of features that are useful when writing your own scripts. You can use it on Windows or Mac as a desktop app – it's recommended to download it directly rather than using it via a third party like Anaconda.

Download and install R and then R-Studio. If you get stuck, Google any errors you run into. When you have R-Studio running, type 'version' into the console section and hit enter. What version of R did you download?

We have one more requirement to satisfy before we can run Robyn: the Nevergrad library. This is what handles hyperparameter optimization for us: finding the saturation and lagged effects of each channel by building lots of models and seeing which ones work best. Unfortunately because it's a Python library and we need to run it in R, setting it up can be a bit complicated. First you need to install Reticulate, which lets you run Python via R. Then install Anaconda, and use it to manage the virtual Python environment you'll be running Nevergrad in. Finally you will need to install Nevergrad in that virtual Python environment. If you're on a Windows machine, you may run into configuration errors, and need to complete additional setup steps.

Install Nevergrad

Most libraries have a quickstart or setup guide: it's always worth reading these, because it gives you an idea of how the authors of the package intended it to work. They usually have links to more information and helpful tips, as well as recommended tools and processes for using their library. In our case the Robyn team has put together a quickstart guide, which walks you through how to install Nevergrad. We recommend installing Anaconda and using that option, as that's what worked for us, but go with whatever works for you.

You might run into an issue with SSL, which gives you an error when installing Nevergrad. If you're facing this or any other issues, we recommend following the below blog post.

Remember to install the Robyn library also, as detailed in the quickstart guide!

Were you able to follow along and get everything ready to run Robyn?

I'm impressed!
I had no idea it required that much setup
Well done for perservering

Cobbling together solutions from the internet

We've got our development environment configured: that was half the battle. Now we're ready to run Robyn, so let's go through it line by line.

Your development environment is working, now it's time to run the code. You start going through the Robyn demo script line by line...

First we need to load in the data we plan to use to build our model. This typically includes any factor that might impact sales or conversions, for example: media spend or impressions for different channels, important events including changes to your products or prices, and other external factors like the impact of COVID or your competitor's estimated sales volume. In our case we'll use the demo data Facebook has provided. Add a new blank R script in R-Studio and copy across Facebook's Demo.R script. Run all the lines up to 55, the end of Step 1. You'll see variables appear in the 'Environment' tab top right if you've done this correctly, as well as comments appearing in the 'Console' section.

Step 1: Load Data

What Data and Values have appeared in the Environment panel (top right) after running Step 1?

Here we have a number of steps to set up the model before we run it. First we specify the inputs, for example telling Robyn what variables we plan to use from our data. Then we define our hyperparameters, which are the adstocks (lagged effects) and saturation (diminishing returns) values. The choices we make here will determine both how long the model takes to run and how many iterations it has. This will affect the results of the model, in some cases improving accuracy but it can also hurt the accuracy of the model if you don't know what you're doing. We'll stick with the defaults in this case. Run everything from line 62 to 270, making no changes but reading the code and comments (in green) as you run each section.

Step 2: Model Specification

How long did the comments say it would take to run?

Take a look at the Robyn documentation on variable transformations. Explain how the alpha and gamma parameters are used in the model for diminishing returns.

We're finally here: we're about to actually build the model for the first time! This step is short, as we just need to trigger the `robyn_run` function with the inputs we specified. As we've been told, running this should take around an hour depending on your computer, and when it's done we'll be ready to look at the results. You just need to run the lines 275 to 281 and sit back to wait (one of the perks of being a data scientist: lots of justification to take breaks).

Step 3: Build the Model

Web comic, XKCD

The number one programmer excuse for legitimately slacking off: my code's compiling

How long did Robyn take to run for you? Did you hit any errors?

Now we've run the algorithm for an hour or so, we have our results. It's created 2,000 iterations x 5 trials = 10,000 models for us, with Nevergrad always steering us towards more accurate algorithms. We can see that in the pareto_front.png visualization that should have been saved in a folder labelled with the date and time then the word 'init'. For example 2021-10-06 23:23 init. We can also see Prophet's decomposition of the seasonal trends, holidays and key events in that same folder, a file called 'prophet_decomp.png'. There are a series of other images, including a one pager for each model. You should have about 100 of these models in your folder - these are the models that did equally as well as each other according to Robyn, out of the 10,000 models it tested. We now need to manually pick whichever of these models best fits what we'd expect for our business - for the demo we can just choose one at random. Choose a model number and enter it on line 297. For example `1_148_10`. You can also run line 296 to get a list of all the model numbers, or look in the pareto_hyperparameters.csv file. Once you've selected your model, go ahead and run lines 297 through 302.

Step 4: Select Model

If you think about it, you want to choose a model that makes sense, but also is accurate in predicting data. In Robyn terms accuracy is measured in this case as NRMSE, or Normalized Root Mean Square Error. This the difference between what the model predicted and what happened each day, squared and summed, then you take the square root. You can just think about it like the percent we were wrong on average each day, so 0.05 would mean our model was 5% off what actually happened. Measuring if the model 'makes sense' is harder, and this is usually done by looking at the results and coefficients, and deciding based on what you know about the business. Facebook needs to do this automatically so it calculates the decomp RSSD, or the Residual Sum of Squared Distance of the Decomposition. This translates to how different the share of spend and share of effect are for the media variables. So for example if you spent a lot of money on TikTok ads, and the model says TikTok doesn't drive sales, that model may be rejected.

Can you think of a reason it might not make sense to optimize to decomp.RSSD?

Don't worry about the rest of the Robyn code for now: it's all for choosing your budget allocation based on the results of the model: something we'll cover in the next chapter.

Run Robyn, run

Robyn has run, you've got the results... but how do you interpret them? You contact someone you met at a conference: maybe they can help.

You're struggling with interpreting Robyn's results - one exactly does the model onepager mean? So you ask someone you met at a data science conference for help.

Hey, great to hear from ya!
I know FB is technically a competitor of ours but we just started using Robyn ourselves!
I'd be glad to explain the one pager, let me tell you what you're looking at...
Top left is a waterfall chart of what share of sales came from each variable
Top right is just predicted vs actual
Then you have share of spend vs share of sales left middle
To the right of that are the channel response curves (diminishing returns)
Bottom left you can see the adstocks (lag effects)
Then bottom right you get the errors.
Check the small text top left for the model metrics (RSQ, NMRSE)
Hope that helps!

This document tells you most of what you need to know about a model in a single page. As well as checking the accuracy of your model, you're also looking for what channels or variables are driving the biggest impact on sales, where you may have misallocated budget, and how changes to that budget might behave. The actual vs predicted chart lets you see how closely your model predicted past data. The fitted vs residual is also useful for checking accuracy, as you can see if there are any patterns in the errors (a sign of a misspecified model or missing variables). The waterfall chart is what tells you what the biggest drivers of sales are. The share of spend vs effect chart tells you what channels are overspending versus underspending. The response curves tell you how quickly the channel would become saturated and see lower ROI if you increase spend, while the adstock decay rate tells you how long the impact of a channel lasts after the period in which each dollar is spent.

Model One-pager

In which channel in file 5_170_7 are we most overspending?

In the same model, what channel has the shortest lasting impact?

RE: Robyn Review
Really well done on this!
This really looks interesting: I'd like to run it on some of our client data side by side to see how it differs from our existing models, but you've done enough for now!
Did you enjoy digging through the code? I bet you feel like you learned a lot, but there's plenty more to learn.
You know there's a spot opening up on the Data Science team if you're interested?
They could use someone with your tenacity :-)

Congratulations on running your first Robyn model! If you made it this far you've seen exactly how even experienced programmers do their job. It's a lot of trial and error, and reading the documentation (and obscure stack overflow comments) to get something working. Don't be discouraged if you found this difficult - this is a complicated setup for a complex piece of statistical software, written by smart engineers from one of the richest companies in Silicon Valley - this was no easy task. If you're interested in a career as a data scientist, labelled 'the sexiest job of the 21st century', well you just experienced a day in the life of one!

You made it!

Harvard Business Review

Data Scientist: The Sexiest Job of the 21st Century

The sexiest job of the 21st century

You’re asked by your boss to run Robyn, Facebook’s new open-source MMM tool. Just one problem: you don’t know how to code! You decide to take on the challenge, because solving attribution would be worth it.

Vexpower Learning Certificate of Completion

Congratulations Pollyanna Ward 💪