Finding Numbers for Greek Letters

Building a model is not just about finding the right parameters, but also the hyperparameters used to transform your data outside of the model. Finding the right numbers can be surprisingly tricky.


Hyperparameter Optimization

Modeling isn't just about finding the right coefficients, you have to find the right transformations as well...More


Mike Taylor

Built a 50-person growth agency.
💪 Useful 0
😓 Difficult 0
🎉 Fun 0
😴 Boring 0
🚨 Errors 0
😕 Confusing 0
🤓 Interesting 0
Premium subscription required.
Excel experience recommended.
1. Scenario
You've been looking for an excuse to get your hands dirty optimizing a model manually. You hear on the grapevine that a colleague is struggling with their model. This could be your chance!
Charlotte Cook
at Vexnomics

Yes you heard correctly

We're struggling with the GoolyBib model for Italy

I'm not satisfied with the model as it stands – the error is over 10%!

I'd appreciate your help

This course is a work of fiction. Unless otherwise indicated, all the names, characters, businesses, data, places, events and incidents in this course are either the product of the author's imagination or used in a fictitious manner. Any resemblance to actual persons, living or dead, or actual events is purely coincidental.

2. Brief

Every model has parameters: these are the variables used to make predictions when the model is built. For example your model might say that Facebook Ads drives a cost per purchase of $5. How the model arrives at that conclusion is dependent on Hyperparameters: parameters that control how the model learns the right answer. To take our Facebook Ads example, our cost per purchase might get worse at higher spend levels due to saturation, or there might be a carryover effect where spend today affects performance tomorrow.

These transformations of the data are key to building an accurate model, but it's hard to know ahead of time what values are correct. The manual way of figuring this out is to choose a value for a parameter, for example an adstock level of 20% (20% of your spend from today has an impact tomorrow), and then move that value up or down to see how it impacts model accuracy. If you increase it to 30% and accuracy of the model goes up, you know that's a better fit. The problem is that each parameter affects the behavior of every other parameter in the model, so it can be hard to find the right value.

So there have emerged several strategies for dealing with this task of Hyperparameter optimization. The brute force approach is a grid search: try every possible value until you find the right one. This isn't always possible however, because there may be more potential combinations than there are stars in the universe! Random search lets you set a limit of how long to look for the right values, randomly guessing within a budget. The method we're using today is a manual approach, which can be a more efficient way than random guessing. I’ve even seen the manual approach outperform machine learning or evolutionary algorithms for simple models, because humans are very good at pattern matching.

3. Tutorial

0:00 Alright, let's learn how to manually optimize hyperparameters. So this is a marketing mix model. We have the data here just to spend on Facebook, on radio, on TV.

4. Exercises
5. Certificate

Share This Course