Your CEO wants you to send a discount code out to your best customers, and any that you think could be upsold or might be slipping away. How do you do the analysis to categorize your customers?

Hey, we need to do something to boost the numbers for the end of the quarter.
I was thinking we could run a CRM campaign to offer a discount to our best customers.
It also makes sense to target anyone who looks like they could be upsold, and those that bought a lot in the past but not recently.
Can you do some analysis of the data and send the customer IDs to IT?
They can schedule the email for sending

This is a simple and intuitive technique for categorizing your customers based on their past purchase behavior. RFM stands for Recency, Frequency, and Monetary value. To conduct this analysis you calculate how many days since they last purchased from you (recency), how many orders or transactions they've completed total (frequency) and the total amount that customers have spent with you (monetary value). Then you categorize customers based on the quartile they fall into in each category. For example an RFM class of 1-1-1 means the customer is in the top 25% of each category: a top customer. A class of 2-1-1 might indicate that this was one of your best customers, but they haven't purchased recently, because they're only in the 2nd quartile for recency.
This information is invaluable for targeting in CRM campaigns. You can use these labels to find your best customers, identify people who used to spend a lot but might be slipping away, and customers who bought recently but haven't made many repeat purchases. Then you can conduct a VIP, winback or upsell campaign depending on what bucket they fall into.

RFM Analysis

Make sure your dates are properly formatted in Excel or GSheets or you're likely to get your Recency number wrong. The date format YYYY-MM-DD is preferred because it's immune to differences between countries.

Hey, I'm going to show you how to do an RFM analysis. So this is the data that we have a transaction ID. Stock code is the different products as just two products, the replacement cover. And there's also. This smart bit which is further down so less of those and the higher price point than the replacement covers.
We also have a customer ID, which is the main thing we're going to be joining in order to share the recency, the frequency and the monetary value of each customer. Okay. So first let's work out the revenue. This is an easy one unit price, multiply it by the conceit of the. So just going to allow that to dropdown and you can see this work out there and then let's get the recency it's gonna shoot both days ago.
And to get days ago, we just wanted to take today and then tracked the invoice date. Okay. So. This was in 2020 and you know, today it's 2022. So yeah, it makes sense is 458 days ago. And you can see that some of these were like 85 basic year. Which makes sense. Cool. Okay. One thing to watch out is if your date is in a weird format you can sometimes are ready to issue.
All right. So now we need to create a pivot table. I would, sorry. I'm just going to press command a or control a a, and then we're going to go to insert and then pivot table and put that in a new sheet. And what we need to do is pivot on the customer ID. So that's the customer ID and that's going to aggregate all of the transactions.
You can just add the Invoice or a transaction ID here. And then change this to account a yeah, I'm sorry. Yeah. And then you can just make this descending by count. You can see that there's one customer that bought 46 6 of course. So let's, so that is the frequency. And then we need to add.
The monetary value. So that's just the revenue that's pretty straightforward. And you can see that this how much revenue this customer has made. And then we also want to add the recency. So we're going to take days ago.
Okay. We can see the days ago is the sum and we're going to change it to a, it's just the mid. So this will find the smallest days ago that, that customer has there we go. Okay. So now that we have that we want to sort these into quantiles and that means basically splitting into groups. You know, 25% of customers are in group one 25% in group two and so on.
And the reason we did that is so we can get a score for, you know, how how high this customer up is up in the, in the, in the pecking order for these three different values. So let's do recency frequency. And we're just going to put them here as well.
All right. So what we want to do and actually let me just rearrange these. So just say they're in the right oldest or recency of days ago, and then frequency is transactions and then. Revenue. And I'm just going to show you one of these and I'll leave the rest for you as an exercise. So we need to sort into courthouse and that I'm just going to say these are the three both tiles.
So let's do recency. So we just need the core tile. And that takes the data and then the course on numbers. So that if we, you know, press to troll shift down and then I'm just gonna make sure I didn't get the total and that I'm just going to lock that in place for pressing a four or you put dollar signs in front.
And then of course I'll just report. And what this is telling me is the the lowest the lowest kind of recency. Like if you, if you want to be in the the lowest 25% of recency your recency needs to be below 260. If we change this to three a, then that then anything above 393 would, would pay you in the worst 25% of recency.
Cool. So so that ones that actually, if you just put zero head and that's the men, so it's the lowest value. We can. Also bring out the second core tile, they should just be medium basically. And then copy copied across, and then we want them to know, we just need a quick function. It's a couple of nests.
So if the basic go is less than this then Oh, sorry. Get rid of that. Yeah, if there's is less than that value the low school. Then and we want to say it's in queue one. I'm just doing that as a string because we're not going to have the scores up. We're just going to have to, to show you 1, 1, 1 0 1, 2, 1.
Okay. So and if it's not a low in that value, then we need to make another statement. So now we want to say if it's if that is. Lower than this, then it's going to be in cool tile too. And then if it's if that another statement. That is lower than quotes three. You know, then it's going to be three.
And then if not, it's going to be pulled off for, because you know, then it means it's above that number. And then sort of close that statement close the second half statement close the first statement and that should work. Okay. So I'm just going to double click. Actually, let me just, before I double click this, to make sure this is in in absolute reference I say press at four or put dollar signs in front of it.
Cool. So just double click or if that's not working, you can kind of click copy and then paste all the way down.
Cool. So everything is broken into . Now I'll let you do the rest. You know, you do the same thing except in this case you you know, think about it, frequency or recency is a, is a low, is a good thing. With frequency high is a good thing. So in this case, you know, we're selecting seed. And then you can see.
You know, rather than quarter one, we actually puts out three here. So if you want to be able to put 25% of frequency, you have to have bought at least three things. Cool. So you know, this is just the same, but reversed. And then same thing with here. These signs will be reversed when you do frequency and then same for monetary.
So I'll leave you to do that exercise. Once you have. You know, the recency frequency and monetary then so say for example, this person is one-on-one. Then the RFM class would just be that I present that, that. It's going to be class 1, 1, 1, that'd be your best customer. Or you know, if it's you know, someone who has bought recently, I mean, it hasn't been recently, but it used to be your best customer.
I, then that might be like a 2, 1, 1. So that's how you should think about the RFM class. Okay. Good luck.

What RFM class would someone fall into if they had bought recently and spent a lot of money, but haven't made many transactions?

If you send the discount to everyone in the 'Best Customers', 'Slipping Away' and 'Upsell Oppty', segments, what percentage of all customers will get an email?

If an RFM class of 121 is a customer who has bought recently and paid a lot of money, but hasn't made many purchases, how would you classify a customer that bought recently and often, but only for small ticket items? 

Management Consultant and Author

Without big data analytics, companies are blind and deaf, wandering out onto the web like deer on a freeway

RE: Campaign results
Thanks so much for that analysis, the campaign looks like an early success.
We got good purchase volume from those upsell opportunities and won back 15% of those customers that were slipping away.
We even got a few thankyou emails from our top buyers!
Best,

Who deserves a discount?

You’d like to reward your best customers… but who are they? Do we rank the highest spenders? Or those with the most repeat purchases? What if they haven’t bought from us recently? We need an RFM analysis.

Who are our best customers?

Your boss wants to know how much a customer is worth to you. Once you know that you can use it to justify the extra budget you asked for.

I know we’re getting outcompeted in the ad auction

But I’m not comfortable letting you go over our CPA target unless we know how long it’ll take to pay the extra back.

As far as I’m aware most of the revenue comes from first orders

So I’m open to seeing some analysis, but unless it’s more than 10% extra revenue coming from repeat purchases in the first few weeks…

I’m not sure we can revisit the budget question

Understanding customer lifetime value (LTV) is essential for any business that wants to be successful in a competitive advertising environment. Calculating LTV helps you understand how much you can afford to pay for a new customer, and how much money you need to generate in order to make a profit.

The first step in calculating LTV is to determine the average customer lifetime, which is the total amount of time a customer spends buying your products or services. To do this, you will need to look at past sales data and analyze the amount of time it takes customers to make a repeat purchase. This data can provide valuable insights into the typical customer purchase cycle.

Once you have the customer lifetime, you can then calculate the total revenue per customer. This is done by multiplying the average customer lifetime by the average purchase amount and the number of repeat purchases. This is an important figure, as it will give you a better understanding of how much money each customer is worth over the course of their lifetime.

You can then use this figure to calculate the customer lifetime value. To do this, you will need to divide the total revenue per customer by the customer acquisition cost. This will give you the dollar value of each customer.

Once you have the customer lifetime value, you can use it to better understand how much money you can afford to pay for a new customer. You can also compare this figure to your customer acquisition costs to determine whether your marketing strategy is effective. If you find that your customer acquisition costs are too high, you can adjust your marketing strategy accordingly.

By understanding customer lifetime value, you can ensure that you are spending your advertising dollars wisely and optimizing your efforts to bring in more customers. Additionally, it can help you better understand the value of your existing customer base and the impact of any changes you make to your marketing or product offerings.

LTV Calculation

00:01 Hey, let's figure out how to calculate lifetime value. So there's obviously ways to do this in code and you know, recommend that you learn SQL or Python to do that if you are doing something in a backend database.
00:15 But typically that's not where you start, right? Like you figure this out in the spreadsheet, first of all because you know, usually you know, before you have a data science team, you need to kind of figure out like what's a customer worth to you.
00:29 And you know, you can do that pretty simply with just some, you know, raw customer data in a spreadsheet and a pivot table and some calculations.
00:39 So that's where we're gonna work through today. So I mean, the, the benefit of this, by the way, is you don't know.
00:46 You need to know how to code, and you don't need to know any math <laugh>, which is a huge help.
00:50 All right, so what we have here is our raw data, and this is what you need to be able to get out of your database or export from Shopify wherever you have this.
00:59 So what we have here is the order id. This is just a unique ID for every order of a customer id.
01:06 And then again, that's a you know, unique ID for each customer. And then we have the order timestamp and the order value.
01:14 So this is the raw data. This is, you know, we just select all of this. So this is like what we actually got from the database.
01:23 Everything else we've figured out. And I'll show you how to do that. So, first thing we need to do is we needed to sort by timestamp.
01:32 So I just said, you know, sort A to Z and that way it would be, you know, the first timestamp first.
01:39 And I also like to make sure, just cuz it can run into some issues with formatting that you do like year, year, year, year dash month, month dash day, day and then the time.
01:51 So hour, hour and then call, and then minute minute. This is the superior date format, <laugh>, because it works across every country.
02:00 You know, and it's, you know, impossible to get confused about or get it wrong. Also you know, you can sort it, if it's in the format, you can sort it, whether it's text or you know, or, or a date format.
02:12 So it actually works pretty nicely because the largest thing is first, right? The year, then the month and the day, okay?
02:21 So that's the date you need. And what do you need to do with it? So first of all, we want to get the first order index.
02:29 This is basically looking through the, the database and finding what was the first time that customer ordered. And this is something you can do in sql, right?
02:37 If you're doing this in an actual database. But we can do it in Google Sheets as well. And we can't just look actually for, you know, the first customer ID because it doesn't really work on a string unfortunately, but we can find the first index because we've sorted everything by timestamp.
02:54 We can use this function here. So we're saying this is a filter wrapped inside a min formula. So first let's look at the filter formula.
03:03 What does that do? So filter basically you give it a range and then you set a condition and it will, you know, spit out that range.
03:12 So normally it takes up more than one cell actually.  And you can see here, this is the range we're giving it, giving you the index.
03:21 And then we're saying filter just for any time that this range. So column C, which is where the customer idea is equals c2, right?
03:34 And because we've put an absolute reference here, it's like F four. Then, you know, as we scroll down here so like, look at this one it, you know, there's C5 references going down.
03:47 It was, you know, C2 before now C5 that's matching with the row but the, the full kind of array we're looking at doesn't change.
03:57 And that's really important to get right? So what that's gonna give you is essentially a list of indexes where that customer ID is true.
04:06 And we can actually see that if we wanted to. So let's just, you know, copy this customer ID and let's just see filter I'm gonna say filter the range customer d I'm gonna say oh, sorry, we need to actually, we wanted to get the index for this.
04:44 So that's what we want to return. And then we want to only get that index if the orders because this, well, the customer ID equals this, We go, okay, so there we go.
05:11 We spat out the range. So that customer ID appears at index three and index 7,933. Okay, cool. And you might be wondering what the index is, is literal tr just an order, right?
05:23 So we put one for the first thing and then plus one, plus one plus one all the way down to the bottom.
05:29 Cool. Okay. So so that's what the filter normally does. And then what we're doing here is we've got the min function here, right?
05:38 So if you look at min, all that does is it just takes some values and it gives you the smallest one, right?
05:45 So it's saying index three is the smallest one. Cool. You could also do like the latest order if you wanted.
05:51 So that could be max, And then it would tell you there the, the, the latest order for this is 7,933.
06:02 Cool. All right. So that is how that function works, and that's the most important function. So if you understand that that's, you know, you're golden because once you have that, then the rest is like relatively easy.
06:14 So this is just a v lookup. So saying you know, find the first order index, right the thing we just found in this array of the index and the order ID, and then just return the order ID when you find it.
06:31 So all that's gonna do is, you know in say like, you know, this is a good example here. So see this, like this this order ID doesn't match, right?
06:48 Because this is actually looking, it's, it's found at 70 90 which was earlier on, that was in index 85.  So that customer, I had another purchase here, and that's the 70 90, that's what we're pulling back.
07:02 Cool. So VLO cup again is just, you know, we're saying scroll down in this part of the spreadsheet, and then when you find the index of one then return the order ID for that index.
07:18 Cool. So what that's getting us is the first order id, so it just relates to this. Then we can also get the first order timestamp exactly the same thing, except we've just got a bigger lookup and it's just saying just give me when for that order, ID just give me the timestamp, right?
07:36 So that's the same thing essentially. Then then once we have the timestamp, we can work out how many weeks it's been since the first order.
07:45 So if you subtract a date from date, you get the number of days that it is between them. So we're taking this order timestamp, and then we're subtracting the first order timestamp.
07:57 So it should be zero, basically, if they match, which they do here. And then and then we're dividing that by seven days.
08:05 So it's gonna give us how many weeks. We're just gonna round that up so it's easier and it's just, you know, there's no decimal places we can see here that like this this order came in a week after the first purchase, right?
08:20 This one came in three weeks after the first purchase. So that's really the key to getting the lifetime value side of the equation.
08:30 Cuz we can't just add up all the, all the orders per, you know, per customer, that wouldn't really, really be that useful.
08:36 Because, you know, some customers bought yesterday and then some customers bought a last year. So if we just summed up all the orders and we, you know, the revenue per customer, that wouldn't really cut it.
08:48 What we need to do is kind of, if we start all customers at the beginning at zero then how much revenue comes in every week of their lifetime.
08:57 And that's what we're gonna see here. Cool. This last thing here is really just total customer orders. So it's just telling us how many orders that customer had in total.
09:07 That's just so we can kind of see like this customer, it's just a account if, right? So this customer ID is shown have three times in this, in this database.
09:16 Cool. All right, now let's look at the pivot table. So now that we have all that, that was all the prep, now we have the pivot table and we can see if we just put in the rows here, we can see how many weeks since the first order.
09:33 So that's in the here. And we have the sum of the total order value brought into the values and then the count of customer IDs.
09:42 Cool. So lemme explain what that means. It means that for all of the orders that happened in the first week of the first order, so week zero there were this many orders, right?
09:56 So most of the orders came in there. And also most of the customers you know all the customers, right? Had a purchase in week zero, essentially, right?
10:08 Only, so seven, if we had 7,000 customers, only 400 made another purchase in week one. And then we, it kind of drops off from there.
10:14 265 made another purchase in week two, and so on and so forth. And you can see there's kind of a long tail here.
10:20 So that's what this is showing. This is the percentage of revenue. So this is how much revenue is this divided by the total site.
10:29 And what we can see here is that the share of lifetime revenue drops off a cliff, right? We get 85% of the revenue in that first week, and then by the second week it's another 4% and then 2% and so on.
10:44 All right? So now that this is kind of normalized, we have, you know, all of our customers based on starting week then we can do our calculations.
10:52 So first we get the initial order value. That order value is just the order value divided by the number of customers.
11:00 And that is 131. So that's our average order value for week zero. Then we can set a cutoff. So I've set four weeks here.
11:09 So this is gonna look at the, just the first four weeks, and it's gonna tell you that as well as that first 85% of revenue, you're also gonna get this extra piece, right?
11:20 You're gonna get like another 9% in revenue here. And what we're doing here is we're just saying expected incremental. If we know the initial order value, how much incremental revenue do we expect to get based on that cutoff?
11:34 So this is just looking at the cutoff and looking at kind of summing up anything below the cutoff. So you know, here it's like 9%, but it's divided by 85%.
11:47 So so we is, is like 10% more revenue than what we got in that first order, if that makes sense.
11:54 Cool. And then and then, you know, once we know how much more revenue to expect from that user in the first four weeks, then we just work out the lifetime value you know, which is the initial value plus expected incremental value, and we could change the cutoff.
12:07 So we could say eight weeks but in this case doesn't really change that much because, you know, by, you know, like, you know, weeks four to eight, it's only like half a percent extra per, per week.
12:18 So it's not, not a huge amount, right? Did also say, you know, 52 weeks, right? If we wanted to and that gives us a pretty, you know, much bigger incremental.
12:29 But you gotta think about payback period, right? Like if we you know, can we afford to front the cash now and pay 150 bucks for a customer when we're only getting 130 bucks straight away?
12:42 And that extra 20 bucks is gonna come, you know, over a whole year, right? Like, it's not really worth it.
12:48 So I'm just gonna say in this case, let's say first two weeks or, you know, first four weeks be relatively conservative, you know, we can pay 'em back in a month, and that's pretty good.
12:58 And now I know my lifetime value based on that cutoff. And that means when I see that I made $130 in revenue from a purchase, I can really treat that as $144 in revenue.
13:11 And that extra 10% incremental is gonna help me be more competitive when I'm running advertising. It's going to, I, I'm gonna be able to push my bids like 10% higher essentially.
13:23 And and, and, and, and like, you know, eat up a bit more inventory, get a bit more volume versus my competitors.
13:29 So that is why you do this calculation. Obviously this is like a typical e-commerce case where vast majority of the revenue just comes in the first purchase and most people don't buy again.
13:41 But you know, for other businesses, like subscription businesses, it could be completely all the way around like maybe 85% of the life of, of the, of the lifetime revenue comes after the first week, right?
13:54 All right. So that's you know, that's pretty much everything. I think one other thing is to think about, you know, we used weeks here, but you could use days.
14:02 We could also use months if it's a, a subscription service. So just think about the, the date format that you use here is all, all right.
14:10 Hopefully that was helpful.

What is the purpose of calculating lifetime value?

What share of lifetime revenue occurs in the second week (i.e. week 1)

Did you know it costs $50 per click to bid on “printer” keywords on Google? Most manufacturers lose money on the first sale, because they know you’ll buy the ink.

How much is a customer worth?

Your web application has a stable number of users, you've finished the core product and want to add on experimental features to improve customer retention.

Your boss sends you a message: this is probably about adding extra functionality to the react application.

Hey, have you looked at building some simple user recommendations yet?
It would be great if our app could recommend relevant products that have been seen together.
Just like how Amazon has that 'People who viewed this product have also viewed these products' section?

Remember to make a copy of the google colab file so that you can follow along!

All right. So we've been tasked with putting some Amazon style recommendations. So you'll see these on Amazon, for example, like products related to this item, we'll be taking an approach where we'll be partitioning the data by a user ID. And this is quite useful for if you want to generate carousels. So for example, if there's a people who viewed this course or people who viewed this product also viewed these courses or these products Before we get into any of the code.
I think it's really important that we describe this process of step by step. And I've done some pseudo code for us to go through this. So we'll be first importing our  modules and some fake dummy data that you'll have access to. Additionally, you'll be able to extract. All of the courses or the products or whatever your item is from the database, we'll then need to extract how users are viewing those items or courses from the database, wrangling the dates to be in a specific format inside of a Panda's data frame.
Once we then have users against courses or users against products will then generate all of those combinations. Were those Horlicks have been seen in combination in part in combination to the user ID we'll also then create the recommendations and then we'll also provide a graceful fallback option just in case a specific course or a specific product is new.
And it hasn't been in combination with other courses or products. And then just a little bit on the fact that we'll also need to be able to store these recommendations inside of the day. So running these two bits of code where we're download some dummy data, because I'm conscious that some of you might not have a production database, so you can connect to we've taken the Liberty of producing some dummy data for you.
And so what you're going to need to do is in your script, you're going to need, when you've connected and extracted your data from your database, you'll need to have it in a format where you have a PIF and list of your product IDs or your course ID. And you'll also need the format like this. So you can see you've got the CR the product ID and some extra properties inside of each individual pro product as, as a, as a pipe in dictionary.
So this data doc, Jason, if we have a look at this we'll load in some of that course as data that looks like this format, for example. So you can see here, you've got some. Quite a large dictionary structure now, but if we go scroll all the way to the top, you can see this as the course ID in our case and some extra information.
And it's exactly this format of a course or a product ID and then some extra information nested in that. So, so that's that and then once you've extracted the, the product IDs and you know, as a list and you've also got the product IDs in a, in a, in a dictionary for. What you'll then need to do is extract the users in, in relation to how they're accessing your courses or your products.

So if you, for example, this will also be a section you would need. Right. And it's completely bespoke. But if you were using Firebase, then maybe you would be storing how users going through your product and interacting with either courses or products or items. And you can see this as an example of like, for example, in our scenario, we have a real-time database, which just connects to the users and we essentially can get all that data in in using hyphen.
There's some sample data provided for you. And essentially the. That we're going to need to wrangle the data in is going to be like this, where we have this core state of debt, where we have item IDs and a list of items, and we have user IDs and whether they viewed certain items. So if we're having a look at this in a very simplistic way, we have the item ID, which could be your course idea.
It could be your product ID. It could be, you know, any sort of item. And then we have the user. And what's important about this, as you can see, like certain users have viewed multiple license and you can do more complicated recommendations if you add in a rating, a rating column as well. For this example, we wanted to build a recommendation engine that just purely uses the co-occurrence or the combinations of courses based upon a user ID.
So we have this form. And you need to get the data into this format. Right. So imagine you've import, you've imported your course ideas as a list up here. And you've also got your course ideas as a dictionary. Yeah. And then what we want to then do as well is we've got our user data in how, wherever they are you know, viewing your courses, et cetera, on a user level of beta on a user basis.
So if we have a look at this one for. You'll see that, you know, our user user ID is the first key. So if we scroll right up to the top on this one, you'll see. So for example, you've got user ID as the top. If I just grow, I'm just going to scroll up quickly and get to the top of this is quite long as you can imagine.
Cause there's quite a lot of user data that we have here, but essentially what you're looking to do is say for each and every individual user, which courses did they view? So that's how you need to wrangle the debate, the data, right? Whether it's you or a developer, you need to basically say we have some users here like this user, for example, and these are the courses that they've they've specifically viewed.
And so that's kind of how you should be thinking about this is we're godless of the shape of your data. You need. The, the actual like courses and pro or products you need some product product dictionary. If you need to insert any more information and then going onto this step, we also need to have how the users are relating to that.
And basically going in through the data, whatever format that's in, and finally creating a structure like this upon this data stream starts like this. So in our case, what we needed to do is we have a dictionary of users. So we're looping over all of these users. And then we're then looking to see does that specific user have a course dictionary inside that course key.
And if they do what we're then trying to do is create a list of courses from that. And then we're checking to see whether those lists of courses exists in our current set of product IDs. And if we do have any courses that are in our product ID, Or our course IDs, then we're basically saying for every course that we found there, add that as a pair, right?
So we're essentially going loop over all the users for each user find all the courses or in your case, it might be all the products that I use are as viewed. And if there is products that are user is viewed. Then whenever you've got five products loop over there, if there are more than one product loop over those five products as a pair of item ID and user IDs.
And then we ended up with this data structure that looks a bit, a little bit like this, right, where we finally got this data structure of items. So if I ran this code. You'll see this changes. So now we have slightly larger data structure where we've got all of the individual item IDs against user IDs.
And this is what it looks like inside of a data frame. So you can see there's 214 rows by two columns. We have every item ID against the specific user ID. So we now understand. Which courses users as a whole have viewed in terms of, or which products users have viewed. And this is quite useful because then we can apply a combinations function to this.
And we can say this section here is essentially. Group the data by the user IDs. We're essentially partitioning the data by the user ID. And what's quite nice about this is we can then figure out aggregated metrics. So for every user count how many times they have an item idea. For example, when you can see this user has two courses that they viewed but we could take this one step further and do a dot apply function.
And what we've done here is say group the data by the user ID. And then after you've grouped the dates by the user ID, we want to take the item ID and apply a function to it. And this function you'll see when we run that actually produces the data back in a user ID with the combinations of those functions.
So what we're looking at here is whichever courses have been viewed more often. In combinations at a user level. So these two courses here, these two course IDs, these items. Viewed a lot more across all of the users than other than other courses. And therefore we probably would want to include those together in a recommendations algorithm for if you're looking at this course, you would also like to be able to look at these courses.
So that's kind of what this group by with the dot apply combined or value counts is doing underneath the hood and to, to put it in a bit more of a simplistic matter and look. You basically end up with two paired combinations of courses and how often those courses are found together. So we can then write something a little bit to say, well, now we've got all these combinations.
We need to look at all of the courses that we have or all the products that we have and see, okay, if we have these products I want to then set up a new dictionary. And for, in our case, it's people also viewed this course, but it could be people also viewed this product for yourself, looping over those product IDs.
And then what we're doing is we're looping over all those product IDs against the data itself. And with. Getting out the individual combination. Of the, so these individual combinations, so we're getting out this first one and that second one. And we're then saying, if we, if we don't have at least three combinations inside of that individual pro inside that individual product, or course then keep going and do this else.
We just want to break out of this four loop, right. And go onto the next product. So if the course has already got three recommends then we're basically going to be breaking out of that loop elsewhere. Look if the combination is equal to that, then we also look and append the other combination and do the reverse.
If we'd have the combination too, and else we don't match, and there might be some products that you don't find a match for, and that's all we're going to handle later. And you can see here that when we do this we then end up with a slightly. Dictionary. So if we look at this, people also viewed this course, we now have every product and the recommended products, the recommended three products to go with that.
And so you can imagine on a product page, you could do this. So you could tune this barometer to be three or five, and then you could have a carousel of the. Main products that are, are, that are often seen with that product after we've partitioned data by user. So that's all great. And then going on from here, we're going well.
Okay. Well, we've got a hundred products. Maybe we've got 20 new products that just came out yesterday and all of our users haven't seen these new products with other products. So what we could do is take the original data from. Where we had, if you remember going back to this this data frame of the item ID against the user ID.
And what we would then like to do is do a value counts on this. And you can see if by looking in doing a value counts on this, what we can get is we can get an understanding as to what products have been viewed. The most in terms of our users. And you could also use Google analytics for data using Bitcoin for this.
So that's also something you might look to implement, but essentially getting some sort of understanding as to what courses have been viewed the most often. And then we could use these for courses that fell in this else block here. So when we didn't find a match what we would like to do is then loop back over that dictionary.
So we'll extract the popular courses columns and do so then we have this data frame here that looks a little bit like. And then you can see you. So this ends up looking like we bought the course and we've gotten the number of times it was viewed. Cool. So we can then loop over that for every course or product that you have.
If the length of that in the dictionary is not equal to three, then we know that we haven't found any courses and therefore we could look and go, okay, we'll get the popular course ID, get the data loop over that, this new data frame that we've made. And then say, if the course isn't equal to itself, then we're going to append that.
And, and then what we then have here is pause for if there are greater than three results or they're equal to three results, then we break out, which basically means you know, we're just going to be taking the top three most popular courses every time. And appending that into into that individual dictionary.
And so then what you end up with is a final dictionary that you could use called people who also viewed this course, and you could change the name of this, but the essentially the way that you can think about this is every single key in here is the product. And the array is the most valuable courses that have been seen with that course.
So the first one is seen more than the second one. And the second one is seen more than the first. And so you could then save this into your backend. So the way that we do that is we have reference called in our database called user course view recommendations, and we set up. And then on the client's side, you could then reference that.
So you can see where you're using Firebase database, and then we have referencing that we can get those recommendations. So this is kind of an end to end way that you could easily implement recommendations. And I think the other important thing for us to connect. The deployment of this. So I just thought I'd touch on the deployments.
So you'll have access to all their source code. And the only thing that you're going to need to change are the sections where the credit will be unique. So this second. This section and also the, the uploading as well back into your database, there's three sections that your developer or yourself will need to write which accustom, but you've got an example of how that's kind of currently implemented in the system.
And then if we're looking at deployment method, you could, for example, have a chronic. That runs every day or every week. And then basically what they can do is it could be hitting a cloud function that runs this place and code inside of this. And that that function would, would do something like this.
So you could create user course recommendations. And you could also separate the fit, the fullback recommendations. If you wanted on a separate. Or you could include that all in in one single pipe in script and that's just running a cloud function. And so every day we could go in and hit this cloud function at three in the morning and generate fresh recommendations for any products that were added in from the night before.
So hopefully this gives you an indication into how you could start to add additional recommendations functionality into your app.

Recommendation Engine

What data do you need to make 'users also viewed' recommendations?

What does 'count of occurrence' mean for section 5 "5. Generate combinations of all courses/products found together, partitioned by user id:"?

Why is it a good idea to have a fall back for recommendations?

How can we generate useful recommendations?

Learn how to easily create 'People who viewed this product, also viewed these products' recommendations.

How can we create Amazon style recommendations?

 Bank of Yu Offices – Monthly Product Meeting

After a successful launch and satisfying early results for user acquisition, your Head of Growth is turning his attention to retention. They want to figure out what the ‘aha’ moment is for your product, similar to Facebook’s famous insight that if they get a user to add 7 friends in 10 days they’re likely to stick around.

If we’re going to focus on retention, we need our ‘north star’ metric
You know, like how Facebook had “7 friends in 10 days”
The ‘aha moment’ where the user has completed enough actions to remain sticky
If we know what correlates with retention we have something to aim for
If we can focus on moving the needle on that metric, growth should follow
Can you do some analysis to find that ‘aha’ moment for us?

How many actions does it take before our users say ‘aha’? If they don’t use your product enough they might not get it, and if you don’t get them to that point in the first few weeks they might never come back. However if you can optimize your product relentlessly towards a ‘north star’ metric that’s correlated with retention, your product will get stickier, and growth will follow. Facebook’s growth team famously optimized to ‘7 friends in 10 days’. They knew that if they got a user to add at least 7 friends, they would get enough value out of the social network to stick around. The goal wasn’t just correlated with success but it was simple: there were thousands of things the team could work on, but with a north star metric in mind they could focus and prioritize only tasks that moved the needle on the metric. 

In order to choose an effective north star metric, you need to do a retention analysis. The ‘aha’ moment is in the intersection of two competing forces. The more actions someone takes the more likely they are to be retained, but also the more people will have been retained without taking that many actions. You need to find a tradeoff between the two. First you find out the share of people who were retained, that performed this many actions. Second you need to find out how many people are retained without performing this action. Bring those two together and you can find the sweet spot, whether it’s 6 friends, 7 friends, or 8 friends, or whatever the equivalent action in your product.

Retention Analysis

Hey, let's learn how to do aha moments, uh, with user retention data. Um, cool. So we're in a Google CoLab notebook.
It's like a Python notebook. Uh, you can think of it as a Google doc with code. Um, so it runs in the cloud and you don't need to set anything up on your computer, which is helpful.
You can share it like any Google doc. Um, so if you run a, if you just shift and enter, or if you could play on any of the, uh, the cells, and then it's going to run that code and output anything that's needed, um, I've already run this part here.
So this is the libraries, and then this is authentication. So we can pull data from the Google spreadsheet. Um, when you run it, uh, it's gonna pop up and ask you, uh, to, uh, kind of authenticate with your Google account.
So you just click on your email and if you're logged in, then, um, then it will connect it. Um, and that's really just to get this data here from the Google spreadsheet.
Um, and that is the data that we're going to use for this analysis. So this is the format you would need it in, uh, we need the user IDs, um, and then whether they were retained or not.
So in this case, uh, we use the date frame of one week, uh, if they're retained a week after they, uh, installed, uh, then we have a one and if there, if they weren't retained, then we have a zero.
Um, the, uh, this is, uh, one of the events that you could do within the app. It's a FinTech app that we can give loans to friends.
So we're going to evaluate the loans, uh, metric, but you could do this analysis for any of these other events that use this do as well.
Cool. So, um, if we run this, it's going to pull in the data, um, and we can see this just the first five rows of that data, just so we can check the, it pulled in correctly.
And if we run this query, uh, the data stored in this DFS data frame variable, um, and then we're just going to group that by how many people were retained.
Um, and then we can get the count of the user IDs. I, so this is essentially what we're going to do, but on a larger scale, um, but you can see here, they have 969 people that retained and then 2,726, that didn't cool.
So, uh, we're using this loans action. And if we run this here, uh, you can see it calculates a table with a zero for people who didn't retain one for people who did retain, um, and then how many loans they gave out.
So what we're looking for is how many loans does someone need to give out to make them very highly correlated with retaining within the app?
So we're trying to find, uh, something similar to Facebook's goal. Uh, they use for their growth team, which was, uh, they figured out if you can get a user to add seven friends in the first 10 days, they're very likely to retain, and we have a trade off to make as so first of all, um, you know, we wanted to find a users who are very likely to retain.
So, um, uh, so people, You know, people who did this action, whether it's say like, you know, five lines, uh, people that did this action were very likely to retain.
Uh, but also, um, that, uh, they, um, that there were a lot of people who did that action. So if loads of people were retained without getting that far without getting to full lens, it wouldn't be a very good measure.
You can see here that actually, um, you know, we only had 75 people who made full lens, so that might not be a good action.
So it, can it go, um, you know, zero loans wouldn't be a good action either because loads of people make zero loans and they're not likely to retain if they do.
Um, so, uh, in order to figure this out, we want to use data visualization. Um, so we're using a map plot, lib and seaborne, uh, to visualization libraries.
Um, and essentially what this is doing is just kind of setting the color codes and then, um, plotting, uh, two bar charts, one on top of each other.
So we can see here, um, that the retention rate, uh, which is, uh, what percentage of people who got to that level, um, retained, uh, is very, very high for people who made like four loans or above.
Um, it's in the middle for, uh, people made two or three lines. So it's like a 40% pretension, right? Um, and then for people who made zero one loans, it's much lower, it's less than 20%.
Um, so that kind of tells us what part of the story that the more loans you make, the more likely you are to retain, which is what we'd expect.
Um, and we can see a couple of reflection points here. It was, it was already telling us something. Uh, the next thing we want to look at though is just the raw numbers.
Uh, and that shows is another part of the story, which is that very few people make it to fall into or above.
Um, uh, so, uh, it wouldn't be an effective target, uh, to go that far, uh, in terms of our goal, because we would just wouldn't have enough data to optimize.
Um, it wouldn't be able to measure that much of an uplift on how many people got to four, uh, plus loans, uh, because, um, you know, very few people do so in the first week.
Cool. So, uh, let's, uh, run this, um, and, uh, what this is doing is working out a few more pieces of information that we're going to use in the next visualization, uh, specifically the share of retained as, so this is how many people who got to that, uh, level and retained, like what share of, of that, uh, um, what share of those people did they make up of all the people who retained meaning, um, you know, did most people who retain, uh, get to that level?
Uh, so that was one piece of what we're trying to look for. The other piece is, uh, how many people retained without that action.
So this is doing a cumulative sum of the retention metric. Um, and you can see here are retained without the action.
Um, so it's saying, um, you know, if we get to two loans, um, then, uh, yes, we had 283 people retain when they go to two lens, I, but 405 people retained without even making it to, to learn.
So they made zero one loan and you can see that 2, 4, 6 plus 1, 5 9 is four or five. Um, the reason we made this negative, I did, uh, you know, multiplied by minus one is just, sorry, could show it nicer in the visualization.
So it makes it on the left-hand side of the axes here. Um, so this visualizes, the problem that we're talking about perfectly.
So as we go down the stack, more people are likely to retain, uh, but, um, as we get down the stack, uh, most people who retained, uh, you know, had, had already been retained at that point.
Um, you know, uh, like if you look at number four, the pink bar here retained without the action is way, way bigger than even the people who made it to this level.
Um, uh, because, uh, you know, most of the people were were before, uh, for loans, they didn't make as many as full loans.
Cool. So, um, now how do we see this in percentages? Because that's gonna make it much easier to decide. Um, so we're just going to run this, but it's works.
Yeah, here we go. So this is, uh, you know, what we've been leading up to this is the money shot.
Um, and, uh, this is the visualization that you'd run around. You could skip a lot of this stuff. Um, you know, if, if you're just running this analysis yourself, um, you can actually, uh, I think here, just go run all.
Um, and then, uh, we'll run everything all the way down to this, if you want. Um, as you don't have to like worry too much about the previous stuff, uh, this is the most important thing.
So this is sharing everything really on, uh, that we are sharing here, but in percentage terms, so over 100% of people, um, you know, uh, the blue is tow people who made it to this level, um, the, uh, dark blue share of Chandra with purple.
Um, that's saying that say 20% of people who made it to one loan, uh, were retained. Um, and then the pink bar is, uh, people who, uh, the share of, um, of, uh, people who retained without taking this action.
So, um, for one loan, the pink bar is pretty low, uh, because, um, uh, you know, not many people who did zero loans, which is above this, uh, were retained, um, as, so that pink bar is pretty low, but then the pink bar girl is a lot just like it did here.
Um, the pink bar grows a lot to then being pretty much everyone by the time we get down to nine months.
Um, so what you're looking at here just to keep it really simple is you want, uh, you know, a huge share of retained.
In this case, we have a big purple buyer here around two loans, uh, relative to the pink bar. So we want like a bigger purple bar.
Then we have a pink bar, um, and we also want a bigger palpable, and we have the blue bar. So, uh, essentially like the bigger, the purple bar and the floor of the pink bar, uh, the, uh, better That is as a target.
And for me, it's pretty clear, like we could choose one loan has a pretty big purple bar and a pretty small pink bar.
Uh, but, uh, this learns, uh, you know, two learns has a much bigger pebble bar and not that much bigger of a peak bar.
So umbrella to temps, this is a better goal. Uh, we wouldn't choose learn three because even though it has a bigger purple Bob than learns one, um, the pink bar is much bigger as well.
By the time you get to three loans, most people have already been retained, uh, without getting to three loans. So this is how you do this analysis.
Um, you can go and run all of this stuff, uh, with another, uh, metric if you want. Um, so you just change it here.
Um, you know, if you want it to be a balance, uh, that you can press shift and search attend to oh yeah.
A lot of, a lot of you balances. Uh, so you could do, you know, like, it just depends on what data you have.
Um, you can, uh, run it with, um, you know, as many as you want, uh, in the, in this case, uh, view balance is probably a bad run to do because, uh, there's lots of these actions.
Um, so like, uh, comments, and maybe that would be a better visualization. All right, here we go. Yeah. So that looks better.
What's more interesting. Um, and you can see, uh, there's people who made like many more comments, but it's a similar effect, uh, to what we had, um, you know, with the other analysis and just to show you, you can click or journal, hopefully that will work, and it just runs everything in order.
Um, and then you just go down to the end. So, um, here you can see as a really big purple bar and a very small pink bar.
Um, so, uh, here, it might be like comment, anyone who makes one comment is very likely to be retained. Cool.
Okay. So, uh, hopefully that's useful. Um, it's, uh, can be a confusing once you try to understand, like, what the difference is between like share of retained and share retained without the action.
Uh, but, uh, just go through it again, if you don't understand and maybe play around with some of the, uh, data to, to, to see what we're talking about.


Why is it not sufficient to just look for the action that had the highest retention rate?

What are you looking for in terms of choosing the number of actions that would make a good north star metric when looking at the final chart?

Run the analysis but for “searches” instead of “loans”. Why wouldn’t this be an effective action to model retention?

What's our 'aha' moment?

Facebook’s growth team famously targeted ‘7 friends in 10 days’, as their north star metric on the path to 1 billion users. If they could get a new user to add enough friends they knew they’d be hooked, so everything they did was in aid of this goal. You can actually calculate the north star metric for your product, and it’s easier than you think.

The key feature of the FinTech app where you work is the ability for users to make loans to friends – it's also how you make money. If they stop doing it over time that's a problem!

Thanks again for staying late
I have a sensitive project we need to work on
Despite all the good work we've done to bring in new users, I'm worried they're not sticking around
After they make their first loans, are they making more?
That's the important thing for us to crack
But of course I don't want to alarm anyone that we're looking into it
I'd appreciate if you could run a cohort analysis for me
Share the results directly with me, no one else

Venture capitalist 

One of our firm's favorite measurements is the cohort analysis

A cohort analysis is a form of longitudinal study used in medicine, social science and marketing. A cohort is a group of users who share something in common. You could group users by their sign-up date, first purchase month, or acquisition channel. By tracking these groups over time, you can spot trends, and understand repeat customer behavior. 
This type of analysis is essential for discovering retention trends because looking at aggregate metrics you can't tell if users are coming back. For example you could have a growing total user count, but that might be bolstered by the number of new users joining. When you group users into cohorts based on the month they signed up, you can more easily notice that a large percentage of them are dropping off.
Cohort analysis requires a cohort period, be it day, week or month, which you use as the X axis when visualizing it. If you're using monthly cohorts, a cohort period of 2 would refer to the second month they were active, 3 would be the 3rd month and so on. This data can either be visualized as a line chart, with one line per cohort and the cohort period on the x axis, or as a heatmap. Heatmaps shade each box representing a cohort and cohort period by the percentage of users who did the relevant action and were retained. Dark or light spots can indicate peaks or dips in retention activity, which makes it easy to scan and highlight anomalies.

Cohort Analysis

When conducting a cohort analysis it's important to choose a behavior to analyze that is supposed to be repeated regularly. For example if you're looking at daily cohorts, is the action something you expect users to do daily?

The following code is a Google Colab notebook which contains the script necessary to run a Cohort Analysis. Make a copy then review the code and run each cell to see what it does. There is an accompanying video underneath the link to the notebook that walks you through it. You will be expected to answer questions about the results of the analysis afterwards.

I ain't going to walk you through how this whole works. So the first section here is just importing all the libraries and you just want to hit shift, enter where you can hit the play button hit, and it's going to connect you to Google CoLab to the backend. And then I should say connected and then finish running and you shouldn't have any issues there.
Then you want to authenticate. So. Let's see if it asks me. Yeah. Okay. So you're going to click through and then choose one of your emails and then just ask you to click allow, and then. Some token to paste in, and then once you enter that, then you'll be authenticated and that's going to let you pull data directly from Google sheets.
So we run this next cell. I mean, you have, it's your shift and to then that's going to pull in the data from this spreadsheet here which is all of the loan data for you know, for this. When, when the loan was originated then what the ID was the user add that loan theme out. And then the amount they learned as well as the settlement.
Which we're not gonna use. And in order to create a cohort analysis, we need to do a few things. First thing we need to do is create a loan period because we're going to look at the month and year that they learned out the money assets, 2021 here, as it's basically coming off this 11. And that that's just using strip time, which is built into pandas.
Then we're going to get the user ID and kind of make it a use that to kind of find the earliest loan period. So that's what this group buys is doing. Our first reset, the user ID is to index and then we grouped by user ID, which is level zero. And once you set it as the index, and then we find the minimum loan period for that user and then we can just you know, reset the index and, and then we have So that gives us the cohort group.
I said, you can see here this user it's all the same user 47. They have a learn period, 20, 21 and a cohort group, 20 20 0 1. And then actually, if we just kind of run this, oh, actually we would just pull up the data. If you've got to run these in term then you'll end up getting errors. I like, I just got.
Cool. So let's follow that. She's a down, so 47. Okay. In the second month then you know, they still have the cohort group of 20 21. So that's doing exactly what we intended. Okay. The other thing we need to do, this is a common problem. We just need to replace the dollar sign in the loan amount because otherwise it's not going to aggregate it.
Cause it's, I thinks it's a string should be a number. And we're going to make that a flute that you can see here that now the D type is float. So you just do that. And then this part is where it gets a bit more interesting. So this is again using a group by, but this time we're grouping by cohort group and London.
I said that's this here. And then and then we're aggregating the data as we're aggregating user ID, loan ID and loan amount. And then we're just renaming them here to total users, total ones. So what this gives you is, you know, for this cohort what, you know, dates they were present in, and then how many total users they have of those dates, as well as how many loans those users made in those dates and the amount of those.
Then this function here basically just takes the loan cohort period. And it gives you a number. So this is, you know, cohort period 1, 2, 3, 4, 5, as this is. So we don't have to worry about the dates so we can just show you know all of the users you know, that their first cohort, the second cohort, the third cohort cohort.
So that. Connect you know, different cohorts together at to see how they trend over time. And you'll, that will be clear in the second. Then there's just a few more things to do before we can see the final the final result. We just need to reset the index and then set it as a, the cohort group, a group of COVID.
And then we create this group hood cohort group size. So that's a group by these two things. And what that does is kind of give you this, which you use down here to divide in order to get percentages. So don't worry too much if you don't understand that. But basically what it's doing is giving the total number of users for each cohort group.
And then you can basically divide. That percentage in order to get this. Okay. So now this is unstacking. So this is kind of like the opposite of pivoting and basically then dividing in order to get this here. So let me just run these parts that we haven't run yet. So we did this.
Nope. Okay. Let's go back here. Yeah. And then. Oh, we've already done this as middle of the science in there anymore. So don't worry about that
or that amount. And then this is the cohort period. So you can see here 1, 2, 3, 4, 5 and then this is creating that group and now this should run. Okay, cool. So what this is doing is now giving us a percentages of. This whole period. So you can see coho period one you know, what was the cohort group?
So you can basically see you know, this type of visualization. Which is you know, by cohort group what percentage of users were still around after like one month, two months, three months, et cetera. So in the first month it's always a hundred percent and then you can see it. You know, the cohort that started in February you know, less of them were around than the cohort that started in January.
So that's indication that the, you know, whatever we did in February or there's uses required in February one, as good as. You know, the ones from January and to see all of this together you can do a heat map and the heat map basically shows all of this stuff. But in percentages you know, in a, kind of an easy to read format so we can see it's always a hundred percent in the first cohort period.
But then everyone in January we have 36% of people who, who made another loan in February 45% in March. So it can actually go up as well as down, you can see here, this kind of a normally where you know, people are starting in January after 13 months, suddenly 50% came back and originated a loan.
So that's interesting to look at and broadly speaking, you can kind of use the heat. These colors in order to see what's important. So you can see that in every, you know, when something is dark, that's bad you know, we can see a lot of the cohorts you know down to like five, 10, 8% you know, after just a few months so that's kind of troubling.
But then, you know, some cohorts we've got pretty strong retention here and the reason it goes diagonal is because. You know, for people who joined in February 20, 21, we only have two months worth of data. So so that's that's why you look at a cohort analysis in this way, and that's why it's always kind of a diagonal.

What makes a heatmap ideal for visualizing cohort analysis?

What would be a good metric to use in a weekly cohort analysis?

Google how to average the columns of a pandas dataframe. How would you do that to get the mean retention per cohort period?

What was the average retention of users in the 6th cohort period?

Yikes, ok that's not as promising as I hoped
Looks like we need to see what we did right in January and try and replicate it
Thanks for doing this, send me the files and I'll present to the CEO
I'll be sure to give you all the credit!

Is our app sticky?

With hundreds of users signing up, your product looks successful. But questions are being asked about “stickiness”. If too many users eventually leave, soon you’ll have none left.

Are we retaining users?

EXECUTIVE SUITE – STRATEGY DISCUSSION POST-LUNCH

You're in a strategy discussion and need to learn how to effectively evaluate your generative AI based applications more effectively.

Alright team, we've got a lot on our plate today. 
Our boss wants us to dive into the world of evaluating generative AI. 
We're going to learn about different techniques for evaluating customer valuation metrics for AI models, including programmatic rule-based evaluation, LLM based evaluation, and human-based evaluation. 
We'll also learn how to calculate accuracy for each method and get some tips for optimizing the evaluation process. 
So grab your thinking caps and let's get started!

Generative AI has revolutionized various industries by enabling the creation of realistic and customized content. However, evaluating the performance and quality of generative AI models can be challenging. In this course, you’ll explore different evaluation techniques and their trade-offs to help you build effective customer valuation metrics for generative AI models.

The first evaluation technique we will discuss is programmatic rule-based evaluation. This approach involves creating custom rules to determine the quality of the generated output. For example, in the context of generating email content, you can evaluate the length of the email or check if it contains specific keywords. The advantage of this technique is its quick feedback loop and cost-effectiveness. However, it may not be suitable for tasks that require nuanced evaluation.

The second technique is LLM (Language Model)-based evaluation. Here, instead of relying on custom rules, another LLM is used to evaluate the generated output. This approach is useful for tasks that are too nuanced for rule-based evaluation, such as evaluating the helpfulness or readability of text. Multiple outputs are generated by the LLM, and another LLM is used to rate each output. While this technique provides more accurate evaluation, it is slower and more costly due to the need for human intervention.

The third technique is human-based evaluation, where human evaluators rate the generated output. This approach is ideal for high-stakes tasks where false positives or false negatives can have significant consequences. However, human-based evaluation is time-consuming, costly, and requires a labeling step. It provides the highest level of accuracy but may not be feasible for large-scale evaluations.

To illustrate these evaluation techniques, we provide code examples using different scenarios. In the programmatic rule-based evaluation example, we demonstrate how to set up evaluation metrics using Python libraries like pandas, numpy, and TQDM. We define custom rules, such as the minimum length of a social media blog post or the presence of specific keywords, and evaluate the generated output accordingly.

In the LLM-based evaluation example, we show how to use one LLM to generate multiple outputs and another LLM to rate each output. We discuss the importance of ground truth labels and how synthetic data generated by a powerful model can be used for evaluation.

In the human-based evaluation example, we generate images using OpenAI's GPT-3.5-Turbo and ask human evaluators to rate the quality of the images. We demonstrate an interactive approval system that allows evaluators to provide feedback on each output.

Evaluating Generative AI

What are the three different types of methodologies mentioned for evaluating LLM responses?

What are some advantages of programmatic rule-based evaluation?

Which evaluation strategy is best suited for tasks that have high customer risk and where false positives or false negatives can cause consumer damage?

Which of the following is NOT mentioned as a way to optimize LLM outputs?

Discover cutting-edge techniques for elevating the accuracy and effectiveness of Large Language Models and Image Generation Models. Our guide delves into innovative evaluation metrics, providing insights to enhance model reliability and drive impactful results.

Maximizing LLM & Image Model Performance: Advanced Evaluation Strategies	

In the upsert office, Sally asks everyone to gather round.

I've seen some AI-generated content that's really convincing, but it's important that we can't be making false claims in our content
AI can be incredibly powerful, but it's also prone to making things up. We need to be able to distinguish fact from fiction. 

Claim Detection: Unveiling the Truth with AI

In the era of AI-generated content, one common issue that arises is claim detection. AI has the ability to create content, but it also has the tendency to fabricate claims, making up quotes, statistics, and even entire events that never actually occurred. This weak point of AI, however, can be effectively addressed using AI itself.

Imagine this scenario: you have an article based on a real story from a reputable source, like The New York Times. You decide to put it to the test by asking an AI model to rewrite the story while adding a few fake elements. These fake elements include stating that a court has voided Elon Musk's pay package, attributing a fabricated quote to a lawyer, and making up a statistic about Musk's ownership percentage in Tesla. Now, the challenge is to detect these false claims.

To tackle this task, a diligent copy editor utilizes AI to detect any claims that need to be fact-checked. The editor uses a prompt that specifies the types of claims to look for, such as statistics, quotes, anecdotes, and instructions. By employing AI, the editor aims to quickly and inexpensively identify any claims that may need verification.

The AI model is then run with a low temperature setting to minimize the likelihood of it making up false claims. The results are promising. The model successfully identifies both the true claims and the fake ones that were intentionally inserted. This demonstrates the potential of AI in detecting claims and investigating their veracity.

To further enhance the fact-checking process, the editor employs a tool called TabT, which provides web search capabilities to AI agents. This tool enables the AI model to validate claims by performing a web search and presenting the results, including links, content, and relevance scores. This information proves invaluable in determining the accuracy of the claims.

To streamline the process even further, the editor utilizes the Q&A search feature within TabT. This feature allows the AI model to directly ask a question based on the claim and context. The model then receives an answer, eliminating the need for additional calls or human intervention. In this case, the claim that the court voided Elon Musk's pay package is found to be invalid, as recent shareholder voting reaffirmed his compensation package.

The power of AI in fact-checking lies in its ability to swiftly access up-to-date information. AI models can easily verify claims by cross-referencing multiple sources, eliminating the need for labor-intensive human fact-checking

Claim Detection

  Hey, you can talk about claim detection. Because this is a common issue in creating content with AI. It makes things up hallucinates and. It comes up with quotes that didn't actually happen. It makes up statistics. They're out of date. It's typically something that is a weak point of AI, but the interesting thing is that you can actually use AI to solve AI's own problems. 

Everything just becomes AI. What we have here is some article text. I made this it's an article based on a real article here in the New York times. I competent paste it into Chach empty, and I asked it to rewrite the story, but add a few fake things in here. 

The fake things. That it's added that it said that the court has not. Yeah. The court has voided his pay package. Elon Musk's pay package. And in reality, they didn't. We also added in this. Fake quote from Greg Varel home. This quote from the lawyer of the disenchanted shareholders, it's fabricated. 

And then we made up a statistic as well. So I'm just saying that we'd. He would own 20% of Tesla, an actual figure that wasn't quoted. Cool. Those are the fake things. We'll see if it finds those. It's a new of times their journalism is pretty good. You would expect all the other things to be true, but that's not true of all journalism either AI generated or human. 

So it's useful for us to be able to detect what claims are being made and then investigate. After again, using AI. Just gonna run that. And I have a basic prompt that I put together here as a diligent copy editor to detect any claims that need to be fractured. And I've given some specifics, a claim could be a statistic, a quote, an anecdote and instruction. So if it tells you this is how this function works in Excel, right? That can actually be a problem. 

Like it might make that up. I've seen that happen in the past. Or say any information that stated as fact which may be out of date incorrect or imagined, I'm asking it to attend Jason then unmanned, unmodified exit from the text. The claim, the type of claim being detected and some reasoning why the claim might need very caged verification. 

So a bit of chain of thought going on here, I'm using  Yeah, it makes mistakes. But it's very fast, very cheap. So I prefer for this type of task and then I'm just passing the article, text tip, and then the temperature I'm putting a 0.2 because I don't want it to make up the X ups. I've had that as a problem in the past, but no, should be good here. 

If we keep the temperature low. And then if we run that.  Can you see what accepts we get back?  

Okay. So here's an example of next up. For months, many tests. Investors are worried about how engaged he would be, blah, blah, blah. That's an And then reasonably the claim judge Deller of what it must pay package needs. Verification. We, that was one of our fake ones, right? It says, especially after a judge in Delaware voided his pay package. 

So it has correctly found that, it also found the statistics. She has a with 48 billion. So we need to look up the latest value of the shares. Yeah. You can see that. Picked up the percentage of Tesla. Here, which is pretty good as well. So I picked up a lot of the claims, both true ones and also the fake ones that we put in him. All right. 

We need to get that into actual Jason. So in order to do that we just basically remove this like beginning pot from.  Yeah. I see it says  triple tick, Jason. Get rid of that and get rid of this triple takes at the end. And then we have Python object that we can work with now. So that's the first claim. 

I'm using  to look up these research claims tabularly is an AI. Tool basically to social web, it gives us a web search. Capabilities to your AI agents. There are other options here, like you could use perplexities API has searched. And Google's AI has such as well. And then you could also use serve API, I think is another one that people use to this. Some of them have free credits, some of them don't but it's heavily I've I've been recommended by a few people. 

It has a thousand requests for free per month. It's worth setting it up and then you don't need a credit card or anything. Cool. His tablet client, you just import that it's going to run this. It's going to ask me for my API key. If I go into Temperley here, I'm just gonna regenerate this.  I'm going to copy it. 

You don't want to show anyone your API key. And pay some. And to. And then hopefully that run. Here we go. So here's the basic search. I've just asked it to validate his claim a formatted it slightly. So I just put the claim. And then the reasoning. Into a string. And then I put that into his, just says, validate this claim. You can see this is the career that I gave it a and you can see, this is the one about the pay package being voided. 

So that's the question I asked. And then these are the results. So it's done basically a Google search and you can see the content of those searches. So it has the links. So if you want to be able to provide citations and things, this is really great. It was really helpful. And you got the content here as well as the score in terms of how relevant they are.  Great. 

We have the context that's really useful, but I'm going to be even lazier here because I don't want to have to make another call and say, okay, based on this context, is this going? Correct? Yeah, maybe it depends on what application you're designing. But for me, I just want to check. And get an answer. 

So I'm going to use the Q and a search. Which is basically just another call in a underneath. It will do this such and then it will give you back the answer. So it says the claim that the judge in Delaware avoided on AMS pay package is not valid. Test to shareholders recently voted to reaffirm a $56 billion compensation package. 

So this actually just happened a couple of days ago. So it's really impressive that you were able to pull out this up-to-date information. And one of the reasons why AI makes fake claims is because they don't know what the up-to-date information is. Really. Powerful to be able to go and fact check. Your sources again with AI, right? 

So you don't have to have a human do this fact checking. When you want to run this across lots of different ones, you could do a four loop. Here I'm just looping through every single one, and then I'll have all my claims checked. You could also. Structure it in a different way so that you would get maybe. A, a, yes, no answer. 

For example, that'd be really helpful because then you could display Invalid claim in your user interface instead. If you do run into an issue. There's also error handling as well. So I've noticed that sometimes it's have really, it gets, I get this like bad request for URL sometimes. 

Or you might have rate limiting. Yes, sir. Maybe you use a multiple solutions and see which ones work best for you. But ultimately what you get here is you have the X up, you have the type of claim, so you can see what types of claims are being made. And then you would see the context. So here we go. 

There's no verified information. To support the claim that the judge voided the pay packet. Here, it's claims that she has a, with 48 billion with has his cousin. It's going to be verified.  It doesn't specify the exact value of the shares. What you could do now, if you were building an AI application, but this. Is. You can have another call to LLM for each one of these. And just say, okay, based on the context, do we need to alert the user? To check this claim, so maybe in user interface, you have some visual. Highlighting that you say, Hey these are the claims you would need to look at and rewrite potentially, or make them up-to-date. 

And you could also use this as a trigger. So you could say, send this back to  to rewrite this, except if there is. Some claim that is, that we have the context to fix. Yeah, there's lots you can do with this. This is a more of a mental model for you to think about using, for lots of different purposes. 

But in particular, I used this. for SEO generated content, just to make sure that it is accurate and it's not. Putting more fake information out there in the internet because that's one of the biggest bug beds people have with AI and hopefully you can help avoid it with this. 

What is the common issue in creating content with AI mentioned in the tutorial?

How can AI be used to solve its own problems?

How can AI be used to solve its own problems in content creation?

What is the purpose of using the TableClient in the tutorial?

How can AI help in fact-checking and verifying claims in content creation?

One of the biggest problems with AI is their tendency to make things up, dubbed hallucination, leading to inaccuracies in content marketing.

What if my AI makes something up?

Alright, team! Gather 'round in the breakroom for a lunchtime exchange. We've got an exciting tutorial today on a concept called role prompting. It's all about instructing language models, like our very own GoolyBib AI, to adopt the perspective of a specific celebrity or job role to generate responses in a desired style. 

Imagine being able to channel the genius of Elon Musk or the creativity of Steve Jobs in our AI's output. That's what role prompting can do! We'll learn how to use this technique to influence our AI's responses and make them more aligned with the style we want.

But hold on, that's not all. We'll also dive into evaluating the model's responses using an evaluation prompt. This helps us determine if the AI's output matches the desired characteristics of the chosen role. It's like putting our AI's responses through a quality check!

So grab your lunch, get comfy, and get ready to act like a genius with role prompting.

I can't wait to see how role prompting can take our AI to the next level. It's like tapping into the minds of the greatest thinkers and innovators. \n
This technique will give us the power to shape our AI's responses and make them truly genius-like. \n
By using evaluation prompts, we'll ensure that our AI's output matches the style and characteristics we're aiming for. It's like having a quality control system for our AI's brilliance. \n
I'm excited to see how this tutorial will help us unlock the full potential of our language model. Let's act like geniuses and revolutionize the way we interact with our AI!

Role Prompting: Unleashing the Power of Language Models

Language models have come a long way in recent years, thanks to advancements in artificial intelligence. These models, such as OpenAI's GPT-3, have the ability to generate human-like text based on given prompts. However, sometimes we want more than just generic responses. We want the language model to adopt a specific persona or style, and that's where role prompting comes into play.

Role prompting involves instructing the language model to take on the persona of a particular celebrity, domain expert, or even a job role. By doing so, we can elicit responses that align with the desired style or expertise. For example, we can ask the model to respond to a math question as if it were a math teacher, or to generate product names as if it were Elon Musk brainstorming.

The power of role prompting lies in its ability to provide subjective preferences. Different organizations or individuals may have different stylistic requirements for their content. Some may prefer a writing style reminiscent of Hemingway, while others may want a more playful and innovative tone, like that of Elon Musk. With role prompting, we can cater to these preferences and get the language model to generate text that aligns with the desired style.

To demonstrate how role prompting works, let's take a closer look at an example. In the provided transcript, the speaker discusses the process of role prompting using GPT-3. They show how the system can be set to a specific role, such as Elon Musk, and then provide prompts related to that role. By giving the model examples of what Elon Musk would say or do, we can guide it to generate responses that match his persona.

The transcript also highlights the importance of providing synthetic examples of what we want from the language model. In the case of role prompting as Elon Musk, the speaker created fake product names that Musk might invent or name. By incorporating these examples into the prompts, the language model can better understand the desired style and generate responses that reflect it.

However, role prompting alone is not enough. We need a way to evaluate whether the generated text truly captures the intended style or persona. For this, the transcript introduces an evaluation prompt. This prompt asks an LLM (language model with knowledge) of Elon Musk's style and naming conventions to evaluate the product names generated by GPT-3. The evaluator provides a percentage score based on how closely the names resemble Musk's style.

By running the evaluation

Role Prompting

 Fencing is one of those common techniques. It's really easy to do. What it is just asking the LN to play the role of a specific celebrity or someone that knows about that is a domain expert. You could also just ask it to play a specific job role. So yeah, as a math teacher, answer this question. And there is some evidence that actually does improve the intelligence of the model slightly to say. You can get a better math scores when you tell it to act like a math teacher, but the way that I tend to use it is really just to get it in the right style, because some things are subjective about L M. Responses and, some organizations want something in the style of Hemingway, whereas another organization might one as something in this solid. You need blights it? 

No. Whatever your preferences are. There's no good answer or right answer. But with role playing, you can at least get to a good subjective preference. So let me show you an example. Now. We have in here, the system pro the role setting and then the normal user prompt hair. And I'm just going to show you how those things come together. 

For the normal prompt. We're just passing in here, brainstorm a list of product names for a shoe that fits any foot size. That's the task that we wanted to do. Whereas with the roll prompt, we are telling it, you are. Elon Musk and you're brainstorming names, new products. I know you've given some examples and this is what I think is necessary to make rural prompting work in. And the shed because. Previous, he used to be able to just say you are. Elon Musk, whatever. 

And it would do a decent job or at least a better job. But what I'm finding now is that role prompting doesn't make that much of a difference unless you give it some examples of what Elon Musk would, I would name these products. So I made up some fake products. Had. So this is a refrigerator. It dispenses bad. 

And then I just lazy kinda called it beer, fridge, X, like space X. So not a beer fridge, like not a fine through. Which had products that Musk has invented or named. So that's that's really helpful, I think is if we can even make some synthetic examples of what you want and put them in that. And then you can get it into the type of character that you want with the Elon Musk Example here. 

So I'm just going to run this and you'll see.  That if we look at the role-play Anset. We're getting something much like Elon Musk. The universal fit shoe company. Fit all ex. Whereas with the previous example, we were just asking it without all the, all of these things that we're just getting. Adaptive fits, dried universal fit flexi foot. 

So I'm not getting Musk like names. So that's role prompting in a nutshell. But that's not where it ends, right? Like we need some evaluation metric. And typically for this type of evaluation metric, you need. An LLM to check essentially whether as a judge, whether you are falling ill, must style or not. 

So we have another prom tear, which has, Is it language model with knowledge of illness, style and naming conventions evaluated. Fucking product names. And determine if they sound like they're credited by ILA MUSC.  So we have the product names and then we need to provide the evaluation as a percentage. And then we have some templates here that we've said like product X or the product company or not the product. I said, I find that this really helps. If you're giving you some examples, just like adding examples to the previous prompt, where they helped as well. And what we're responding with here is instead of the product description and product names, we're responding with an explanation. 

And then the Musk likelihood. Now the reason I do it this way round by the way, is that if you ask it to provide an explanation first or some analysis or thinking first, it does a better job of getting an accurate percentage or a reliable percentage that you know, will come back again. And again, if you run it multiple times,  And so this is like a small chain of thought implementation.  All right. 

So we're going to run that. Which is going to see. What we're doing. Is running the.  Role play names with descendant names and just seeing  which ones are more Musk role-play names to really well. We have 90%, a hundred percent likely to be Elon Musk. We also have the explanation. 

So if we don't agree. And we can just check this and see. I think that, that works really well. And then if we look at the standard names, we've got a 60%, 50%, 70%. So we can really tell the difference. Like these names still sound like Musk, according to this. Because it's playful suggested innovation. Yeah. 

Like some boldness, a wound school nature often found a Musk's branding. Yeah, so yeah we could optimize maybe the the percentages hit by changing this prompt. And, obviously this is a prompt as well, so we need to optimize this prompt to. But it's usually much easier to optimize an evaluation prompt. Because you can. Rewrite the percentages yourself and say this should have been 20%, 

and then you could test again. This evaluator and then get to a prompt that gets it right and starts accurately repeating the 20% rule. Cool. So anyway, now you have this, you could AB test your original prompt provision. Now you can evaluate the must names. And then go back and just check whether it's actually following that role or not. 

What is the purpose of role playing in language models?

How can role prompting improve the performance of a language model?

What is the significance of evaluation metrics in assessing the effectiveness of role prompting?

How can role prompting be used to improve the quality of responses from a language model?

How can evaluation prompts be optimized in role prompting?

The funny thing about AI is that you can make it smarter just by telling it to act like a genius.

Act like a genius

UPSERT HEADQUARTERS – THE DAY BEFORE END OF QUARTER

You’re in a boardroom at UPSERT Headquarters and your boss, William Winters, Brand Strategist at Whipple, is asking you to figure out a way to protect the company’s data from Open AI. You’ve been researching the LM Studio and think it might be the answer.

We need to keep our data safe from Open AI.
I've looked into LM Studio, and it seems like it could be the answer.
But we need to decide how to use it.
What are our options? Do you have any ideas on how we can protect our data?\

LM Studio is an open source model created by Matter that is used to run models on a local computer. Despite being a smaller model than Chatpetite, LM Studio is slower and doesn't usually follow instructions as well as GPT-4 does. However, a new model created by Mystro 7 is getting closer to the results that one would normally get from GPT-4. 

For those looking to use LM Studio, it is important to understand the technical aspects of the model as running into errors is something that is common. Despite this, the text generated by LM Studio is still decent and with some editing, can be turned into something worthwhile. 

What makes LM Studio so unique is that it was created with the intention of creating narrative and character development. This is done by having two paragraphs per section. This allows for a unique fantasy style that is not found when using other models.

Despite the capabilities of LM Studio, it is still not as good as the free version of Chatpetite. This is why many people are hoping that the model will get better, as having something that is as good as GPT-4 would be extremely beneficial to those who want to create interesting stories. 

Overall, LM Studio is a great model that is able to generate interesting stories and narratives. Despite its shortcomings, if used properly, LM Studio can create some unique stories. It is important for those using the model to understand the technical aspects as well as the capabilities of LM Studio in order to have the best experience. With some practice and patience, LM Studio can be used to generate some great stories.

LM Studio

 Okay meet. Lama this is an open source model from matter. They release it which is really great because they spend all the money training it and then just released to open source. That means I can actually run this somewhere on computer and then there are different. Sources online where you can run this, like on hugging face or replicate. But I really wanted to run it. At my own computer, I'm using something called L M studio. Which you can download is just a Mac app. And and then once you've got that, Set up, you can download anything from hugging face. 

Like this is a downloaded the Lama to chat 13 B. Model and you can prompt it just like you would  So just going to paste in my prompt. And see how it does. What I've found in general is that obviously running it on your local computer is a little bit slower. So even though this is a smaller model than chatting petite that the response has actually come out relatively slower. The other thing is that it doesn't follow the instructions usually as well as GPT four, does. Does. These models, these episodes model is getting closer to what you would get from the free version of challenging.  And in fact, there's a new model out by Mistral which the Mystro seven. The model, which is actually pretty good. And pretty close to chat cheap T levels if you're comparing it to the free version, but nobody has managed to get to the actual results that you would normally get. From GPD for. 

So while this is generating, let's familiarize ourselves with what we came up with jazzy beauty. You can see. Wrote some pretty interesting stuff here. And, we have two paragraphs per section. And it's a unique fantasy style. With actual narrative and character development. Let's see if we can get that We run into an error. Sorry. You do run into this sometimes, unfortunately running these models require some technical ability in some cases. 

But I think in general, I would say these models aren't quite there yet. They're not as good as what you get from chat, GBT, even the free version. We're all hoping that these models will get better now because it would be very cool to have. Something that's as good. 

Now you can see here that the text isn't bad, it's come up with some interesting stuff here. It has, pretty decent introduction. But again, I prefer T's version.

What is the name of the open source model used in the tutorial?

What is the name of the software used to run the model?

What is the name of the open source model from matter?

What is the difference between running the model on a local computer versus online?

What is the name of the new model released by Mistral?

We're just now getting to the point where OS models are as good as ChatGPT (the free version), except they have one major advantage: no data sharing with OpenAI!

I don't want to share my data with OpenAI

Alright, team, here's the situation. We've been tasked with making some creative decisions for our coffee shop interiors, but we don't have enough data to work with. Our boss wants us to use memetic analysis to analyze and tag images of coffee shop interiors using AI. This will help us identify patterns and trends in the industry, and inform our design decisions. So, let's roll up our sleeves and get started!

I know it's frustrating that we don't have enough data to make these decisions, but memetic analysis can really help us out here. \nBy analyzing and tagging images of coffee shop interiors, we can uncover valuable insights that will guide our design choices. \nThis process will allow us to understand the trends in the industry and make informed decisions that resonate with our target audience. \nSo, let's dive in and see what we can discover through memetic analysis!

Memetic Analysis: Uncovering Insights with AI and Vision

In today's digital age, understanding consumer preferences and market trends is crucial for businesses to stay competitive. This is especially true for industries like coffee shops, where interior design and ambiance play a significant role in attracting customers. But how can businesses gain insights into their competitors' designs and make informed decisions about their own establishments?

One innovative approach is using memetic analysis, a method that combines artificial intelligence (AI) and computer vision to analyze images and identify patterns. This technique can help businesses gain valuable insights into the typical interiors of their competitors and make data-driven decisions about their own establishments.

The first step in conducting a memetic analysis is to gather a large dataset of images from various coffee shops. Manually analyzing these images can be time-consuming and inefficient. However, with the advancements in AI and computer vision, it is now possible to automatically tag these images using machine learning algorithms.

To scrape a large number of images from Google, a Google image scraper tool can be used. This tool automates the process of collecting images based on specific keywords, such as "coffee shop interiors." Once the images are downloaded, they can be analyzed using the AI-powered Google Cloud Vision API.

The analysis begins with inductive coding, where two different images are randomly selected and compared. The images are converted into base64 encoding and sent to the API for analysis. The prompt asks the AI to compare the two coffee shops using inductive coding and identify similar and different features and design elements. The response from the API provides valuable insights into the shared and unique attributes of the two coffee shops.

The next step is deductive coding, which involves analyzing each image against a set of predefined labels. This step is more time-consuming and expensive, but it provides a comprehensive understanding of the design elements present in each coffee shop. To speed up the process, asynchronous coding can be utilized, allowing multiple API calls to be processed simultaneously.

The results of the deductive coding are stored in a dictionary, which includes the labels and corresponding coffee shop images. These labels can be customized based on the specific design elements or features that the business wants to analyze, such as natural light, wooden elements, or suspended lighting.

Once the analysis is complete, businesses can gain valuable insights into the design choices of their competitors. By comparing the frequency of different labels across the dataset, businesses can identify popular trends and common design elements. For example, if a high percentage of coffee shop images include patrons,

Memetic Analysis

  Okay. Say you're designing a coffee shop and you want to figure out what the typical interiors for your competitors. So you want to understand what type of furniture they using, whether they have plants whether there's a chalk, a chalkboard on the wall for the menu. Like this. 

And you wanted to do that analysis and see how often a certain things come up versus not. So one interesting way to do this. Is instead of manually looking through a bunch of images. You could tag them automatically with AI because aegypti vision is actually pretty good at this now. What you need is you have to scrape. Google images, which is actually the hotpot. But thankfully there is a tool that does this. 

What do you say if you just search a Google image scraper? Yeah, literally just. That's just such a Google image of scraper. And that was the Fest result here. But however you get the data. It doesn't really matter, but I'll show you how it works. I.  I just, this is in the main.py. So when you download this to your computer from get hub then you have this curd you just put in here. What are the different keywords that you want to search for? 

I downloaded a bunch and you can see. Got a bunch of, coffee shops in Beijing. Got a cough coffee shops in the room. And then also New York, I'm willing to compare them potentially. So I'm going to just run an abbreviated. Part of this analysis I would say by the way, it takes a while to down the them say I downloaded a bunch and it doesn't always work. 

So you can see, I didn't get 80 every time we got 36 and Sydney. But once you literally just run Python, main dot PI. And you should see these images. Stop appearing in hand. But that's not the part I'm going to teach you because that is someone else's code. And I'll let them teach you that in that read me. But what I'm gonna teach you is what's due when you have the data, right? 

This is a script for the Jeep T vision API. And you just need to run. These libraries here, but I have all the code written here. I had to pull in. What are the different coffee shops, right? As you can see how many photos you got just looking at the first five here. See, I've got 4,500 just from New York. 

Got a bunch of Sydney. Et cetera. So. What I'm going to do is I'm just going to.  Just limit just to Let me just a little bit to Sydney.  

Just for testing because a visionary pair is expensive. If you're running a bunch of times, it also takes a little bit of time. So I'm just gonna run it on these 33 images and in Sydney.  All right. And there's two pastas. One is inductive coding. And then the second is deductive coding. 

Inductive coding is when you're looking at these and you look from, for patents and you're comparing the Migo. Okay. That one has a coffee. You know that one has a chalkboard for the menu. And then there's other ones that have chalkboard for many use as well. So maybe that's something that I want to hear again, is another chalkboard. 

So I've noticed that pattern. And now that I know there's a pattern. I want to find that everywhere. So there's another truck board. There's another chalkboards. So that's deductive coding, inductive coding is coming up with the patterns, comparing and contrasting to find what's different or what's similar. And then deductive coding is then going and finding those patents everywhere else. So we're going to do those two things here with a GP vision. All right. 

The first thing we're doing is inductive coding, and we need to be able to send an image to opening eyes. So the way that you send an image is with the base 64 encoding. This literally just takes an image and then turns it into some texts. And that you consent along with your API call.  And this could does, is it just looks in the folder and then it just chooses two images at random. From the different folders that we have. 

So we just have the one folder right now, but if we have multiple folders that. It would look in all of them. Oh, I did mention, I forgot to mention one thing when using Google images scraper you'll probably need to download. I had to download this Mac. 64 Chrome crim for testing. 

When it scraping these images you need that in that. Here I've just put us in folder called WebDriver and then it opens that up. That's, I've just mentioned that because I ran into an error. All right. So back to the inductive coding. So choosing an image pairs, choosing two different images at random. And then it's comparing them. So it's getting the base 64 string, and then it's using this prompt here compared these two coffee shops using inductive coding. What features attributes, elements of design are similar, different. 

And then I'm asking for a structured response here. That's just how you pass it into GPC vision, by the way. You just. To client chat completions. And then you pass the prompt. But then you also pass in the two images as well in the same content string. And I'm just using the user prom tear, but you could also have a system prompt at the top. Cool. 

So let me just run this, see what we got. So it's selected to these images at random. Number 30. 25 and number 31.  And you can see, it takes a little bit of time for the image response is this is opening eyes, biggest modal. Here we go. So now we've got some labels the there's some things that they share. 

So coffee shop a and B. Both have modern designs. They both have wooden elements. But only cough shot B has natural light. And own the coffee shop a has a spacious interior. So that's really interesting. Now we have some labels. We have a dictionary of labels. We can run that and then save it. So if we look at. We should have hit some clean levels so you can see. 

Yeah, there's a bunch of labels here. For these different things and it's rewritten them. We've got the actual kind of folder that they're in the label. And then the phone name.  Cool. So now you have your level dictionary and that's useful in itself because you could run that multiple times across multiple images. 

So we just compared to those images, you could compare every single image against each other. It just takes a little bit of time. That you can run again as well and get a bigger dictionary of data. Now it comes to the deductive coding. And this is the one that's more time consuming, more expensive. 

The way, the reason why I've set this up as asynchronous is. Because otherwise it takes a very long time to run because you're going through every single image and saying and every single label and going, is this label in this image, does this coffee shop have wooden elements? Does this coffee shop? 

How bought-in elements. Does this coffee shop have wooden elements, right? And then you go through all the other labels about, does this coffee shop have suspended lighting and does this coffee shop have a seating areas? There are different tricks for doing this faster or slower. But for me, I found if I do an asynchronous. It means I can get all the labels at once for a single image. And they all run quite quickly. So I runs a little faster. So again, this is here, we're just passing one image in and we're just passing the user prompt, which is much simpler. 

If the label applies to this image, reten one, otherwise reten zero on the output one, a Sierra.  All right. If we get a one, then we are going to return the label. And if we don't, then we're not going to attend on. So this. This does this, the asynchronous code would just set up a bunch of tasks. 

So we run it for every image and then we wait until after all the tasks had done for that image. It's processing all the labels at the same time. And open AI can handle multiple API calls at the same time. What will happen is instead of waiting for each one to come back, it's going to process all of them said time. 

We're going to get through asset. If we're doing 10 labels, we'll get through all 10 at the same time, rather than waiting. If one takes a minute. We're not waiting 10 minutes. We're getting all of them in one minute. So that's the benefit of asynchronous. Cool. And then we're just going to return the labels. 

If the. If the label wasn't there.  Great. And then actually running the deductive code and we just pull open the cleaned labels, Jason, that we have before. And then we have the clean labels. Which which loads to Jason from that dictionary we had, and then it finds the values. 

I've also limited it to just five labels per coffee shop. Otherwise, it's going to take a bit of time.  Cool. All right. So let me just run this. And the processing coffee shop Sydney, you can see that all three of those came back to this in time. But some of them are taking a little bit longer. But we're not waiting for one to finish it before we got the other. 

Cool. So here we go. So now we're processing coffee, shop Sydney.  

And let's see how quickly they come back. Yeah, this is Sydney 30. Okay. Now we're processing Sydney 10. Getting hanging light fixtures. So this is only coming back. As positive if it does have that label, if that makes sense.  

Cool. That's how. Movantik analysis works with vision it's really powerful. Once you figure that you can apply this to pretty much anything. Like any time you have a lot of images, like maybe in your ads or on your website. You can do some analysis to figure out what labels are up. And you can define the labels as well. 

You could sell it. What sorts of things you're looking for. You could give it a D list of things. However you want to do it, that's fine too. You can just add new stuff to this. This clean labels thing, right? You could just add in some new.  New ideas, right? 

Like I want to. I wanna find, this is customers queuing, right? I want to find out if there's a barista that has a tattoo, right? Like you could just add that as another option, copy. This one has Tattooed.  

Barista.  

All right. And there's already funding. If the barista is present in this bit, then we're going to see if they have tattoos, visible tattoos, cool. That's how this works. And then what you're going to get back is you're going to get some, a file name like this. This is just a big list of all the different labels, right? 

So it says coffee shop, New York 42 has the label of street view. This one has the label patrons present, right? And then you can do some analysis after this. To basically figure out okay. How many time, how many does it, how many labels does this apply to you? 

So how many of the coffee shops has this label? I'll give you an example. I'll show you an example here, cause that's still processing. Cause it's going through 33, images is an example from New York. Then I ran earlier, so you can get a list of all the labels into a data frame. This is like an Excel spreadsheet. And then you can print out what percentage of the total images have these different labels? 

So here we're finding that 41% of the patrons that 40% of the coffee shop images, show patrons present. So to me you want to really pay attention to the ones that are on almost everything. Because that's almost like a rule of the game. If you don't have patrons present in your coffee shop image, that's probably really bad. And you definitely want to at least have a stance on that. 

Like maybe have a good reason for not having patrons present in the image of your coffee shop. When you're uploading that image to Google my business. But you should at least address that have the strategic choice here. Similarly, for things that are not very popular you also might want to think about these as a differentiator. 

So exposed brick interior, you could argue that, that's not very fashionable anymore, and that's why only 10% of people had it. Or you could say, this is going to be fashionable. I can see that this is a trend as, or I'm going to stop using this right. That's how. You do this type of analysis. 

Obviously what it doesn't tell you is like which choices to make and what designs to have. But if you're not a designer like me and you want to be a little bit more data to make your decision, this can help inform your decision to give you a map of. What is what are the different creative decisions you need to make? 

When you're doing something designed or visual. All right. Hopefully that's useful. This is my little hack for, being a data guy, but also playing in the world of branding. Hopefully you find it useful too.

What is one potential benefit of using AI to tag images in a coffee shop analysis?

What is the difference between inductive coding and deductive coding in the context of analyzing coffee shop interiors?

What is the purpose of using AI to tag images in the coffee shop analysis?

What is inductive coding in the coffee shop analysis?

How can the coffee shop analysis help inform design decisions?

There are millions of decisions you need to make when starting a business that are too small to test, but are too risky to make without data.

How can I make creative decisions when I don't have enough data

You’re in a meeting with the CEO and your boss, as well as the rest of the executive team. They’re discussing whether the product is priced too low, and you offer a way to find out: Gabor Granger pricing analysis.

That’s interesting, so we basically survey people and ask them what they would pay
But not directly like “what would you be willing to pay”
Instead it’s like “would you buy this product if it was $10?”
Then we keep increasing or decreasing the price depending on if they say yes or no?
Yeah I’m willing to give that a shot: it’ll be better than just guessing!


Gabor Granger pricing is a type of economic analysis that attempts to identify the optimum price for a good or service. It is named after Zoltan Gabor, who developed the concept in the 1930s. It works by asking a customer if they would buy the product at one price, then if they say yes asking if they’d buy at a higher price, if they say no asking if they’d buy at a lower price. This keeps going until the prices are exhausted, and we can draw a demand curve out of the aggregate responses. This method is designed to avoid asking customers “How much would you pay?” because most people don’t know off the top of their heads. However if you show them a price and ask “Would you buy this product for $10?”, they will get an instant reaction and be able to decide if it feels high or low. 
There are a number of advantages associated with Gabor Granger pricing analysis. Firstly, it is relatively simple to understand and use. Secondly, it takes into account a wide range of factors that can affect prices, such as costs, demand and competition. Third, it can assist in setting prices that are competitive, yet still provide a good level of customer satisfaction. Finally, it is a good option for businesses that are looking to maximise profits, or decide on the tradeoff between profit and volume. 
There are also a number of disadvantages associated with Gabor Granger pricing analysis. It can be time-consuming to carry out and may not always produce accurate results if people don’t answer truthfully about their willingness to pay. Gabor Granger pricing analysis is most appropriate in situations where there is a need to price a product or service with not much competition, as it doesn’t take into account competitor price changes and therefore is prone to causing bidding wars. It is important to remember that Gabor Granger pricing analysis is only one tool that can be used to price goods and services. There are a number of other pricing methods available, and it is important to select the most appropriate option for each individual situation.

Pricing Analysis

00:01 Hey, let's learn how to do G or Grainer pricing analysis. So this is the end result that we're trying to get to, and what we're trying to do is draw this to man curve and see at what price should we set the price of our product in order to maximize the amount of profit that we make.
00:18 So I'm gonna walk you through how to do this, but first, just take a look at how the survey is designed.
00:23 So a Ga Granger survey is a little bit complicated. You usually need some specialist survey software, or you need to use custom variables and something like Typeform.
00:34 But essentially what you need to do is ask at the starting price, would you purchase this product for $11? If they say yes, then you increase and say, Would you, you know, purchase at 13?
00:44 And then you kind of keep going and test the different prices. You could also say, if they say no, would you purchase at eight?
00:52 If they say yes, would you purchase at 10? You know, so basically it's kind of, you know, starts at a certain point and then goes up or down depending on what they answer.
01:02 So this is a, you know, once you have the survey data, you can draw the pricing curve so the demand as in how many people are willing to buy at that price.
01:12 And then also the, you can figure out the profit as well, which is what we're gonna do. Cool. So once you've done this survey then this is the kind of data you'd have.
01:26 Like, so you'd have respondent ID on the side, then you'd have different prices.  I've done from $10 to $50 because this is a this is for the brand Ultra which is a men's grooming brand.
01:40 And you know, we can see respondent one said yes at $10, yes, at 15, yes, at 20, and then yes, at 25.
01:49 But then they stopped saying yes after that. So it was all zeros afterwards. Whereas say respondent four only said yes at $10 and they said zero for the rest.
01:59 So then you can randomize this as well to make the survey more it can kind of randomize the starting point to make it better.
02:08 Because one problem you might have is if you always start at 10 then you're anchoring them towards a low figure.
02:15 Or if you always start at 50, then, you know, then maybe you're kind of anchoring them too high and they're gonna say a price that they actually wouldn't buy it.
02:24 But whatever the kind of statistical issues with this, really, you're just looking for a smooth curve at the end. And, and this's gonna give you a decent range a respectable range for you to be in.
02:36 Cool. So you have all the survey data, in this case surveyed 94 people. And then all you need to do is just add up the ones essentially, right?
02:46 So I do it in average and an average of, of any column that's either a one or zero average would just be the percentage that were ones, right?
02:55 So 93% of these were ones was only 5% of People said 50. So then you just need to make some assumption about the market size.
03:05 In this case. We've assumed there are a hundred thousand people in the market. And what this is telling us is that 93% of people so like, yeah, it's actually 92.5 percent of people would become customers if we priced it at $10.
03:20 Which is great, you know, you wanna capture the whole market, right? But it's not so great because actually you know, there are, you know, there are options with higher revenue and and we actually making a loss here.
03:32 So the way you work that out is that the revenue is just the number of customers, tons the price, right?
03:39 So, you know, lower price, you need more customers in order to make the same revenue. And then the profit, we've just made some assumptions for this business.
03:47 So they have to work out the profit, we take the revenue, and then we subtract this they've got in this case it's a million dollars in fixed costs, which is important because, you know, whatever happens if they don't sell any units they're gonna lose a million dollars.
04:05 So yeah, that's a more realistic way to look at it rather than just margin because you know, if you're a startup, you have like a, you know, you have to pay for a warehouse to house all these products, right?
04:16 Like you have to pay for staff and that you, the staff still need to get paid no matter how many units you sell.
04:22 So you usually have some fixed costs that you have to factor in, and that necessitates you selling at least a minimum number of units in order to kind of get to the right level.
04:31 And then you have the variable costs. So in this case, it only costs two bucks to make the product. So you know, we're just multiplying the number of customers by two bucks and the, these two things into place.
04:41 So you can see that we lose money on the $10 even though we have 92,000 customers with with $15 we have 82,000, so we lose 10,000 customers.
04:55 We're gaining a lot in revenue. So now we're making 1.2 million in revenue, and that's enough to cover our fixed investment of a million.
05:03 And also the unit costs. So we're actually turning a small profit here. And and then you know, don't worry about this line.
05:10 I'll explain that in a second. But then it kind of keeps going on, and these are basically the estimates. And you can see that you know, this is the revenue maximizing point but you're not making very much profit.
05:22 Whereas if you increase the price to say, $25, you're only making half of the market by your product. But you know, and, and your revenue is lower, right?
05:31 Like it's $20,000 lower than the $15 option. But that might actually be better to have less market share and higher profit.
05:39 So think about this, like Apple, you know, Apple could price their phones cheaper and they would get a lot more people buying iPhones, but they would make less profit.
05:47 And, and right now they make the most profit out of any of the other phones in the industry. So they've kind of chosen a price point that's higher up here.
05:56 And You can see $125,000 in profit. Now, if we keep going it's actually gonna get worse because yes, we're making more per unit but we're selling less units, right?
06:09 So we're losing a lot of market share here, this share here. So you know, we're getting like a hundred K in profit by selling 39,000 units.
06:18 We're getting 78 k in profit by selling 82,000 units. So, you know, I mean, it's really up to you in terms of like what the business goals are and how much risk you wanna take, you know, things like that.
06:29 But it might actually be better to have a higher price product and sell less units because it maybe than you know, that's easier business to run or, you know, whatever.
06:38 So like a lot of luxury or boutique businesses operate these, you know, higher price and you know, higher price levels but you need to sell a certain amount of units to cover those fixed costs.
06:50 So for example if you, if you only sell 30% of the market you're only just making a million dollars in revenue and that's, that barely covers your fixed costs and it doesn't cover, doesn't left anything over, left over for the $2 per unit.
07:05 So, so you actually lose money $17,000 in this case, and that kind of keeps going on. So yeah, you could price it 50 bucks, but you know, only 5% of people wanna buy it that rate, right?
07:15 So you can actually lose, you know, a quarterly, three quarters of a million dollar dollars here. So that's a terrible choice.
07:23 So it just shows that, yeah, like yes, you can choose your pricing and you can be strategic. And once you map this out, you can make a more informed decision.
07:31 Really, it's a market that decides, you know, what you can get. Like, you can't just say, Well, my product's worth 50 bucks.
07:37 That's really for the market to decide. Cool. So now what is this line at the bottom here? Well, what we've done is we've drawn the curve here, see this curve, this trend line.
07:49 And the way that we've done that is if we go to edit chart and then we go to customize series or on data series one.
07:56 So that's that if you click on trend line, and there's a slightly different way to this in Excel, but you know, works is the same you can stick a trend line on and then you can say use equation and then show the R Square.
08:09 And what that does for you is basically just plots the nearest fit to all the, all the points here. So you can see some of the points are slightly off that one, slightly above, that one's slightly lower.
08:18 And this is a in this case we used a a polynomial. So there's like different options but polynomial polynomial fits best should do that polynomial fits best cuz it lets you kind of do this curve up and down.
08:36 Cool. So the, in this case, the, the Oscars was 0.95, A perfect fit would be one. So this is a very good fit.
08:45 And then and then that gives you the equation to derive the, the profit, right? And so it predicts given a price what profit you're gonna get.
08:55 And the way it does that is just this complicated formula, but you don't have to worry too much about it.
09:00 Basically it's saying you start with the value of minus 73,000 or sorry, minus 736,000 7 68. So then you add 7,020 yeah, so 70,278 multiply by x and x the price.
09:20 So if it was $10 and it'd be 70,278 times $10. So it's about 700,000. And then you also add minus one for, for two times x to the power of two.
09:37 And that's basically what gives this kind of coving up and down chip. So you don't need to worry too much about the statistics here you know, just know that that's what that's doing.
09:46 But all I've done here is just copy this function across, right? So is minus 7 36, 7, 6 8, and then plus 70 2 78 times B one B one is the price plus minus 1 4 42 times B one the price again to the power of two.
10:03 And there we go. Now we have our prediction. So we could do this for any price, but we've done this across all the prices here.
10:10 And the reason why you wanna do this is you see there are some lines are, you know, dots are above and below.
10:14 And really what we're trying to do is smooth this out. Like there's some statistical noise here, right? So we're just kinda smooth this out and find the optimum price, and it is at the $25 mark.
10:25 So making, we, we would predict we're making 118,000 in revenue. And the reason why the smoothing out is important is that because, you know, of the survey data, the $15 price mark, it's actually like higher, higher than we would expect.
10:38 So we might look at this and say, Hey, we're still gonna make this 78 grand in profit, but actually the the function or the prediction is that we lose 7,000.
10:47 So we're playing with fire there if we're gonna, you know, stick with this ROS estimate. And and instead we would kind of wanna stick at least, you know, somewhere in this range.
10:57 If you wanna optimize, you know, slightly more to revenue, then you could be at the lower end of this range.
11:01 You wanna slight optimize slightly more to, to margin then you could be at the upper end of this range. You know, but you know, it's giving you a place to choose your price.
11:12 Cool, Hopefully that was helpful.

Why are fixed costs important to take into account with regards to profit?

What might be the rationale for maximizing revenue over profits?

Why might Gabor Granger analysis not be a good fit for industries with intense competition?

Are you charging enough? Most businesses guess at pricing and move on, but that approach is going to leave money on the table. How do you set the right price to maximize profits?

What price maximizes profit?

Building a marketing mix model is not just about finding the right parameters, but also the hyperparameters used to transform your data to account for adstocks and saturation.

You've been looking for an excuse to learn how to use evolutionary algorithms for building models automatically, and now this might be your chance. You hear on the grapevine that a colleague is struggling with their model.

Yes you heard correctly
We're struggling with the GoolyBib model for Italy
I'm not satisfied with the model as it stands – the error is over 10%!
This could be a good use case for automated hyperparamter optimization
Doing it manually is getting us nowhere fast!
I'd appreciate your help

Every model has parameters: these are the variables used to make predictions when the model is built. For example your model might say that Facebook Ads drives a cost per purchase of $5. How the model arrives at that conclusion is dependent on Hyperparameters: parameters that control how the model learns the right answer. To take our Facebook Ads example, our cost per purchase might get worse at higher spend levels due to saturation, or there might be a carryover effect where spend today affects performance tomorrow. 
These transformations of the data are key to building an accurate model, but it's hard to know ahead of time what values are correct. The manual way of figuring this out is to choose a value for a parameter, for example an adstock level of 20% (20% of your spend from today has an impact tomorrow), and then move that value up or down to see how it impacts model accuracy. If you increase it to 30% and accuracy of the model goes up, you know that's a better fit. The problem is that each parameter affects the behavior of every other parameter in the model, so it can be hard to find the right value.
So there have emerged several strategies for dealing with this task of Hyperparameter optimization. The brute force approach is a grid search: try every possible value until you find the right one. This isn't always possible however, because there may be more potential combinations than there are stars in the universe! Random search lets you set a limit of how long to look for the right values, randomly guessing within a budget. The method we're using today is an evolutionary approach, which learns in a more efficient way than random guessing. It deploys a population of potential solutions, then kills off the worst ones (with the highest error) and replaces them by mutating the best  performers in the batch. With hundreds or thousands of iterations we can efficiently arrive at the best estimate for even a very complex set of parameters.

Hyperparameter Optimization

GoolyBib Italy model
Hi,
Thanks for offering to help: please see model attached. 

We have been choosing the values for theta (adstocks) and beta (saturation) manually and this is the best combination we've got to so far.

Best,

Statistician and Management Consultant

To optimize the whole, we must sub-optimize the parts.

Marketing campaigns tend not to have a linear relationship with sales. As you spend more money on a channel, the efficiency decreases because you have to outcompete a greater number of other advertisers for ad inventory. In order to model a non-linear variable in a linear model, we must transform the data. The Power saturation curve we're using simply takes the advertising spend to the power of beta. But what is beta? It's a Greek letter, because by convention mathematicians use them to identify parameters in a formula. 
We don't know what the correct value of beta is – it's our job to estimate it. The way this is done is by trying a lot of beta values from ~0 (completely saturated) to 1 (linear) and see which value best improves the model. If we see the accuracy increase from 0.3 to 0.4, but then decline again at 0.5, we know the true value could be around 0.4.
We do a similar job for adstocks, to estimate the percentage of advertising impact that's felt the next day after spend. For example if we choose 0.2 then 20% is felt the day after, 20% of that the next day (4%), and so on. The process is the same: calculate the accuracy of the model at each level of adstock, and see what it does to accuracy. Of course this can get complicated to do across multiple variables, as if you change one it changes the whole model, and the odds of ending up with the best possible combination are low.

Theta and Beta

For models with up to 5 or 6 parameters, it can be feasible to adjust the model manually and find a good enough solution. However this task quickly becomes impossible with more parameters because the number of potential combinations increases exponentially.

Finding numbers for Greek letters

Rather than manually choosing the value for each parameter, let's get Nevergrad to do it for us. It uses an evolutionary algorithm to build hundreds or thousands of models to find the right parameters for us.

After digging through the Nevergrad documentation and using some marketing mix modeling code you had from another project, you have something ready to run.

Nevergrad is a Python package created by the Meta (Facebook) team, for gradient-free optimization using techniques. It works using differential evolution and particle swarm optimization, which optimizes a problem by iteratively trying to improve a candidate solution with regard to a given measure of quality. 
This process is called evolutionary, because it is based on the ideas behind evolution in nature. The simplest evolution strategy operates on a population of size two: the current point (parent) and the result of its mutation. Only if the mutant's fitness is at least as good as the parent one, it becomes the parent of the next generation. Otherwise the mutant is disregarded.
Such methods are useful because they make few or no assumptions about the problem being optimized and can search very large spaces of candidate solutions. Because it does not use gradients like other optimization methods, it can be used on optimization problems that are not even continuous, are noisy, change over time, etc. This is perfect for machine learning where the actual model might be a black box, where changes in one part of the model can give non-linear responses in another part of the model. The downside of evolutionary algorithms is that they are computationally intensive, so take a while to run, and do not guarantee an optimal solution is ever found. 

Nevergrad Optimization

Hi, I'm going to walk you through how Nevergrad works
I'm gonna take the file here, and just going to make a copy, so we have our own copy in drive
First you want to install Nevergrad
So if you just click into that cell and press shift + enter, or you can click play here
That's just downloading the library and installing in on the machine Google is giving you for colab.
Now we just need to load in all of the libraries that we're going to use
Just shift + enter, or click the play button.
And these are some functions we're going to use as well
We need to authenticate Google drive so we can import the data
So if we just run that cell, then it'll give you a link to click through, and you just want to authorize, then copy and paste
Hit enter
Now if we run this cell
Its going to pull in the data from this Google doc, and it's going to display it here as well
So we know that we've got the right data, its the number of days, then facebook spend, radio spend, tv spend and then revenue
Now let's plot the data so we see how the data is trending over time
So we see here it's a plot of revenue, and then we see all the facebook, radio and tv spend underneath
Now in order to run Nevergrad we need to do a little bit of setup
First we need to get all the spend columns, so the media variables are if 'spend' is in the column, and then we just need to create the X and Y variables
Y is revenue, that's what we're trying to predict
Then X is what we're trying to predict it with
This is an adstock formula, it helps us take into account that there will be some lagged effect of advertising
And Nevergrad is going to help us find the right adstock values
It's going to try lots of theta values here
And it needs that function in order to transform the data in each run
Same for saturation, this is diminishing returns its the same thing, its just a power transformation, with just one variable
You can have more variables, if you do something more complex, like a weibull adstock or a hill saturation curve.
Now we need one big function to model everything together
So it needs to take in the facebook spend theta, facebook spend beta, and then same thing for each other channel
And then it just pulls it together, and then creates the adstocks, and the saturation, so it does adstock first, and saturation after that, which is best practice
Finally it creates a new data frame, basically an excel spreadsheet
then creates the X and Y values, and splits them into X train, X test, and X train X test
So this is a train test split that lets us check the accuracy, with data we haven't seen before, in the model
This prevents overfitting of the data
Finally we have a linear regression model
Although you could replace this with whatever model you want to use, if you want to use Ridge regression or something different, and then it fits the data, and gives us predictions, we then calculate the accuracy metrics
So normalized root mean squared error, is just an accuracy metric that gives a percentage, that penalizes really high percentage values, more than mape 
The mape is the mean absolute percentage value, and that's just a general percentage you were wrong each day
We also have the R squared as well
Then we kind of wrap this function and get all of the results, but only return the mape value
This function is what nevergrad is going to use to optimize to
So here we're optimizing to the mape
But we could swap this out for nrmse or r squared if we wanted to
Now we just need to make these scalar values
Don't worry too much about this, but we're essentially just setting the bounds of what the parameters could be, and then we're going to run the optimizer here
The optimizer, this is actually Nevergrad doing its thing
We're passing in the instrumentation into the parameters, and then we're giving it a budget, and we're saying this is how many times we want you to run a trial
And the trial is basically the algorithm choosing what kind of parameters to set or to try next and then run that trial to see if those parameters worked
Then we just want to log the error values, so we can do a chart afterwards of how quickly it approached the lowest mape, 
so that's what we're doing here
we're just registering the callback
every time it tells us the answer for that trial
ok and we're going to set a timer, so we just see how long it takes to run
this is the optimizer function, this is where its actually minimizing the error
this is like where all of the work is actually happening
so when you run all of this
oh I have an error
main function is not defined
if you ever get this, its because you probably ran things out of order
so I'm just going to go back and run this one here
then run this one again
there you go, you can see it took 1.8 seconds to do those 100 trials
and these are the values that it came back with for facebook beta and facebook spend theta
So these were the parameters for the adstocks and saturations, that got us the best accuracy out of all of the models they tried
You can also kind of push those parameters into the function that you created in order to get the MAPE
So it's 0.076 so it's 7.6%
Then you can see if you plot
So just run that so you can see if it works
It's going to be a little bit different each time, because its an evolutionary algorithm, so there's some random elements in there and it wont' always be the exact values
Just going to run this
We can see here that we started off with a very high mape, it's 18%, and then even spiked up to 20%
Then over time it kind of evolved towards the better values
It got a lot lower, to 0.08 and then stayed around there for most of the 100 trials
Then if we just run the build model function, its going to return us the mape value and the nrmse, r squared value, the model and also the result
So we just run that and its going to give us those values
We can also plot the accuracy
So if we plot the figure, and we take the revenue and the prediction from the result, which is the data frame, then we are showing that
We're seeing here the revenue is in purple, vs the red prediction
That's how you run nevergrad in order to find the right parameter values




Use Nevergrad

What are the benefits of using Nevergrad vs other optimization libraries?

Describe in your own words the steps taken in the Google Colab notebook to use the Nevergrad library

That's great to hear
8% error is much better than we had on our manual model
Looking forward to seeing if we can improve it further by dialling up the number of models

This approach works if you're running up 2,000 models, but it takes a while and hits an error if you set the budget too high. To reach 10,000 models we have to parallelize the operation, meaning apply multiple 'workers' to solving the calculation, rather than calculating each iteration one at a time.

Better than random guessing

We need to make some modifications if we want to scale up the number of models we're building. It will help to parallelize the operation to allow multiple calculations to happen at once.

We have a model that works – we're already getting better accuracy. Let's modify our code so we can run a much higher number of models.

Nevergrad offers parallelization right out of the box, you just need one other library called 'futures' to make it work. Futures lets you create a ThreadPool which manages the multiple workers you create to do the calculations. These workers can carry out tasks asynchronously, as if they were their own mini computers, and then pass back the information to the main function. We run this within a 'with' statement to ensure they're cleaned up properly and shut down after everything finishes executing. 
You can have as many workers as you have cores available for processing on your computer, which you can find with the os library `os.cpu_count()`, though it makes sense to leave 1 core for anything else your computer needs to calculate. For demonstration purposes we're going to just use 2 cores, which is the minimum needed to calculate 10,000 models. We set this in the optimizer on the first line with `num_workers=2`. The following file has the updated script we'll use.

Parallelizing Nevergrad

Copy the code from the last model to show the output of the parallelized code. What is the MAPE for the next model? Was it significantly better?

The parameter for TV spend beta changed from 0.45 to 0.10 in the new model. How would we interpret that?

What does the chart showing mape over time tell you? What does it look like Nevergrad was doing? Is this behavior what we'd expect?

Thanks for sending the results across - very interesting
I thought dialling up the models would get us a better result
However seeing that it jumped around a lot and tried lots of options gives me confidence that we really did end up with the best allocation
Thanks again for your work on this
We can reuse the script with modifications for every new model we need to optimize!

Scaling to 10,000 models

It's tedious to try different diminishing returns manually to find the best one. Facebook's Nevergrad package can try thousands of models for us!

Let's try 10,000 models

Alright team, let’s dive into today's tutorial on creating consistent characters in AI-generated images. We'll explore using a character reference URL for consistency and in-painting to vary scenes with the same character. We’ll also look at a blending hack, though it might not be available forever. Let’s get started and elevate our creations!

I'm not sure how creating consistent characters will really benefit our fizzy drink company. \n
I mean, isn't it more important to focus on our product and marketing strategies? \n
But I guess if it can enhance our brand image or help us in storytelling, it might be worth exploring. \n
I'm curious to see how this tutorial will change my perspective. \n
Let's give it a shot and see what we can learn!

Consistency is key when it comes to creating memorable and believable characters in storytelling. Whether you're writing a book, a screenplay, or a video game, ensuring that your characters remain consistent throughout different situations and scenarios is essential for maintaining their authenticity. In this blog post, we'll explore some techniques and tools that can help you achieve consistent characters in your creative projects.

One effective method for creating consistent characters is demonstrated in the transcript above, using OpenAI's DALL-E model. The transcript highlights how you can generate different variations of a character by using different prompts. However, to maintain consistency, a specific parameter can be added to the prompt, known as the "C dot E F character reference." By adding this parameter along with the character's URL and specifying a desired consistency level, the model can generate variations of the same character that remain consistent with each other.

Additionally, the transcript introduces another technique that utilizes in-painting to create consistent characters. By taking an initial image of the character and manipulating it using in-painting tools, you can create multiple variations of the same character in different scenarios. This technique allows you to cut out specific parts of the image and replace them with new backgrounds or outfits, resulting in a consistent character that can be used in various situations.

One of the advantages of using the in-painting technique is that you can generate multiple images of the same character, each with different expressions, poses, or settings. This can be particularly useful when mapping out different scenarios that your character needs to be in. For example, if you're writing a detective novel, you can create images of the character in various crime scenes or interrogation rooms. By having a range of consistent character images, you can effectively illustrate the character's development and experiences throughout the story.

Furthermore, the transcript suggests that you can use the generated character images as training data for your own stable diffusion model. By refining the generated images and iteratively improving upon them, you can create a more customized and consistent character. This can be especially valuable if you want to have more control over the generated variations and ensure that they align with your creative vision.

It's worth noting that while the techniques mentioned in the transcript primarily rely on AI models and tools, there are other traditional methods for developing consistent characters. These include carefully crafting character profiles, defining their personalities, motivations, and backstories, and ensuring that their actions and dialogue remain consistent with their established traits. These traditional methods can work in conjunction with AI-generated images to create well

Consistent Characters

 Okay, I'm going to show you how to do consistent characters in mid Jenny. So this is an example of a character I wanted to replicate. Like I've got a picture of her and the party just generated this, using this prompt. But say for example, I was writing a story about. She was one of the main characters I'd want to show her in different situations, maybe in the dentist office or at a birthday party. 

Whatever it is. And if we use the same prompt again, we're just going to get different results. And see all of these like very clearly different people, even though they're quite similar. So how do you keep them consistent? So mid Jenny actually added one way to do this. 

So if you click a copy of the link and then let's put in the same prompt again. Actually, yeah, that'd be grab the prompt.  

Just say 

At a dentist office.  

All right. A.  

Okay. So that is a new prompt. And obviously we just run that. We're going to get like a very different person. But no, to keep it similar, just going to click copy link.  And then the parameter you need to add here. Is , 



C dot E F character reference and then paste in the URL. Then you want to add dash. 

You want to say C w. And let's just say. It's, this is from zero to a hundred and so we want to keep it at that's how consistent you want it to be? Let's keep it like 90 on it to be very consistent. All right. Now let's pace that and see. How it works.  And what a man Jenny should do here is keep that character relatively consistent. Over time. 

So that would be really helpful.  

But obviously, while that's running, I'm going to tell you about a different technique that I use. That doesn't rely on this proprietary feature. It uses in painting instead. So I'm going to. And to show you that in a second. Now let me see. Is this look like the same person is. Pretty similar. 

So this is her.  

And yeah, you can see a condo looks the same. So a little bit wrong though. I think to be honest and look, she's wearing the same sequent dress. I didn't want it to be wearing that. I want it to be. In a dentist office. So we'll see if we can address that.  All right. How do I do it? 

What I do, my trick is I want to pan out. I'm going to click pan.  Submit.  And that's going to add some extra bits to the right-hand side.  And we're going to do that twice in order to get a two to one aspect ratio.  

And you can see here that we've added some extra background side.  

And then we need to upscale one of them. Does really matter which one.  And the moon to pan that, again.  

When you get here is basically. Enough space for two of the same character.  

Okay, let me just upscale one of these.  I'm just doing the one with the least amount of noise and the. Background. Okay. So now we have our character. We have, a consistent background on this side. All right. What we can do now is we can do a very region.  



And this is where the magic happens. 

So you can do this actually with any AI tool that offers in painting. So we're just going to select.  This sort of. Basically cutting out this part of the image. So the right-hand side of the image. And instead of. Doing one image with a background on the right hand side, are we going to change it? 

So we're going to say. Sorry. Two images.  Is side by side. The same.  



And you say At an 1975. Priority.  

And then we're going to say.  

This.  Hey. Your sweater.  

Alright.  

And then we're going to get rid of this stuff. We don't need that.  All right. And we've got the aspect ratio two to one.  So let's see if that works and this is not always consistent by the way. Sometimes it does go a little bit crazy, but that should give us the best chance of success. So how does this work? When you're in painting you're basically erasing part of the image and then prompting it for what you want. They image new part of the image to be well. The. Here we go. 

So what we're doing is we're taking instead of this, a wide angle, A shot. We're cutting it in half, or we're saying it's just two pictures of the same. Person right.  

And you can see here that, okay. It didn't get the dentist office saying properly, but it is the same person, right? It's very consistent, which is good. And she is wearing a blue sweater. So that is pretty good. And now all of these have our eyes closed. So one of the things I'm going to try now is I can go back to a very region. Did I say one it's still cut out. You're saying let's just try. The. She's Chinese five for potty left. At wearing the blue sweat 

Staring straight at the camera.  

Alright.  

All right, the, see if that fixes the problem, you're back into any normal kind of prompting techniques that you're going to use to do this. But the key is that. We're in painting. So there's two. Pictures of the same character now. The reason why this is useful is now we can basically cut out. If we drop this into Figma or Photoshop or whatever, We can cut out the different images that we want. 

So we can do this a bunch of times and then get Maybe 10 20 images that the same woman. And there's a few different things. We can do that, but one is we can. Map out all the different scenarios that we need her to be in. But the more interesting thing is that you could then use that as training data to train your own stable diffusion model on this specific character. 

And that's something I've done. In the past as well. You can see that if some of these are better than others you might take a bit of an iterative process here. Like maybe you need to get her with her eyes opened first, keep the sequined dress and then try and work on. Maybe a different background and they'll try and work on a different outfit. And it is a bit of an issue that you're active process, but the really cool thing here is that now you have a method for getting to different. Perspectives or two different behaviors or situations from the same character. 

And that character is consistent. And this works really well. Now the other thing actually I've found.  In general is that you can't do this with uploaded images. There is a little bit of a hack. They have this blend functionality. And if you blend two images together, they're similar. 

You do get a generated version in the same style. If you did need to hack it, I don't know how much longer that. But if he didn't need to hack it and wanted to get a character that you didn't generate that as like an upload character, then you could do that. But for now they've disabled. The ability to in paint, uploaded images. But yeah, hopefully that's helpful for you guys. 

And it gives you a lot of different opportunities to create more interesting character development in your stories.

What is the purpose of using the "C dot E F character reference" parameter in the tutorial?

What technique does the tutorial suggest using instead of the proprietary feature to achieve consistent characters?

How does in painting work in maintaining consistency in characters?

How can the generated images be further utilized in character development?

What is a limitation of the in painting technique mentioned in the tutorial?

There is a random element to diffusion models, so you get a different result every time you run the same prompt, making it hard to keep characters in a story consistent.

Every time I prompt it generates a different person

Ultra’s Offices – Quarterly Strategy Meeting

The hot topic being discussed is SEO: your blog content is good quality but not yet ranking on search engines. How do you build authority and start ranking on the first page of Google?

I think the quality of our blog content is great
But right now nobody is seeing it
Of course we all know SEO takes time
However if we don’t show tangible progress soon we won’t be able to justify the level of investment we’re at right now
The key issue has to be links
In my last company none of our content ranked until we got over 2,000 backlinks from a post that went viral
The rising tide lifts all boats…
We went from 200 to 20,000 visits a week
Of course we can’t expect to just go viral here with our skincare brand, that’s a bad strategy
But we can start building links
Let’s start with identifying broken links on competitor’s sites and redirecting them to us
Should be an easy place to start

In order to rank on Google you need authority. Google’s PageRank algorithm is somewhat of a popularity contest: if websites who have a lot of authority link to your website, they bestow authority on you, and your ranking increases. Of course the links still have to be relevant to the content that you produce. They also have to be legitimate: if our link profile looks unnatural like we’re buying links we can get penalized and de-indexed.
So we need to build authority in a legitimate way, ideally without spamming a bunch of people, or taking up a lot of time with websites that aren’t relevant. The cleanest, simplest and most useful way to do this, is to identify broken links to competitor websites. This occurs when other websites have linked to your competitor but the link is no longer working, it’s leading to a 404 page. This is an opportunity to contact the author and let them know, in the hopes that they’ll reciprocate and link to you as well, perhaps even replacing your competitor’s link with a link to your product instead.

Broken Links

Hey, let's do some broken link building. Uh, so we're going to use <inaudible>, uh, for this analysis. Uh, you can really do this in most of the SEO tools and potentially even, um, you know, use social operators to do this in Google.
Uh, but, um, I'm just going to show you a trust, cause that's what I use. Uh, I think you can pay $10 or something for a trial and, and do kind of the analysis that you need and then cancel.
Uh, so, uh, we are doing this for a skincare brand called ultra start-up and they compete with, uh, this competitive set here.
Uh, so, uh, they've done some surveys of the potential customers and they're finding that a lot of people use, uh, Harry's skincare, um, uh, solutions or they use, uh, Jack Black, which is the market leader in the U S um, and then they also use melamine guts, uh, because this is a very trendy skincare, which say usually encounter when they travel, uh, to different hotels.
Um, so, uh, we're going to go through these and figure out, uh, whether they have broken links, uh, because anyone that's looking to them, uh, could potentially be looking to us as well, especially if we notify them that one of their links is broken.
Cool. So, um, let's go to Harry's. Um, this is for the U S market. So you're going to copy your out, um, wherein within <inaudible> in the site Explorer, just going to put that in there and then hit enter.
Uh, so this is basically bringing up all the data on Harry's. You can see that they have a very high, um, you know, relevance in terms of, uh, domain, like they have very high authority.
Um, but, uh, the main thing we want to go to here and nature estimates is pretty simple to get to best by links.
Um, this is one of the best, uh, pages you could look at. It kind of shows you what pages have the most backlinks to them.
Uh, but, uh, the important thing for this link-building strategy is we can filter by HTTP code here. So we can basically say, um, four for not found.
Uh, and that's going to tell us all the different pages within Harry's that have a lot of links that, uh, haven't been, uh, found.
Um, so, um, we can, uh, you know, potentially search within the results of we're looking for something specific. We can export it into Excel and do some filtering.
Uh, but, um, I just like to kind of rank it based on, um, the Euro rating, cause it's a good proxy for like the most important URLs.
And then I just scroll down until, uh, there's not that many links coming through. So, uh, here they have a, you know, the social mission, which I guess say, give up on, uh, it's not found on the website anymore, our story.
So some of these are like, uh, mistakes. You can see, this is like the France page. It's got France, France.
Uh, so maybe there's some errors and that's why that's for a forum. Uh, but here's something interesting. Uh, they did a post for, um, I guess for, uh, pride week.
Um, and I, it's not showing up on the website anymore. So if we go to the way back machine, internet archive, this is sometimes a store, uh, different pages on the internet.
So maybe we get lucky and kind of see it hit. Okay. So it looks okay how one record and, uh, we can sometimes see what the website is.
So although it doesn't always load all of the, uh, all the images, uh, so let's just see what this page was just going to wait for that to load.
Hmm. It's taken awhile. Uh, it doesn't look like this living anything. Okay. Well, we can get kind of guessed what it is shave with pride set.
Um, we can see if there's another page on the left side That talks about this. Here you go. So they do have like a shave with pride set.
Now this is specific product, but I guess this was a blood post about this product and it's a pretty visually appealing one looks pretty cool.
Uh, so, um, so now we have an opportunity, I think because, um, we have here, um, there's 40 referring domains, uh, to this page that are going to a broken link.
So that's a user experience issue for the people who read about this. Um, there's also, uh, you know, from those 40 demands is 62 different links.
Um, so I like to kind of click through here and just see what they are and what you're seeing here.
Um, this is the referring page, the article that referred to them. Um, and then this is the anchor texts, like the texts that was highlighted and the URL, which is the shared with pride set.
So, um, we can see that this is about awesome packaging designs. That's kind of interesting. Um, this is the corporatization of pride month.
Maybe we don't know the year in here. Uh, that's kind of a negative article, um, you know, but like we would want to appear in packaging design reviews.
That's pretty cool, especially if we have a good package, uh, in terms of, uh, the, um, you know, the, the, uh, uh, the, the skincare products that we, that we created.
Um, there's also here 15 awesome limited edition product collaborations. Um, and then, uh, some article hit. Um, so there's some opportunities here we can go through.
Um, and, um, you know, we can contact these people. So if we click through to here and see the article, um, we can see, you know, we search the page for Harry's.
Um, we can see what they're read about. Um, and it looks like they're kind of listing different ones. So we have Harry's who gives us scrap.
So, um, this might be the sort of article that we want to get included into. And, um, if we, um, if we have a package that, um, in this case, uh, you know, fits the bill, we can contact the author of this post.
And so this is massly, um, Alrighty. And, um, you can usually click through and kind of, you know, find that Twitter or, um, you know, the, your, uh, their email.
Um, so, you know, you can just search for, uh, dialogue. Um, so that's like a little bit more about, uh, this is her personal website.
Uh, so she's on my crack as a PR kind of, uh, platform. So, um, yeah, ZoomInfo has a contact details.
It looks like. Uh, so, uh, yeah, that's, um, that's interesting, right? That's how you kind of go through, um, and then you can contact her and say, Hey, I noticed that, uh, the page you're linking to, um, isn't, isn't working anymore.
Uh, but, um, you know, I think actually, you know, we, we have a new package, uh, that you might want to check out that we might want to include.
Um, let me know if you want me to send you, um, you know, some of the products for free, just so you can check it out and, uh, you know, give, give it to a friend and see what they think, because it's a men's product in this case.
Uh, so, um, you know, that's a really good way to build that relationship, even if they don't link to you directly here, I, they might, you know, tweet about you or, um, you know, they might, uh, include you in a different article that they're working on.
Um, and, and you can't predict any of this. You also can't like specifically ask them, like, uh, pay them to do this, uh, because that goes against Google's guidelines.
Um, you know, but, uh, I think, you know, this is an opportunity to find someone who is looking to a competitor that is running about the space, um, that, um, you know, potentially will be influential and willing to, uh, to link through to you, uh, in the article or in a few trout skills.
So that's really what we're trying to find. Okay. So that's, that's one opportunity, uh, let's move on from Harris and see if we have anything for Jack Black.
This is the school, this is the leader here. So this is telling those that total, um, backlinks is good to hear best buy links again, um, and then filter it down to four, four.
Okay. So no, uh, for a force for their websites. So this is pretty good for them, uh, bad for us.
I mean, so they, uh, you know, whoever is managing our website, uh, you know, it hasn't made some of these messy changes.
Um, so they haven't taken off any of the pages in their site. So that's good. I would also look internal.
Um, it looks like none of the internal links are for, for either. Um, so that's, that's a, you know, that that's going to happen like, uh, relatively frequently.
Um, there, isn't always an opportunity to kind of have to search for the next one we're going to look at is not an, a goats.
Uh, this is the really trendy one, and we're on best by links here. So let's filter it down for, for, for, and here we go.
There's a lot of two TCS, so they have this face, grapefruit face cleaner. Um, so, uh, that has, uh, some, uh, it looks like they've changed some of these, uh, the URL structures here.
So, um, you know, him, men's journal fashionista.com, Jacob, Jacob, and Sebastian. Um, so it looks like there's a lot of these companies that have, um, you know, looking through to this and to hear that you're washing your face or wrong, et cetera.
Now in here we go. So they just have a few links to the bottom, so we can go to them and say, Hey, by the way, that link is broken and also, you know, did you want to look at our product, um, and, and see, see if you like it, if we have a face cleanser and then, um, and then maybe they'll add us as the fourth one here.
Uh, so I think that's, uh, that's like pretty straightforward. This is like ideal. This is exactly what we wanted. This is a lot less complicated than the, uh, than the pride week, uh, opportunity.
Um, the, uh, the other thing I think is useful to do is just see if they actually have that product elsewhere on the site.
Maybe they just change the URL, I guess. So they do, um, this is, this is useful to know just because, um, if they, if that product still exists, uh, then it's, um, I think like good practice to give them the updated link, uh, to the, uh, you know, remember, like you're not trying to do this at scale you're, should we get good quality links from influential people and build that relationship?
Um, so if you think about it that way, it's worth the extra effort to say, Hey, here's the correct link for that grapefruit face cleaner, because he now doesn't have face in the Euro, whereas before it did have face, um, and now it has the different fluid ounces in hairs off.
So, uh, that's, that's very useful. Um, I think that the, um, if, if the, this product doesn't exist, uh, then that's an opportunity for you to change the messaging and say, Hey, they didn't actually sell that anymore, but we think we have a good replacement.
So think about that. Uh, it's, it's, you know, trying to spot these opportunities. Um, but it's not really about scale.
It's more about quality. We're trying to find, um, you know, relationships with people who are LinkedIn to similar content and therefore, um, you know, this is just an opportunity to reach out.
Um, it's not like the end of the relationship. Um, so try and invest as much as you can into each one.


What part of Ahrefs do you need to go to in order to find broken links from a competitor?

What are the benefits of broken link building?

How would you pitch a broken link backlink opportunity?

How do we build authority

The difference between a website that gets 200 visits a month and 20,000 is backlinks. It’s night and day: if you don’t have authority on a relevant topic, Google won’t rank you anywhere near the first page.

GoolyBib Headquarters – Website Planning Meeting

Your CEO just read news online that Google was going to deprecate Optimize, their A/B testing tool, and is worried: his Q2 plans depend on that tool!

Even if the rumors aren’t true, they’re spreading because there’s something to it
Google hasn’t updated Optimize in years
They won’t offer a free tool forever
So let’s use this as an excuse to migrate off
What can we do to execute our testing plan if 


As the internet marketing world changes, so do the tools and solutions used to optimize websites and campaigns. One of the biggest changes in coming years might be a move away from Google Optimize. Google Optimize has long been the go-to solution for setting up A/B tests and other experiments, and is used on 500,000+ websites. But with Google making changes and discontinuing the service, many are left wondering what to do. 

While Google Optimize might be going away, you can still set up A/B tests and experiments with the same degree of accuracy and ease in another free Google tool - using Google Tag Manager. Google Tag Manager (GTM) is another free service from Google that allows you to add, update and manage tags on your website. It’s a powerful tool that can be used for a variety of tasks, including setting up A/B tests and experiments. 

So how do you set up an A/B test in Google Tag Manager? Let’s break it down. 

The first step is to create a new tag in GTM. You can do this by clicking the ‘New Tag’ button in the top right corner of the screen. You’ll then be taken to the tag creation page where you can choose the type of tag you want to create. For an A/B test, you’ll want to select the ‘Custom HTML’ tag type.

Once you’ve selected the tag type, you’ll need to enter the code for your A/B test. This is where you’ll use JavaScript to randomize visitors, and then apply filters to conditionally show each version. For example, you could use JavaScript to randomly assign visitors to either the ‘A’ or ‘B’ version of the page. 

You can also use GTM’s built-in variables to create the filters for each version. For example, you could use the ‘Page Path’ variable to filter for visitors on a specific page. Or you could use the ‘Referrer’ variable to filter for visitors from particular sources.

Once you’ve finished creating your A/B test tag, you can then set up triggers to determine when the tag should fire. GTM has a variety of built-in triggers, such as page view, click, form submission, etc. You can also create your own custom triggers if you want to be more specific. 

Once the tag is created and the triggers are set, you can then click the ‘Submit’ button to save your changes. After this, your A/B test will be up and running. The only thing you won’t have is the custom user interface that let you click and edit to make changes to the page, however that was only useful for simple tests anyway.

Test Implementation

0:01 All right. How to ab test when you can't use Google Optimize. So this rumors right now that Google Optimize is going away.
0:09 And that sucks because one of the cool things about it was it was free, easy to use and it let you do interesting stuff.
0:16 The webpages, without knowing how to code luckily you only need to know a very little amount of code in order to replicate it.
0:24 So this is maybe even a good idea to do in general. I found with these ab testing tools, there's like a flicker.
0:32 So when you land on the page, it's like not fast enough. Cause that's to load all of its crazy library, and then it makes the changes.
0:38 So I like to do this anyway. So say you wanted to change this, right? So one of the cool things you could do in Google Optimize and then optimize them for it.
0:49 So you could like click on it in the tool and then change it to whatever you want. You know, growth hackers, whatever, and it would change.
0:58 And then you'd go, great, this is what I want to test. And you'd hit the button. Well, you know, that's not possible anymore.
1:04 And obviously if you do this in the browser and you refresh the page then you know nothing's gonna happen because it's just gonna reload.
1:13 But you can still kind of plan your tests out this way. So if you say you wanted to change the button color you inspect then you can see that this has an idea of intent indicator.
1:26 You can actually click on it and say copy selector, that's useful. And then you can, you know, again, you only need to know a little bit of code if you just type in document and then call it like query sector.
1:47 And then piece that in. Then you, oh, did I do something wrong? Oh, query. Oh yeah, you didn't need this section.
2:00 Oh, here you go. Then you get your button. And you know, here it is. So if you want to kind of change that button, you can actually just make changes directly here.
2:10 So you can say like, what's this, you know, style you background color, color to be like green. Why not? It looks terrible, but there you go.
2:26 If that's what you wanted to do you could also change it and say, I want the text to be free access.
2:34 And maybe that's the ab test that you, you know, your heart desires you wanna run. So stupid example, but like, that's what you want to do, right?
2:42 So how do we actually make this persist? Well, the good thing is we have Google Tag Manager and Google Tag Manager can help us make changes in JavaScripts on our website without needing to deploy anything to our website.
2:55 We don't need to bother developers to do this, and we probably should tell them, right? It's not gonna bother them to tell them that you're changing something.
3:03 It'll bother them a lot actually, if you change things and you don't tell them. So just be careful here. Cool.
3:10 So how do we make this actually run on the website? Well, we can go to tags and we can set up you know some code.
3:22 So you know, you can actually don't even need all this stuff, right? Here's some code I made earlier. You know I, when I'm testing the stuff, I, I like to just go have an alert, which is like, it's loaded, right?
3:37 So you can just see that whether it worked or not pretty easily. So here's the tag, right? And don't worry about the triggers and stuff right now.
3:47 But if, if you wanted to, you can just like save this and then when you hit preview, you go through to your website.
3:56 If you load that page, then when that loads should say, yeah, look, it's loaded, right? And then you know, we've made this change here, the green button.
4:09 Cool. So that's not an AB test, so that's just making a change. But how do we make an AB test?
4:14 So if you look in here, I have the version B, right? What does that mean? So I'm only triggering it based on the version.
4:23 So if we look at this, it says on page views, but not all page views, just some page views. And the only one I find on the version when the page views the version matches, this is using regular expressions.
4:35 Don't worry too much about that slot. And then either slot, 6, 7, 8, 9, 10. So if you wanted to do this, you could do something similar.
4:45 You set this up. Now how does it know what this version is, right? Where does that come from? Well that is is just coming from if we go to variables and if we go down to custom variables we made a little custom variable for this.
5:11 So let's just look at here, yeah, version. And this is the code that we use to do this. So we have to randomize the visitors for an AB test, and that's what this part is doing.
5:25 It's giving you a random number times 10 slots. So that's gonna give us, you know, slots zero through nine. And that will assign a random number on every page, you to everyone.
5:38 But then we're not gonna get the same number again. So if I visit another page or if I refresh page, I'm gonna get something different.
5:44 So how do I persist it? Well, that's what the local storage thing is. What that's doing is, is first looking in local storage, which is just in the browser, you know, is there a version for this person and it's you know, gonna get the version and then pass that along.
6:01 And if there isn't a version, then it's gonna create one and then store it in local storage for me. So if we go back to this page, we right click inspect and then we go through the application.
6:15 We can see we are in slot nine. Now what if I change this to slot two? Lemme just refresh the page.
6:24 Ah, not in the test. Cool, let's inspect again and let's change it back to slot nine and then refresh, and then refresh.
6:43 Mess up. Slot nine. Yeah, I think I broke it. Oh, yeah, here we go. <laugh>. So it opened up a different page.
7:00 Yeah. So now I am in the experiment, right? Cool. So that should give you a really good idea of how this works.
7:09 It just randomizes the users, stores it in local storage, and then if they're in slots, 6, 7, 8, 9, 10 then you know, you can show them a version.
7:20 And and that's really all you have to do. You can obviously run as many experience as you want, like you could you know instead of having this trigger for version B, right?
7:32 You can actually have you know, a multiple triggers. You can have version A, B, C, D, E, F, G up to 10, you know and and, and then you could run multiple different experiments or like multiple different versions.
7:46 It's really up to you. But hopefully this gives you a good mental model for how that works. The other thing to think about is in terms of tracking you probably want to add, like, you know, add something to your, like if you're doing GA for I am you know, then you probably want to add something.
8:12 I think, oh yeah, no, it's not here. It's user properties, right? And user property. I'm just gonna make this version oh, yeah, sorry, I got this wrong to, oh, what did I do?
8:32 That should be version, sorry. So that's gonna input the value of the version for that user. And then I'm just gonna call this version.
8:44 So it's a property name. And if you do that now when you're in Google Analytics four, you be able to filter down to those users based on their version and then report On it afterwards.
8:56 So you know, this is a simple way to do that in any analytics package. So you know, there's always like a user level property you can set.
9:05 Cool. And that's how you reporting your OB test now. So you can just filter down and say like, people who got to that page and then what version were they in, and then whether they did, you know, any conversions afterwards.
9:16 Cool. Hopefully that was helpful.

Multiple Choice: What does the tutorial describe as a free alternative to Google Optimize for AB testing? 

How can you track the results of an AB test in Google Analytics?

What does the tutorial suggest you do to make sure the changes persist?

What is the purpose of using a random number times 10 slots in the tutorial?

I’m hearing rumors that Google Optimize is being deprecated (classic Google). If that happens it looks like over half a million websites are going to be without a solution that doesn’t cost $$$ enterprise money. 

5 Retention Courses

Courses - It's time to grow👇

Learn the skills to be a Data-Driven Marketer