Your CEO wants you to send a discount code out to your best customers, and any that you think could be upsold or might be slipping away. How do you do the analysis to categorize your customers?

Hey, we need to do something to boost the numbers for the end of the quarter.
I was thinking we could run a CRM campaign to offer a discount to our best customers.
It also makes sense to target anyone who looks like they could be upsold, and those that bought a lot in the past but not recently.
Can you do some analysis of the data and send the customer IDs to IT?
They can schedule the email for sending

This is a simple and intuitive technique for categorizing your customers based on their past purchase behavior. RFM stands for Recency, Frequency, and Monetary value. To conduct this analysis you calculate how many days since they last purchased from you (recency), how many orders or transactions they've completed total (frequency) and the total amount that customers have spent with you (monetary value). Then you categorize customers based on the quartile they fall into in each category. For example an RFM class of 1-1-1 means the customer is in the top 25% of each category: a top customer. A class of 2-1-1 might indicate that this was one of your best customers, but they haven't purchased recently, because they're only in the 2nd quartile for recency.
This information is invaluable for targeting in CRM campaigns. You can use these labels to find your best customers, identify people who used to spend a lot but might be slipping away, and customers who bought recently but haven't made many repeat purchases. Then you can conduct a VIP, winback or upsell campaign depending on what bucket they fall into.

RFM Analysis

Make sure your dates are properly formatted in Excel or GSheets or you're likely to get your Recency number wrong. The date format YYYY-MM-DD is preferred because it's immune to differences between countries.

Hey, I'm going to show you how to do an RFM analysis. So this is the data that we have a transaction ID. Stock code is the different products as just two products, the replacement cover. And there's also. This smart bit which is further down so less of those and the higher price point than the replacement covers.
We also have a customer ID, which is the main thing we're going to be joining in order to share the recency, the frequency and the monetary value of each customer. Okay. So first let's work out the revenue. This is an easy one unit price, multiply it by the conceit of the. So just going to allow that to dropdown and you can see this work out there and then let's get the recency it's gonna shoot both days ago.
And to get days ago, we just wanted to take today and then tracked the invoice date. Okay. So. This was in 2020 and you know, today it's 2022. So yeah, it makes sense is 458 days ago. And you can see that some of these were like 85 basic year. Which makes sense. Cool. Okay. One thing to watch out is if your date is in a weird format you can sometimes are ready to issue.
All right. So now we need to create a pivot table. I would, sorry. I'm just going to press command a or control a a, and then we're going to go to insert and then pivot table and put that in a new sheet. And what we need to do is pivot on the customer ID. So that's the customer ID and that's going to aggregate all of the transactions.
You can just add the Invoice or a transaction ID here. And then change this to account a yeah, I'm sorry. Yeah. And then you can just make this descending by count. You can see that there's one customer that bought 46 6 of course. So let's, so that is the frequency. And then we need to add.
The monetary value. So that's just the revenue that's pretty straightforward. And you can see that this how much revenue this customer has made. And then we also want to add the recency. So we're going to take days ago.
Okay. We can see the days ago is the sum and we're going to change it to a, it's just the mid. So this will find the smallest days ago that, that customer has there we go. Okay. So now that we have that we want to sort these into quantiles and that means basically splitting into groups. You know, 25% of customers are in group one 25% in group two and so on.
And the reason we did that is so we can get a score for, you know, how how high this customer up is up in the, in the, in the pecking order for these three different values. So let's do recency frequency. And we're just going to put them here as well.
All right. So what we want to do and actually let me just rearrange these. So just say they're in the right oldest or recency of days ago, and then frequency is transactions and then. Revenue. And I'm just going to show you one of these and I'll leave the rest for you as an exercise. So we need to sort into courthouse and that I'm just going to say these are the three both tiles.
So let's do recency. So we just need the core tile. And that takes the data and then the course on numbers. So that if we, you know, press to troll shift down and then I'm just gonna make sure I didn't get the total and that I'm just going to lock that in place for pressing a four or you put dollar signs in front.
And then of course I'll just report. And what this is telling me is the the lowest the lowest kind of recency. Like if you, if you want to be in the the lowest 25% of recency your recency needs to be below 260. If we change this to three a, then that then anything above 393 would, would pay you in the worst 25% of recency.
Cool. So so that ones that actually, if you just put zero head and that's the men, so it's the lowest value. We can. Also bring out the second core tile, they should just be medium basically. And then copy copied across, and then we want them to know, we just need a quick function. It's a couple of nests.
So if the basic go is less than this then Oh, sorry. Get rid of that. Yeah, if there's is less than that value the low school. Then and we want to say it's in queue one. I'm just doing that as a string because we're not going to have the scores up. We're just going to have to, to show you 1, 1, 1 0 1, 2, 1.
Okay. So and if it's not a low in that value, then we need to make another statement. So now we want to say if it's if that is. Lower than this, then it's going to be in cool tile too. And then if it's if that another statement. That is lower than quotes three. You know, then it's going to be three.
And then if not, it's going to be pulled off for, because you know, then it means it's above that number. And then sort of close that statement close the second half statement close the first statement and that should work. Okay. So I'm just going to double click. Actually, let me just, before I double click this, to make sure this is in in absolute reference I say press at four or put dollar signs in front of it.
Cool. So just double click or if that's not working, you can kind of click copy and then paste all the way down.
Cool. So everything is broken into . Now I'll let you do the rest. You know, you do the same thing except in this case you you know, think about it, frequency or recency is a, is a low, is a good thing. With frequency high is a good thing. So in this case, you know, we're selecting seed. And then you can see.
You know, rather than quarter one, we actually puts out three here. So if you want to be able to put 25% of frequency, you have to have bought at least three things. Cool. So you know, this is just the same, but reversed. And then same thing with here. These signs will be reversed when you do frequency and then same for monetary.
So I'll leave you to do that exercise. Once you have. You know, the recency frequency and monetary then so say for example, this person is one-on-one. Then the RFM class would just be that I present that, that. It's going to be class 1, 1, 1, that'd be your best customer. Or you know, if it's you know, someone who has bought recently, I mean, it hasn't been recently, but it used to be your best customer.
I, then that might be like a 2, 1, 1. So that's how you should think about the RFM class. Okay. Good luck.

What RFM class would someone fall into if they had bought recently and spent a lot of money, but haven't made many transactions?

If you send the discount to everyone in the 'Best Customers', 'Slipping Away' and 'Upsell Oppty', segments, what percentage of all customers will get an email?

If an RFM class of 121 is a customer who has bought recently and paid a lot of money, but hasn't made many purchases, how would you classify a customer that bought recently and often, but only for small ticket items? 

Management Consultant and Author

Without big data analytics, companies are blind and deaf, wandering out onto the web like deer on a freeway

RE: Campaign results
Thanks so much for that analysis, the campaign looks like an early success.
We got good purchase volume from those upsell opportunities and won back 15% of those customers that were slipping away.
We even got a few thankyou emails from our top buyers!
Best,

Who deserves a discount?

You’d like to reward your best customers… but who are they? Do we rank the highest spenders? Or those with the most repeat purchases? What if they haven’t bought from us recently? We need an RFM analysis.

Who are our best customers?

Your boss wants to know how much a customer is worth to you. Once you know that you can use it to justify the extra budget you asked for.

I know we’re getting outcompeted in the ad auction

But I’m not comfortable letting you go over our CPA target unless we know how long it’ll take to pay the extra back.

As far as I’m aware most of the revenue comes from first orders

So I’m open to seeing some analysis, but unless it’s more than 10% extra revenue coming from repeat purchases in the first few weeks…

I’m not sure we can revisit the budget question

Understanding customer lifetime value (LTV) is essential for any business that wants to be successful in a competitive advertising environment. Calculating LTV helps you understand how much you can afford to pay for a new customer, and how much money you need to generate in order to make a profit.

The first step in calculating LTV is to determine the average customer lifetime, which is the total amount of time a customer spends buying your products or services. To do this, you will need to look at past sales data and analyze the amount of time it takes customers to make a repeat purchase. This data can provide valuable insights into the typical customer purchase cycle.

Once you have the customer lifetime, you can then calculate the total revenue per customer. This is done by multiplying the average customer lifetime by the average purchase amount and the number of repeat purchases. This is an important figure, as it will give you a better understanding of how much money each customer is worth over the course of their lifetime.

You can then use this figure to calculate the customer lifetime value. To do this, you will need to divide the total revenue per customer by the customer acquisition cost. This will give you the dollar value of each customer.

Once you have the customer lifetime value, you can use it to better understand how much money you can afford to pay for a new customer. You can also compare this figure to your customer acquisition costs to determine whether your marketing strategy is effective. If you find that your customer acquisition costs are too high, you can adjust your marketing strategy accordingly.

By understanding customer lifetime value, you can ensure that you are spending your advertising dollars wisely and optimizing your efforts to bring in more customers. Additionally, it can help you better understand the value of your existing customer base and the impact of any changes you make to your marketing or product offerings.

LTV Calculation

00:01 Hey, let's figure out how to calculate lifetime value. So there's obviously ways to do this in code and you know, recommend that you learn SQL or Python to do that if you are doing something in a backend database.
00:15 But typically that's not where you start, right? Like you figure this out in the spreadsheet, first of all because you know, usually you know, before you have a data science team, you need to kind of figure out like what's a customer worth to you.
00:29 And you know, you can do that pretty simply with just some, you know, raw customer data in a spreadsheet and a pivot table and some calculations.
00:39 So that's where we're gonna work through today. So I mean, the, the benefit of this, by the way, is you don't know.
00:46 You need to know how to code, and you don't need to know any math <laugh>, which is a huge help.
00:50 All right, so what we have here is our raw data, and this is what you need to be able to get out of your database or export from Shopify wherever you have this.
00:59 So what we have here is the order id. This is just a unique ID for every order of a customer id.
01:06 And then again, that's a you know, unique ID for each customer. And then we have the order timestamp and the order value.
01:14 So this is the raw data. This is, you know, we just select all of this. So this is like what we actually got from the database.
01:23 Everything else we've figured out. And I'll show you how to do that. So, first thing we need to do is we needed to sort by timestamp.
01:32 So I just said, you know, sort A to Z and that way it would be, you know, the first timestamp first.
01:39 And I also like to make sure, just cuz it can run into some issues with formatting that you do like year, year, year, year dash month, month dash day, day and then the time.
01:51 So hour, hour and then call, and then minute minute. This is the superior date format, <laugh>, because it works across every country.
02:00 You know, and it's, you know, impossible to get confused about or get it wrong. Also you know, you can sort it, if it's in the format, you can sort it, whether it's text or you know, or, or a date format.
02:12 So it actually works pretty nicely because the largest thing is first, right? The year, then the month and the day, okay?
02:21 So that's the date you need. And what do you need to do with it? So first of all, we want to get the first order index.
02:29 This is basically looking through the, the database and finding what was the first time that customer ordered. And this is something you can do in sql, right?
02:37 If you're doing this in an actual database. But we can do it in Google Sheets as well. And we can't just look actually for, you know, the first customer ID because it doesn't really work on a string unfortunately, but we can find the first index because we've sorted everything by timestamp.
02:54 We can use this function here. So we're saying this is a filter wrapped inside a min formula. So first let's look at the filter formula.
03:03 What does that do? So filter basically you give it a range and then you set a condition and it will, you know, spit out that range.
03:12 So normally it takes up more than one cell actually.  And you can see here, this is the range we're giving it, giving you the index.
03:21 And then we're saying filter just for any time that this range. So column C, which is where the customer idea is equals c2, right?
03:34 And because we've put an absolute reference here, it's like F four. Then, you know, as we scroll down here so like, look at this one it, you know, there's C5 references going down.
03:47 It was, you know, C2 before now C5 that's matching with the row but the, the full kind of array we're looking at doesn't change.
03:57 And that's really important to get right? So what that's gonna give you is essentially a list of indexes where that customer ID is true.
04:06 And we can actually see that if we wanted to. So let's just, you know, copy this customer ID and let's just see filter I'm gonna say filter the range customer d I'm gonna say oh, sorry, we need to actually, we wanted to get the index for this.
04:44 So that's what we want to return. And then we want to only get that index if the orders because this, well, the customer ID equals this, We go, okay, so there we go.
05:11 We spat out the range. So that customer ID appears at index three and index 7,933. Okay, cool. And you might be wondering what the index is, is literal tr just an order, right?
05:23 So we put one for the first thing and then plus one, plus one plus one all the way down to the bottom.
05:29 Cool. Okay. So so that's what the filter normally does. And then what we're doing here is we've got the min function here, right?
05:38 So if you look at min, all that does is it just takes some values and it gives you the smallest one, right?
05:45 So it's saying index three is the smallest one. Cool. You could also do like the latest order if you wanted.
05:51 So that could be max, And then it would tell you there the, the, the latest order for this is 7,933.
06:02 Cool. All right. So that is how that function works, and that's the most important function. So if you understand that that's, you know, you're golden because once you have that, then the rest is like relatively easy.
06:14 So this is just a v lookup. So saying you know, find the first order index, right the thing we just found in this array of the index and the order ID, and then just return the order ID when you find it.
06:31 So all that's gonna do is, you know in say like, you know, this is a good example here. So see this, like this this order ID doesn't match, right?
06:48 Because this is actually looking, it's, it's found at 70 90 which was earlier on, that was in index 85.  So that customer, I had another purchase here, and that's the 70 90, that's what we're pulling back.
07:02 Cool. So VLO cup again is just, you know, we're saying scroll down in this part of the spreadsheet, and then when you find the index of one then return the order ID for that index.
07:18 Cool. So what that's getting us is the first order id, so it just relates to this. Then we can also get the first order timestamp exactly the same thing, except we've just got a bigger lookup and it's just saying just give me when for that order, ID just give me the timestamp, right?
07:36 So that's the same thing essentially. Then then once we have the timestamp, we can work out how many weeks it's been since the first order.
07:45 So if you subtract a date from date, you get the number of days that it is between them. So we're taking this order timestamp, and then we're subtracting the first order timestamp.
07:57 So it should be zero, basically, if they match, which they do here. And then and then we're dividing that by seven days.
08:05 So it's gonna give us how many weeks. We're just gonna round that up so it's easier and it's just, you know, there's no decimal places we can see here that like this this order came in a week after the first purchase, right?
08:20 This one came in three weeks after the first purchase. So that's really the key to getting the lifetime value side of the equation.
08:30 Cuz we can't just add up all the, all the orders per, you know, per customer, that wouldn't really, really be that useful.
08:36 Because, you know, some customers bought yesterday and then some customers bought a last year. So if we just summed up all the orders and we, you know, the revenue per customer, that wouldn't really cut it.
08:48 What we need to do is kind of, if we start all customers at the beginning at zero then how much revenue comes in every week of their lifetime.
08:57 And that's what we're gonna see here. Cool. This last thing here is really just total customer orders. So it's just telling us how many orders that customer had in total.
09:07 That's just so we can kind of see like this customer, it's just a account if, right? So this customer ID is shown have three times in this, in this database.
09:16 Cool. All right, now let's look at the pivot table. So now that we have all that, that was all the prep, now we have the pivot table and we can see if we just put in the rows here, we can see how many weeks since the first order.
09:33 So that's in the here. And we have the sum of the total order value brought into the values and then the count of customer IDs.
09:42 Cool. So lemme explain what that means. It means that for all of the orders that happened in the first week of the first order, so week zero there were this many orders, right?
09:56 So most of the orders came in there. And also most of the customers you know all the customers, right? Had a purchase in week zero, essentially, right?
10:08 Only, so seven, if we had 7,000 customers, only 400 made another purchase in week one. And then we, it kind of drops off from there.
10:14 265 made another purchase in week two, and so on and so forth. And you can see there's kind of a long tail here.
10:20 So that's what this is showing. This is the percentage of revenue. So this is how much revenue is this divided by the total site.
10:29 And what we can see here is that the share of lifetime revenue drops off a cliff, right? We get 85% of the revenue in that first week, and then by the second week it's another 4% and then 2% and so on.
10:44 All right? So now that this is kind of normalized, we have, you know, all of our customers based on starting week then we can do our calculations.
10:52 So first we get the initial order value. That order value is just the order value divided by the number of customers.
11:00 And that is 131. So that's our average order value for week zero. Then we can set a cutoff. So I've set four weeks here.
11:09 So this is gonna look at the, just the first four weeks, and it's gonna tell you that as well as that first 85% of revenue, you're also gonna get this extra piece, right?
11:20 You're gonna get like another 9% in revenue here. And what we're doing here is we're just saying expected incremental. If we know the initial order value, how much incremental revenue do we expect to get based on that cutoff?
11:34 So this is just looking at the cutoff and looking at kind of summing up anything below the cutoff. So you know, here it's like 9%, but it's divided by 85%.
11:47 So so we is, is like 10% more revenue than what we got in that first order, if that makes sense.
11:54 Cool. And then and then, you know, once we know how much more revenue to expect from that user in the first four weeks, then we just work out the lifetime value you know, which is the initial value plus expected incremental value, and we could change the cutoff.
12:07 So we could say eight weeks but in this case doesn't really change that much because, you know, by, you know, like, you know, weeks four to eight, it's only like half a percent extra per, per week.
12:18 So it's not, not a huge amount, right? Did also say, you know, 52 weeks, right? If we wanted to and that gives us a pretty, you know, much bigger incremental.
12:29 But you gotta think about payback period, right? Like if we you know, can we afford to front the cash now and pay 150 bucks for a customer when we're only getting 130 bucks straight away?
12:42 And that extra 20 bucks is gonna come, you know, over a whole year, right? Like, it's not really worth it.
12:48 So I'm just gonna say in this case, let's say first two weeks or, you know, first four weeks be relatively conservative, you know, we can pay 'em back in a month, and that's pretty good.
12:58 And now I know my lifetime value based on that cutoff. And that means when I see that I made $130 in revenue from a purchase, I can really treat that as $144 in revenue.
13:11 And that extra 10% incremental is gonna help me be more competitive when I'm running advertising. It's going to, I, I'm gonna be able to push my bids like 10% higher essentially.
13:23 And and, and, and, and like, you know, eat up a bit more inventory, get a bit more volume versus my competitors.
13:29 So that is why you do this calculation. Obviously this is like a typical e-commerce case where vast majority of the revenue just comes in the first purchase and most people don't buy again.
13:41 But you know, for other businesses, like subscription businesses, it could be completely all the way around like maybe 85% of the life of, of the, of the lifetime revenue comes after the first week, right?
13:54 All right. So that's you know, that's pretty much everything. I think one other thing is to think about, you know, we used weeks here, but you could use days.
14:02 We could also use months if it's a, a subscription service. So just think about the, the date format that you use here is all, all right.
14:10 Hopefully that was helpful.

What is the purpose of calculating lifetime value?

What share of lifetime revenue occurs in the second week (i.e. week 1)

Did you know it costs $50 per click to bid on “printer” keywords on Google? Most manufacturers lose money on the first sale, because they know you’ll buy the ink.

How much is a customer worth?

Your web application has a stable number of users, you've finished the core product and want to add on experimental features to improve customer retention.

Your boss sends you a message: this is probably about adding extra functionality to the react application.

Hey, have you looked at building some simple user recommendations yet?
It would be great if our app could recommend relevant products that have been seen together.
Just like how Amazon has that 'People who viewed this product have also viewed these products' section?

Remember to make a copy of the google colab file so that you can follow along!

All right. So we've been tasked with putting some Amazon style recommendations. So you'll see these on Amazon, for example, like products related to this item, we'll be taking an approach where we'll be partitioning the data by a user ID. And this is quite useful for if you want to generate carousels. So for example, if there's a people who viewed this course or people who viewed this product also viewed these courses or these products Before we get into any of the code.
I think it's really important that we describe this process of step by step. And I've done some pseudo code for us to go through this. So we'll be first importing our  modules and some fake dummy data that you'll have access to. Additionally, you'll be able to extract. All of the courses or the products or whatever your item is from the database, we'll then need to extract how users are viewing those items or courses from the database, wrangling the dates to be in a specific format inside of a Panda's data frame.
Once we then have users against courses or users against products will then generate all of those combinations. Were those Horlicks have been seen in combination in part in combination to the user ID we'll also then create the recommendations and then we'll also provide a graceful fallback option just in case a specific course or a specific product is new.
And it hasn't been in combination with other courses or products. And then just a little bit on the fact that we'll also need to be able to store these recommendations inside of the day. So running these two bits of code where we're download some dummy data, because I'm conscious that some of you might not have a production database, so you can connect to we've taken the Liberty of producing some dummy data for you.
And so what you're going to need to do is in your script, you're going to need, when you've connected and extracted your data from your database, you'll need to have it in a format where you have a PIF and list of your product IDs or your course ID. And you'll also need the format like this. So you can see you've got the CR the product ID and some extra properties inside of each individual pro product as, as a, as a pipe in dictionary.
So this data doc, Jason, if we have a look at this we'll load in some of that course as data that looks like this format, for example. So you can see here, you've got some. Quite a large dictionary structure now, but if we go scroll all the way to the top, you can see this as the course ID in our case and some extra information.
And it's exactly this format of a course or a product ID and then some extra information nested in that. So, so that's that and then once you've extracted the, the product IDs and you know, as a list and you've also got the product IDs in a, in a, in a dictionary for. What you'll then need to do is extract the users in, in relation to how they're accessing your courses or your products.

So if you, for example, this will also be a section you would need. Right. And it's completely bespoke. But if you were using Firebase, then maybe you would be storing how users going through your product and interacting with either courses or products or items. And you can see this as an example of like, for example, in our scenario, we have a real-time database, which just connects to the users and we essentially can get all that data in in using hyphen.
There's some sample data provided for you. And essentially the. That we're going to need to wrangle the data in is going to be like this, where we have this core state of debt, where we have item IDs and a list of items, and we have user IDs and whether they viewed certain items. So if we're having a look at this in a very simplistic way, we have the item ID, which could be your course idea.
It could be your product ID. It could be, you know, any sort of item. And then we have the user. And what's important about this, as you can see, like certain users have viewed multiple license and you can do more complicated recommendations if you add in a rating, a rating column as well. For this example, we wanted to build a recommendation engine that just purely uses the co-occurrence or the combinations of courses based upon a user ID.
So we have this form. And you need to get the data into this format. Right. So imagine you've import, you've imported your course ideas as a list up here. And you've also got your course ideas as a dictionary. Yeah. And then what we want to then do as well is we've got our user data in how, wherever they are you know, viewing your courses, et cetera, on a user level of beta on a user basis.
So if we have a look at this one for. You'll see that, you know, our user user ID is the first key. So if we scroll right up to the top on this one, you'll see. So for example, you've got user ID as the top. If I just grow, I'm just going to scroll up quickly and get to the top of this is quite long as you can imagine.
Cause there's quite a lot of user data that we have here, but essentially what you're looking to do is say for each and every individual user, which courses did they view? So that's how you need to wrangle the debate, the data, right? Whether it's you or a developer, you need to basically say we have some users here like this user, for example, and these are the courses that they've they've specifically viewed.
And so that's kind of how you should be thinking about this is we're godless of the shape of your data. You need. The, the actual like courses and pro or products you need some product product dictionary. If you need to insert any more information and then going onto this step, we also need to have how the users are relating to that.
And basically going in through the data, whatever format that's in, and finally creating a structure like this upon this data stream starts like this. So in our case, what we needed to do is we have a dictionary of users. So we're looping over all of these users. And then we're then looking to see does that specific user have a course dictionary inside that course key.
And if they do what we're then trying to do is create a list of courses from that. And then we're checking to see whether those lists of courses exists in our current set of product IDs. And if we do have any courses that are in our product ID, Or our course IDs, then we're basically saying for every course that we found there, add that as a pair, right?
So we're essentially going loop over all the users for each user find all the courses or in your case, it might be all the products that I use are as viewed. And if there is products that are user is viewed. Then whenever you've got five products loop over there, if there are more than one product loop over those five products as a pair of item ID and user IDs.
And then we ended up with this data structure that looks a bit, a little bit like this, right, where we finally got this data structure of items. So if I ran this code. You'll see this changes. So now we have slightly larger data structure where we've got all of the individual item IDs against user IDs.
And this is what it looks like inside of a data frame. So you can see there's 214 rows by two columns. We have every item ID against the specific user ID. So we now understand. Which courses users as a whole have viewed in terms of, or which products users have viewed. And this is quite useful because then we can apply a combinations function to this.
And we can say this section here is essentially. Group the data by the user IDs. We're essentially partitioning the data by the user ID. And what's quite nice about this is we can then figure out aggregated metrics. So for every user count how many times they have an item idea. For example, when you can see this user has two courses that they viewed but we could take this one step further and do a dot apply function.
And what we've done here is say group the data by the user ID. And then after you've grouped the dates by the user ID, we want to take the item ID and apply a function to it. And this function you'll see when we run that actually produces the data back in a user ID with the combinations of those functions.
So what we're looking at here is whichever courses have been viewed more often. In combinations at a user level. So these two courses here, these two course IDs, these items. Viewed a lot more across all of the users than other than other courses. And therefore we probably would want to include those together in a recommendations algorithm for if you're looking at this course, you would also like to be able to look at these courses.
So that's kind of what this group by with the dot apply combined or value counts is doing underneath the hood and to, to put it in a bit more of a simplistic matter and look. You basically end up with two paired combinations of courses and how often those courses are found together. So we can then write something a little bit to say, well, now we've got all these combinations.
We need to look at all of the courses that we have or all the products that we have and see, okay, if we have these products I want to then set up a new dictionary. And for, in our case, it's people also viewed this course, but it could be people also viewed this product for yourself, looping over those product IDs.
And then what we're doing is we're looping over all those product IDs against the data itself. And with. Getting out the individual combination. Of the, so these individual combinations, so we're getting out this first one and that second one. And we're then saying, if we, if we don't have at least three combinations inside of that individual pro inside that individual product, or course then keep going and do this else.
We just want to break out of this four loop, right. And go onto the next product. So if the course has already got three recommends then we're basically going to be breaking out of that loop elsewhere. Look if the combination is equal to that, then we also look and append the other combination and do the reverse.
If we'd have the combination too, and else we don't match, and there might be some products that you don't find a match for, and that's all we're going to handle later. And you can see here that when we do this we then end up with a slightly. Dictionary. So if we look at this, people also viewed this course, we now have every product and the recommended products, the recommended three products to go with that.
And so you can imagine on a product page, you could do this. So you could tune this barometer to be three or five, and then you could have a carousel of the. Main products that are, are, that are often seen with that product after we've partitioned data by user. So that's all great. And then going on from here, we're going well.
Okay. Well, we've got a hundred products. Maybe we've got 20 new products that just came out yesterday and all of our users haven't seen these new products with other products. So what we could do is take the original data from. Where we had, if you remember going back to this this data frame of the item ID against the user ID.
And what we would then like to do is do a value counts on this. And you can see if by looking in doing a value counts on this, what we can get is we can get an understanding as to what products have been viewed. The most in terms of our users. And you could also use Google analytics for data using Bitcoin for this.
So that's also something you might look to implement, but essentially getting some sort of understanding as to what courses have been viewed the most often. And then we could use these for courses that fell in this else block here. So when we didn't find a match what we would like to do is then loop back over that dictionary.
So we'll extract the popular courses columns and do so then we have this data frame here that looks a little bit like. And then you can see you. So this ends up looking like we bought the course and we've gotten the number of times it was viewed. Cool. So we can then loop over that for every course or product that you have.
If the length of that in the dictionary is not equal to three, then we know that we haven't found any courses and therefore we could look and go, okay, we'll get the popular course ID, get the data loop over that, this new data frame that we've made. And then say, if the course isn't equal to itself, then we're going to append that.
And, and then what we then have here is pause for if there are greater than three results or they're equal to three results, then we break out, which basically means you know, we're just going to be taking the top three most popular courses every time. And appending that into into that individual dictionary.
And so then what you end up with is a final dictionary that you could use called people who also viewed this course, and you could change the name of this, but the essentially the way that you can think about this is every single key in here is the product. And the array is the most valuable courses that have been seen with that course.
So the first one is seen more than the second one. And the second one is seen more than the first. And so you could then save this into your backend. So the way that we do that is we have reference called in our database called user course view recommendations, and we set up. And then on the client's side, you could then reference that.
So you can see where you're using Firebase database, and then we have referencing that we can get those recommendations. So this is kind of an end to end way that you could easily implement recommendations. And I think the other important thing for us to connect. The deployment of this. So I just thought I'd touch on the deployments.
So you'll have access to all their source code. And the only thing that you're going to need to change are the sections where the credit will be unique. So this second. This section and also the, the uploading as well back into your database, there's three sections that your developer or yourself will need to write which accustom, but you've got an example of how that's kind of currently implemented in the system.
And then if we're looking at deployment method, you could, for example, have a chronic. That runs every day or every week. And then basically what they can do is it could be hitting a cloud function that runs this place and code inside of this. And that that function would, would do something like this.
So you could create user course recommendations. And you could also separate the fit, the fullback recommendations. If you wanted on a separate. Or you could include that all in in one single pipe in script and that's just running a cloud function. And so every day we could go in and hit this cloud function at three in the morning and generate fresh recommendations for any products that were added in from the night before.
So hopefully this gives you an indication into how you could start to add additional recommendations functionality into your app.

Recommendation Engine

What data do you need to make 'users also viewed' recommendations?

What does 'count of occurrence' mean for section 5 "5. Generate combinations of all courses/products found together, partitioned by user id:"?

Why is it a good idea to have a fall back for recommendations?

How can we generate useful recommendations?

Learn how to easily create 'People who viewed this product, also viewed these products' recommendations.

How can we create Amazon style recommendations?

 Bank of Yu Offices – Monthly Product Meeting

After a successful launch and satisfying early results for user acquisition, your Head of Growth is turning his attention to retention. They want to figure out what the ‘aha’ moment is for your product, similar to Facebook’s famous insight that if they get a user to add 7 friends in 10 days they’re likely to stick around.

If we’re going to focus on retention, we need our ‘north star’ metric
You know, like how Facebook had “7 friends in 10 days”
The ‘aha moment’ where the user has completed enough actions to remain sticky
If we know what correlates with retention we have something to aim for
If we can focus on moving the needle on that metric, growth should follow
Can you do some analysis to find that ‘aha’ moment for us?

How many actions does it take before our users say ‘aha’? If they don’t use your product enough they might not get it, and if you don’t get them to that point in the first few weeks they might never come back. However if you can optimize your product relentlessly towards a ‘north star’ metric that’s correlated with retention, your product will get stickier, and growth will follow. Facebook’s growth team famously optimized to ‘7 friends in 10 days’. They knew that if they got a user to add at least 7 friends, they would get enough value out of the social network to stick around. The goal wasn’t just correlated with success but it was simple: there were thousands of things the team could work on, but with a north star metric in mind they could focus and prioritize only tasks that moved the needle on the metric. 

In order to choose an effective north star metric, you need to do a retention analysis. The ‘aha’ moment is in the intersection of two competing forces. The more actions someone takes the more likely they are to be retained, but also the more people will have been retained without taking that many actions. You need to find a tradeoff between the two. First you find out the share of people who were retained, that performed this many actions. Second you need to find out how many people are retained without performing this action. Bring those two together and you can find the sweet spot, whether it’s 6 friends, 7 friends, or 8 friends, or whatever the equivalent action in your product.

Retention Analysis

Hey, let's learn how to do aha moments, uh, with user retention data. Um, cool. So we're in a Google CoLab notebook.
It's like a Python notebook. Uh, you can think of it as a Google doc with code. Um, so it runs in the cloud and you don't need to set anything up on your computer, which is helpful.
You can share it like any Google doc. Um, so if you run a, if you just shift and enter, or if you could play on any of the, uh, the cells, and then it's going to run that code and output anything that's needed, um, I've already run this part here.
So this is the libraries, and then this is authentication. So we can pull data from the Google spreadsheet. Um, when you run it, uh, it's gonna pop up and ask you, uh, to, uh, kind of authenticate with your Google account.
So you just click on your email and if you're logged in, then, um, then it will connect it. Um, and that's really just to get this data here from the Google spreadsheet.
Um, and that is the data that we're going to use for this analysis. So this is the format you would need it in, uh, we need the user IDs, um, and then whether they were retained or not.
So in this case, uh, we use the date frame of one week, uh, if they're retained a week after they, uh, installed, uh, then we have a one and if there, if they weren't retained, then we have a zero.
Um, the, uh, this is, uh, one of the events that you could do within the app. It's a FinTech app that we can give loans to friends.
So we're going to evaluate the loans, uh, metric, but you could do this analysis for any of these other events that use this do as well.
Cool. So, um, if we run this, it's going to pull in the data, um, and we can see this just the first five rows of that data, just so we can check the, it pulled in correctly.
And if we run this query, uh, the data stored in this DFS data frame variable, um, and then we're just going to group that by how many people were retained.
Um, and then we can get the count of the user IDs. I, so this is essentially what we're going to do, but on a larger scale, um, but you can see here, they have 969 people that retained and then 2,726, that didn't cool.
So, uh, we're using this loans action. And if we run this here, uh, you can see it calculates a table with a zero for people who didn't retain one for people who did retain, um, and then how many loans they gave out.
So what we're looking for is how many loans does someone need to give out to make them very highly correlated with retaining within the app?
So we're trying to find, uh, something similar to Facebook's goal. Uh, they use for their growth team, which was, uh, they figured out if you can get a user to add seven friends in the first 10 days, they're very likely to retain, and we have a trade off to make as so first of all, um, you know, we wanted to find a users who are very likely to retain.
So, um, uh, so people, You know, people who did this action, whether it's say like, you know, five lines, uh, people that did this action were very likely to retain.
Uh, but also, um, that, uh, they, um, that there were a lot of people who did that action. So if loads of people were retained without getting that far without getting to full lens, it wouldn't be a very good measure.
You can see here that actually, um, you know, we only had 75 people who made full lens, so that might not be a good action.
So it, can it go, um, you know, zero loans wouldn't be a good action either because loads of people make zero loans and they're not likely to retain if they do.
Um, so, uh, in order to figure this out, we want to use data visualization. Um, so we're using a map plot, lib and seaborne, uh, to visualization libraries.
Um, and essentially what this is doing is just kind of setting the color codes and then, um, plotting, uh, two bar charts, one on top of each other.
So we can see here, um, that the retention rate, uh, which is, uh, what percentage of people who got to that level, um, retained, uh, is very, very high for people who made like four loans or above.
Um, it's in the middle for, uh, people made two or three lines. So it's like a 40% pretension, right? Um, and then for people who made zero one loans, it's much lower, it's less than 20%.
Um, so that kind of tells us what part of the story that the more loans you make, the more likely you are to retain, which is what we'd expect.
Um, and we can see a couple of reflection points here. It was, it was already telling us something. Uh, the next thing we want to look at though is just the raw numbers.
Uh, and that shows is another part of the story, which is that very few people make it to fall into or above.
Um, uh, so, uh, it wouldn't be an effective target, uh, to go that far, uh, in terms of our goal, because we would just wouldn't have enough data to optimize.
Um, it wouldn't be able to measure that much of an uplift on how many people got to four, uh, plus loans, uh, because, um, you know, very few people do so in the first week.
Cool. So, uh, let's, uh, run this, um, and, uh, what this is doing is working out a few more pieces of information that we're going to use in the next visualization, uh, specifically the share of retained as, so this is how many people who got to that, uh, level and retained, like what share of, of that, uh, um, what share of those people did they make up of all the people who retained meaning, um, you know, did most people who retain, uh, get to that level?
Uh, so that was one piece of what we're trying to look for. The other piece is, uh, how many people retained without that action.
So this is doing a cumulative sum of the retention metric. Um, and you can see here are retained without the action.
Um, so it's saying, um, you know, if we get to two loans, um, then, uh, yes, we had 283 people retain when they go to two lens, I, but 405 people retained without even making it to, to learn.
So they made zero one loan and you can see that 2, 4, 6 plus 1, 5 9 is four or five. Um, the reason we made this negative, I did, uh, you know, multiplied by minus one is just, sorry, could show it nicer in the visualization.
So it makes it on the left-hand side of the axes here. Um, so this visualizes, the problem that we're talking about perfectly.
So as we go down the stack, more people are likely to retain, uh, but, um, as we get down the stack, uh, most people who retained, uh, you know, had, had already been retained at that point.
Um, you know, uh, like if you look at number four, the pink bar here retained without the action is way, way bigger than even the people who made it to this level.
Um, uh, because, uh, you know, most of the people were were before, uh, for loans, they didn't make as many as full loans.
Cool. So, um, now how do we see this in percentages? Because that's gonna make it much easier to decide. Um, so we're just going to run this, but it's works.
Yeah, here we go. So this is, uh, you know, what we've been leading up to this is the money shot.
Um, and, uh, this is the visualization that you'd run around. You could skip a lot of this stuff. Um, you know, if, if you're just running this analysis yourself, um, you can actually, uh, I think here, just go run all.
Um, and then, uh, we'll run everything all the way down to this, if you want. Um, as you don't have to like worry too much about the previous stuff, uh, this is the most important thing.
So this is sharing everything really on, uh, that we are sharing here, but in percentage terms, so over 100% of people, um, you know, uh, the blue is tow people who made it to this level, um, the, uh, dark blue share of Chandra with purple.
Um, that's saying that say 20% of people who made it to one loan, uh, were retained. Um, and then the pink bar is, uh, people who, uh, the share of, um, of, uh, people who retained without taking this action.
So, um, for one loan, the pink bar is pretty low, uh, because, um, uh, you know, not many people who did zero loans, which is above this, uh, were retained, um, as, so that pink bar is pretty low, but then the pink bar girl is a lot just like it did here.
Um, the pink bar grows a lot to then being pretty much everyone by the time we get down to nine months.
Um, so what you're looking at here just to keep it really simple is you want, uh, you know, a huge share of retained.
In this case, we have a big purple buyer here around two loans, uh, relative to the pink bar. So we want like a bigger purple bar.
Then we have a pink bar, um, and we also want a bigger palpable, and we have the blue bar. So, uh, essentially like the bigger, the purple bar and the floor of the pink bar, uh, the, uh, better That is as a target.
And for me, it's pretty clear, like we could choose one loan has a pretty big purple bar and a pretty small pink bar.
Uh, but, uh, this learns, uh, you know, two learns has a much bigger pebble bar and not that much bigger of a peak bar.
So umbrella to temps, this is a better goal. Uh, we wouldn't choose learn three because even though it has a bigger purple Bob than learns one, um, the pink bar is much bigger as well.
By the time you get to three loans, most people have already been retained, uh, without getting to three loans. So this is how you do this analysis.
Um, you can go and run all of this stuff, uh, with another, uh, metric if you want. Um, so you just change it here.
Um, you know, if you want it to be a balance, uh, that you can press shift and search attend to oh yeah.
A lot of, a lot of you balances. Uh, so you could do, you know, like, it just depends on what data you have.
Um, you can, uh, run it with, um, you know, as many as you want, uh, in the, in this case, uh, view balance is probably a bad run to do because, uh, there's lots of these actions.
Um, so like, uh, comments, and maybe that would be a better visualization. All right, here we go. Yeah. So that looks better.
What's more interesting. Um, and you can see, uh, there's people who made like many more comments, but it's a similar effect, uh, to what we had, um, you know, with the other analysis and just to show you, you can click or journal, hopefully that will work, and it just runs everything in order.
Um, and then you just go down to the end. So, um, here you can see as a really big purple bar and a very small pink bar.
Um, so, uh, here, it might be like comment, anyone who makes one comment is very likely to be retained. Cool.
Okay. So, uh, hopefully that's useful. Um, it's, uh, can be a confusing once you try to understand, like, what the difference is between like share of retained and share retained without the action.
Uh, but, uh, just go through it again, if you don't understand and maybe play around with some of the, uh, data to, to, to see what we're talking about.


Why is it not sufficient to just look for the action that had the highest retention rate?

What are you looking for in terms of choosing the number of actions that would make a good north star metric when looking at the final chart?

Run the analysis but for “searches” instead of “loans”. Why wouldn’t this be an effective action to model retention?

What's our 'aha' moment?

Facebook’s growth team famously targeted ‘7 friends in 10 days’, as their north star metric on the path to 1 billion users. If they could get a new user to add enough friends they knew they’d be hooked, so everything they did was in aid of this goal. You can actually calculate the north star metric for your product, and it’s easier than you think.

The key feature of the FinTech app where you work is the ability for users to make loans to friends – it's also how you make money. If they stop doing it over time that's a problem!

Thanks again for staying late
I have a sensitive project we need to work on
Despite all the good work we've done to bring in new users, I'm worried they're not sticking around
After they make their first loans, are they making more?
That's the important thing for us to crack
But of course I don't want to alarm anyone that we're looking into it
I'd appreciate if you could run a cohort analysis for me
Share the results directly with me, no one else

Venture capitalist 

One of our firm's favorite measurements is the cohort analysis

A cohort analysis is a form of longitudinal study used in medicine, social science and marketing. A cohort is a group of users who share something in common. You could group users by their sign-up date, first purchase month, or acquisition channel. By tracking these groups over time, you can spot trends, and understand repeat customer behavior. 
This type of analysis is essential for discovering retention trends because looking at aggregate metrics you can't tell if users are coming back. For example you could have a growing total user count, but that might be bolstered by the number of new users joining. When you group users into cohorts based on the month they signed up, you can more easily notice that a large percentage of them are dropping off.
Cohort analysis requires a cohort period, be it day, week or month, which you use as the X axis when visualizing it. If you're using monthly cohorts, a cohort period of 2 would refer to the second month they were active, 3 would be the 3rd month and so on. This data can either be visualized as a line chart, with one line per cohort and the cohort period on the x axis, or as a heatmap. Heatmaps shade each box representing a cohort and cohort period by the percentage of users who did the relevant action and were retained. Dark or light spots can indicate peaks or dips in retention activity, which makes it easy to scan and highlight anomalies.

Cohort Analysis

When conducting a cohort analysis it's important to choose a behavior to analyze that is supposed to be repeated regularly. For example if you're looking at daily cohorts, is the action something you expect users to do daily?

The following code is a Google Colab notebook which contains the script necessary to run a Cohort Analysis. Make a copy then review the code and run each cell to see what it does. There is an accompanying video underneath the link to the notebook that walks you through it. You will be expected to answer questions about the results of the analysis afterwards.

I ain't going to walk you through how this whole works. So the first section here is just importing all the libraries and you just want to hit shift, enter where you can hit the play button hit, and it's going to connect you to Google CoLab to the backend. And then I should say connected and then finish running and you shouldn't have any issues there.
Then you want to authenticate. So. Let's see if it asks me. Yeah. Okay. So you're going to click through and then choose one of your emails and then just ask you to click allow, and then. Some token to paste in, and then once you enter that, then you'll be authenticated and that's going to let you pull data directly from Google sheets.
So we run this next cell. I mean, you have, it's your shift and to then that's going to pull in the data from this spreadsheet here which is all of the loan data for you know, for this. When, when the loan was originated then what the ID was the user add that loan theme out. And then the amount they learned as well as the settlement.
Which we're not gonna use. And in order to create a cohort analysis, we need to do a few things. First thing we need to do is create a loan period because we're going to look at the month and year that they learned out the money assets, 2021 here, as it's basically coming off this 11. And that that's just using strip time, which is built into pandas.
Then we're going to get the user ID and kind of make it a use that to kind of find the earliest loan period. So that's what this group buys is doing. Our first reset, the user ID is to index and then we grouped by user ID, which is level zero. And once you set it as the index, and then we find the minimum loan period for that user and then we can just you know, reset the index and, and then we have So that gives us the cohort group.
I said, you can see here this user it's all the same user 47. They have a learn period, 20, 21 and a cohort group, 20 20 0 1. And then actually, if we just kind of run this, oh, actually we would just pull up the data. If you've got to run these in term then you'll end up getting errors. I like, I just got.
Cool. So let's follow that. She's a down, so 47. Okay. In the second month then you know, they still have the cohort group of 20 21. So that's doing exactly what we intended. Okay. The other thing we need to do, this is a common problem. We just need to replace the dollar sign in the loan amount because otherwise it's not going to aggregate it.
Cause it's, I thinks it's a string should be a number. And we're going to make that a flute that you can see here that now the D type is float. So you just do that. And then this part is where it gets a bit more interesting. So this is again using a group by, but this time we're grouping by cohort group and London.
I said that's this here. And then and then we're aggregating the data as we're aggregating user ID, loan ID and loan amount. And then we're just renaming them here to total users, total ones. So what this gives you is, you know, for this cohort what, you know, dates they were present in, and then how many total users they have of those dates, as well as how many loans those users made in those dates and the amount of those.
Then this function here basically just takes the loan cohort period. And it gives you a number. So this is, you know, cohort period 1, 2, 3, 4, 5, as this is. So we don't have to worry about the dates so we can just show you know all of the users you know, that their first cohort, the second cohort, the third cohort cohort.
So that. Connect you know, different cohorts together at to see how they trend over time. And you'll, that will be clear in the second. Then there's just a few more things to do before we can see the final the final result. We just need to reset the index and then set it as a, the cohort group, a group of COVID.
And then we create this group hood cohort group size. So that's a group by these two things. And what that does is kind of give you this, which you use down here to divide in order to get percentages. So don't worry too much if you don't understand that. But basically what it's doing is giving the total number of users for each cohort group.
And then you can basically divide. That percentage in order to get this. Okay. So now this is unstacking. So this is kind of like the opposite of pivoting and basically then dividing in order to get this here. So let me just run these parts that we haven't run yet. So we did this.
Nope. Okay. Let's go back here. Yeah. And then. Oh, we've already done this as middle of the science in there anymore. So don't worry about that
or that amount. And then this is the cohort period. So you can see here 1, 2, 3, 4, 5 and then this is creating that group and now this should run. Okay, cool. So what this is doing is now giving us a percentages of. This whole period. So you can see coho period one you know, what was the cohort group?
So you can basically see you know, this type of visualization. Which is you know, by cohort group what percentage of users were still around after like one month, two months, three months, et cetera. So in the first month it's always a hundred percent and then you can see it. You know, the cohort that started in February you know, less of them were around than the cohort that started in January.
So that's indication that the, you know, whatever we did in February or there's uses required in February one, as good as. You know, the ones from January and to see all of this together you can do a heat map and the heat map basically shows all of this stuff. But in percentages you know, in a, kind of an easy to read format so we can see it's always a hundred percent in the first cohort period.
But then everyone in January we have 36% of people who, who made another loan in February 45% in March. So it can actually go up as well as down, you can see here, this kind of a normally where you know, people are starting in January after 13 months, suddenly 50% came back and originated a loan.
So that's interesting to look at and broadly speaking, you can kind of use the heat. These colors in order to see what's important. So you can see that in every, you know, when something is dark, that's bad you know, we can see a lot of the cohorts you know down to like five, 10, 8% you know, after just a few months so that's kind of troubling.
But then, you know, some cohorts we've got pretty strong retention here and the reason it goes diagonal is because. You know, for people who joined in February 20, 21, we only have two months worth of data. So so that's that's why you look at a cohort analysis in this way, and that's why it's always kind of a diagonal.

What makes a heatmap ideal for visualizing cohort analysis?

What would be a good metric to use in a weekly cohort analysis?

Google how to average the columns of a pandas dataframe. How would you do that to get the mean retention per cohort period?

What was the average retention of users in the 6th cohort period?

Yikes, ok that's not as promising as I hoped
Looks like we need to see what we did right in January and try and replicate it
Thanks for doing this, send me the files and I'll present to the CEO
I'll be sure to give you all the credit!

Is our app sticky?

With hundreds of users signing up, your product looks successful. But questions are being asked about “stickiness”. If too many users eventually leave, soon you’ll have none left.

Are we retaining users?

EXECUTIVE SUITE – STRATEGY DISCUSSION POST-LUNCH

You're in a strategy discussion and need to learn how to effectively evaluate your generative AI based applications more effectively.

Alright team, we've got a lot on our plate today. 
Our boss wants us to dive into the world of evaluating generative AI. 
We're going to learn about different techniques for evaluating customer valuation metrics for AI models, including programmatic rule-based evaluation, LLM based evaluation, and human-based evaluation. 
We'll also learn how to calculate accuracy for each method and get some tips for optimizing the evaluation process. 
So grab your thinking caps and let's get started!

Generative AI has revolutionized various industries by enabling the creation of realistic and customized content. However, evaluating the performance and quality of generative AI models can be challenging. In this course, you’ll explore different evaluation techniques and their trade-offs to help you build effective customer valuation metrics for generative AI models.

The first evaluation technique we will discuss is programmatic rule-based evaluation. This approach involves creating custom rules to determine the quality of the generated output. For example, in the context of generating email content, you can evaluate the length of the email or check if it contains specific keywords. The advantage of this technique is its quick feedback loop and cost-effectiveness. However, it may not be suitable for tasks that require nuanced evaluation.

The second technique is LLM (Language Model)-based evaluation. Here, instead of relying on custom rules, another LLM is used to evaluate the generated output. This approach is useful for tasks that are too nuanced for rule-based evaluation, such as evaluating the helpfulness or readability of text. Multiple outputs are generated by the LLM, and another LLM is used to rate each output. While this technique provides more accurate evaluation, it is slower and more costly due to the need for human intervention.

The third technique is human-based evaluation, where human evaluators rate the generated output. This approach is ideal for high-stakes tasks where false positives or false negatives can have significant consequences. However, human-based evaluation is time-consuming, costly, and requires a labeling step. It provides the highest level of accuracy but may not be feasible for large-scale evaluations.

To illustrate these evaluation techniques, we provide code examples using different scenarios. In the programmatic rule-based evaluation example, we demonstrate how to set up evaluation metrics using Python libraries like pandas, numpy, and TQDM. We define custom rules, such as the minimum length of a social media blog post or the presence of specific keywords, and evaluate the generated output accordingly.

In the LLM-based evaluation example, we show how to use one LLM to generate multiple outputs and another LLM to rate each output. We discuss the importance of ground truth labels and how synthetic data generated by a powerful model can be used for evaluation.

In the human-based evaluation example, we generate images using OpenAI's GPT-3.5-Turbo and ask human evaluators to rate the quality of the images. We demonstrate an interactive approval system that allows evaluators to provide feedback on each output.

Evaluating Generative AI

What are the three different types of methodologies mentioned for evaluating LLM responses?

What are some advantages of programmatic rule-based evaluation?

Which evaluation strategy is best suited for tasks that have high customer risk and where false positives or false negatives can cause consumer damage?

Which of the following is NOT mentioned as a way to optimize LLM outputs?

Discover cutting-edge techniques for elevating the accuracy and effectiveness of Large Language Models and Image Generation Models. Our guide delves into innovative evaluation metrics, providing insights to enhance model reliability and drive impactful results.

Maximizing LLM & Image Model Performance: Advanced Evaluation Strategies	

UPSERT HEADQUARTERS – THE DAY BEFORE END OF QUARTER

You’re in a boardroom at UPSERT Headquarters and your boss, William Winters, Brand Strategist at Whipple, is asking you to figure out a way to protect the company’s data from Open AI. You’ve been researching the LM Studio and think it might be the answer.

We need to keep our data safe from Open AI.
I've looked into LM Studio, and it seems like it could be the answer.
But we need to decide how to use it.
What are our options? Do you have any ideas on how we can protect our data?\

LM Studio is an open source model created by Matter that is used to run models on a local computer. Despite being a smaller model than Chatpetite, LM Studio is slower and doesn't usually follow instructions as well as GPT-4 does. However, a new model created by Mystro 7 is getting closer to the results that one would normally get from GPT-4. 

For those looking to use LM Studio, it is important to understand the technical aspects of the model as running into errors is something that is common. Despite this, the text generated by LM Studio is still decent and with some editing, can be turned into something worthwhile. 

What makes LM Studio so unique is that it was created with the intention of creating narrative and character development. This is done by having two paragraphs per section. This allows for a unique fantasy style that is not found when using other models.

Despite the capabilities of LM Studio, it is still not as good as the free version of Chatpetite. This is why many people are hoping that the model will get better, as having something that is as good as GPT-4 would be extremely beneficial to those who want to create interesting stories. 

Overall, LM Studio is a great model that is able to generate interesting stories and narratives. Despite its shortcomings, if used properly, LM Studio can create some unique stories. It is important for those using the model to understand the technical aspects as well as the capabilities of LM Studio in order to have the best experience. With some practice and patience, LM Studio can be used to generate some great stories.

LM Studio

 Okay meet. Lama this is an open source model from matter. They release it which is really great because they spend all the money training it and then just released to open source. That means I can actually run this somewhere on computer and then there are different. Sources online where you can run this, like on hugging face or replicate. But I really wanted to run it. At my own computer, I'm using something called L M studio. Which you can download is just a Mac app. And and then once you've got that, Set up, you can download anything from hugging face. 

Like this is a downloaded the Lama to chat 13 B. Model and you can prompt it just like you would  So just going to paste in my prompt. And see how it does. What I've found in general is that obviously running it on your local computer is a little bit slower. So even though this is a smaller model than chatting petite that the response has actually come out relatively slower. The other thing is that it doesn't follow the instructions usually as well as GPT four, does. Does. These models, these episodes model is getting closer to what you would get from the free version of challenging.  And in fact, there's a new model out by Mistral which the Mystro seven. The model, which is actually pretty good. And pretty close to chat cheap T levels if you're comparing it to the free version, but nobody has managed to get to the actual results that you would normally get. From GPD for. 

So while this is generating, let's familiarize ourselves with what we came up with jazzy beauty. You can see. Wrote some pretty interesting stuff here. And, we have two paragraphs per section. And it's a unique fantasy style. With actual narrative and character development. Let's see if we can get that We run into an error. Sorry. You do run into this sometimes, unfortunately running these models require some technical ability in some cases. 

But I think in general, I would say these models aren't quite there yet. They're not as good as what you get from chat, GBT, even the free version. We're all hoping that these models will get better now because it would be very cool to have. Something that's as good. 

Now you can see here that the text isn't bad, it's come up with some interesting stuff here. It has, pretty decent introduction. But again, I prefer T's version.

What is the name of the open source model used in the tutorial?

What is the name of the software used to run the model?

What is the name of the open source model from matter?

What is the difference between running the model on a local computer versus online?

What is the name of the new model released by Mistral?

We're just now getting to the point where OS models are as good as ChatGPT (the free version), except they have one major advantage: no data sharing with OpenAI!

I don't want to share my data with OpenAI

Alright, team, here's the situation. We've been tasked with making some creative decisions for our coffee shop interiors, but we don't have enough data to work with. Our boss wants us to use memetic analysis to analyze and tag images of coffee shop interiors using AI. This will help us identify patterns and trends in the industry, and inform our design decisions. So, let's roll up our sleeves and get started!

I know it's frustrating that we don't have enough data to make these decisions, but memetic analysis can really help us out here. \nBy analyzing and tagging images of coffee shop interiors, we can uncover valuable insights that will guide our design choices. \nThis process will allow us to understand the trends in the industry and make informed decisions that resonate with our target audience. \nSo, let's dive in and see what we can discover through memetic analysis!

Memetic Analysis: Uncovering Insights with AI and Vision

In today's digital age, understanding consumer preferences and market trends is crucial for businesses to stay competitive. This is especially true for industries like coffee shops, where interior design and ambiance play a significant role in attracting customers. But how can businesses gain insights into their competitors' designs and make informed decisions about their own establishments?

One innovative approach is using memetic analysis, a method that combines artificial intelligence (AI) and computer vision to analyze images and identify patterns. This technique can help businesses gain valuable insights into the typical interiors of their competitors and make data-driven decisions about their own establishments.

The first step in conducting a memetic analysis is to gather a large dataset of images from various coffee shops. Manually analyzing these images can be time-consuming and inefficient. However, with the advancements in AI and computer vision, it is now possible to automatically tag these images using machine learning algorithms.

To scrape a large number of images from Google, a Google image scraper tool can be used. This tool automates the process of collecting images based on specific keywords, such as "coffee shop interiors." Once the images are downloaded, they can be analyzed using the AI-powered Google Cloud Vision API.

The analysis begins with inductive coding, where two different images are randomly selected and compared. The images are converted into base64 encoding and sent to the API for analysis. The prompt asks the AI to compare the two coffee shops using inductive coding and identify similar and different features and design elements. The response from the API provides valuable insights into the shared and unique attributes of the two coffee shops.

The next step is deductive coding, which involves analyzing each image against a set of predefined labels. This step is more time-consuming and expensive, but it provides a comprehensive understanding of the design elements present in each coffee shop. To speed up the process, asynchronous coding can be utilized, allowing multiple API calls to be processed simultaneously.

The results of the deductive coding are stored in a dictionary, which includes the labels and corresponding coffee shop images. These labels can be customized based on the specific design elements or features that the business wants to analyze, such as natural light, wooden elements, or suspended lighting.

Once the analysis is complete, businesses can gain valuable insights into the design choices of their competitors. By comparing the frequency of different labels across the dataset, businesses can identify popular trends and common design elements. For example, if a high percentage of coffee shop images include patrons,

Memetic Analysis

  Okay. Say you're designing a coffee shop and you want to figure out what the typical interiors for your competitors. So you want to understand what type of furniture they using, whether they have plants whether there's a chalk, a chalkboard on the wall for the menu. Like this. 

And you wanted to do that analysis and see how often a certain things come up versus not. So one interesting way to do this. Is instead of manually looking through a bunch of images. You could tag them automatically with AI because aegypti vision is actually pretty good at this now. What you need is you have to scrape. Google images, which is actually the hotpot. But thankfully there is a tool that does this. 

What do you say if you just search a Google image scraper? Yeah, literally just. That's just such a Google image of scraper. And that was the Fest result here. But however you get the data. It doesn't really matter, but I'll show you how it works. I.  I just, this is in the main.py. So when you download this to your computer from get hub then you have this curd you just put in here. What are the different keywords that you want to search for? 

I downloaded a bunch and you can see. Got a bunch of, coffee shops in Beijing. Got a cough coffee shops in the room. And then also New York, I'm willing to compare them potentially. So I'm going to just run an abbreviated. Part of this analysis I would say by the way, it takes a while to down the them say I downloaded a bunch and it doesn't always work. 

So you can see, I didn't get 80 every time we got 36 and Sydney. But once you literally just run Python, main dot PI. And you should see these images. Stop appearing in hand. But that's not the part I'm going to teach you because that is someone else's code. And I'll let them teach you that in that read me. But what I'm gonna teach you is what's due when you have the data, right? 

This is a script for the Jeep T vision API. And you just need to run. These libraries here, but I have all the code written here. I had to pull in. What are the different coffee shops, right? As you can see how many photos you got just looking at the first five here. See, I've got 4,500 just from New York. 

Got a bunch of Sydney. Et cetera. So. What I'm going to do is I'm just going to.  Just limit just to Let me just a little bit to Sydney.  

Just for testing because a visionary pair is expensive. If you're running a bunch of times, it also takes a little bit of time. So I'm just gonna run it on these 33 images and in Sydney.  All right. And there's two pastas. One is inductive coding. And then the second is deductive coding. 

Inductive coding is when you're looking at these and you look from, for patents and you're comparing the Migo. Okay. That one has a coffee. You know that one has a chalkboard for the menu. And then there's other ones that have chalkboard for many use as well. So maybe that's something that I want to hear again, is another chalkboard. 

So I've noticed that pattern. And now that I know there's a pattern. I want to find that everywhere. So there's another truck board. There's another chalkboards. So that's deductive coding, inductive coding is coming up with the patterns, comparing and contrasting to find what's different or what's similar. And then deductive coding is then going and finding those patents everywhere else. So we're going to do those two things here with a GP vision. All right. 

The first thing we're doing is inductive coding, and we need to be able to send an image to opening eyes. So the way that you send an image is with the base 64 encoding. This literally just takes an image and then turns it into some texts. And that you consent along with your API call.  And this could does, is it just looks in the folder and then it just chooses two images at random. From the different folders that we have. 

So we just have the one folder right now, but if we have multiple folders that. It would look in all of them. Oh, I did mention, I forgot to mention one thing when using Google images scraper you'll probably need to download. I had to download this Mac. 64 Chrome crim for testing. 

When it scraping these images you need that in that. Here I've just put us in folder called WebDriver and then it opens that up. That's, I've just mentioned that because I ran into an error. All right. So back to the inductive coding. So choosing an image pairs, choosing two different images at random. And then it's comparing them. So it's getting the base 64 string, and then it's using this prompt here compared these two coffee shops using inductive coding. What features attributes, elements of design are similar, different. 

And then I'm asking for a structured response here. That's just how you pass it into GPC vision, by the way. You just. To client chat completions. And then you pass the prompt. But then you also pass in the two images as well in the same content string. And I'm just using the user prom tear, but you could also have a system prompt at the top. Cool. 

So let me just run this, see what we got. So it's selected to these images at random. Number 30. 25 and number 31.  And you can see, it takes a little bit of time for the image response is this is opening eyes, biggest modal. Here we go. So now we've got some labels the there's some things that they share. 

So coffee shop a and B. Both have modern designs. They both have wooden elements. But only cough shot B has natural light. And own the coffee shop a has a spacious interior. So that's really interesting. Now we have some labels. We have a dictionary of labels. We can run that and then save it. So if we look at. We should have hit some clean levels so you can see. 

Yeah, there's a bunch of labels here. For these different things and it's rewritten them. We've got the actual kind of folder that they're in the label. And then the phone name.  Cool. So now you have your level dictionary and that's useful in itself because you could run that multiple times across multiple images. 

So we just compared to those images, you could compare every single image against each other. It just takes a little bit of time. That you can run again as well and get a bigger dictionary of data. Now it comes to the deductive coding. And this is the one that's more time consuming, more expensive. 

The way, the reason why I've set this up as asynchronous is. Because otherwise it takes a very long time to run because you're going through every single image and saying and every single label and going, is this label in this image, does this coffee shop have wooden elements? Does this coffee shop? 

How bought-in elements. Does this coffee shop have wooden elements, right? And then you go through all the other labels about, does this coffee shop have suspended lighting and does this coffee shop have a seating areas? There are different tricks for doing this faster or slower. But for me, I found if I do an asynchronous. It means I can get all the labels at once for a single image. And they all run quite quickly. So I runs a little faster. So again, this is here, we're just passing one image in and we're just passing the user prompt, which is much simpler. 

If the label applies to this image, reten one, otherwise reten zero on the output one, a Sierra.  All right. If we get a one, then we are going to return the label. And if we don't, then we're not going to attend on. So this. This does this, the asynchronous code would just set up a bunch of tasks. 

So we run it for every image and then we wait until after all the tasks had done for that image. It's processing all the labels at the same time. And open AI can handle multiple API calls at the same time. What will happen is instead of waiting for each one to come back, it's going to process all of them said time. 

We're going to get through asset. If we're doing 10 labels, we'll get through all 10 at the same time, rather than waiting. If one takes a minute. We're not waiting 10 minutes. We're getting all of them in one minute. So that's the benefit of asynchronous. Cool. And then we're just going to return the labels. 

If the. If the label wasn't there.  Great. And then actually running the deductive code and we just pull open the cleaned labels, Jason, that we have before. And then we have the clean labels. Which which loads to Jason from that dictionary we had, and then it finds the values. 

I've also limited it to just five labels per coffee shop. Otherwise, it's going to take a bit of time.  Cool. All right. So let me just run this. And the processing coffee shop Sydney, you can see that all three of those came back to this in time. But some of them are taking a little bit longer. But we're not waiting for one to finish it before we got the other. 

Cool. So here we go. So now we're processing coffee, shop Sydney.  

And let's see how quickly they come back. Yeah, this is Sydney 30. Okay. Now we're processing Sydney 10. Getting hanging light fixtures. So this is only coming back. As positive if it does have that label, if that makes sense.  

Cool. That's how. Movantik analysis works with vision it's really powerful. Once you figure that you can apply this to pretty much anything. Like any time you have a lot of images, like maybe in your ads or on your website. You can do some analysis to figure out what labels are up. And you can define the labels as well. 

You could sell it. What sorts of things you're looking for. You could give it a D list of things. However you want to do it, that's fine too. You can just add new stuff to this. This clean labels thing, right? You could just add in some new.  New ideas, right? 

Like I want to. I wanna find, this is customers queuing, right? I want to find out if there's a barista that has a tattoo, right? Like you could just add that as another option, copy. This one has Tattooed.  

Barista.  

All right. And there's already funding. If the barista is present in this bit, then we're going to see if they have tattoos, visible tattoos, cool. That's how this works. And then what you're going to get back is you're going to get some, a file name like this. This is just a big list of all the different labels, right? 

So it says coffee shop, New York 42 has the label of street view. This one has the label patrons present, right? And then you can do some analysis after this. To basically figure out okay. How many time, how many does it, how many labels does this apply to you? 

So how many of the coffee shops has this label? I'll give you an example. I'll show you an example here, cause that's still processing. Cause it's going through 33, images is an example from New York. Then I ran earlier, so you can get a list of all the labels into a data frame. This is like an Excel spreadsheet. And then you can print out what percentage of the total images have these different labels? 

So here we're finding that 41% of the patrons that 40% of the coffee shop images, show patrons present. So to me you want to really pay attention to the ones that are on almost everything. Because that's almost like a rule of the game. If you don't have patrons present in your coffee shop image, that's probably really bad. And you definitely want to at least have a stance on that. 

Like maybe have a good reason for not having patrons present in the image of your coffee shop. When you're uploading that image to Google my business. But you should at least address that have the strategic choice here. Similarly, for things that are not very popular you also might want to think about these as a differentiator. 

So exposed brick interior, you could argue that, that's not very fashionable anymore, and that's why only 10% of people had it. Or you could say, this is going to be fashionable. I can see that this is a trend as, or I'm going to stop using this right. That's how. You do this type of analysis. 

Obviously what it doesn't tell you is like which choices to make and what designs to have. But if you're not a designer like me and you want to be a little bit more data to make your decision, this can help inform your decision to give you a map of. What is what are the different creative decisions you need to make? 

When you're doing something designed or visual. All right. Hopefully that's useful. This is my little hack for, being a data guy, but also playing in the world of branding. Hopefully you find it useful too.

What is one potential benefit of using AI to tag images in a coffee shop analysis?

What is the difference between inductive coding and deductive coding in the context of analyzing coffee shop interiors?

What is the purpose of using AI to tag images in the coffee shop analysis?

What is inductive coding in the coffee shop analysis?

How can the coffee shop analysis help inform design decisions?

There are millions of decisions you need to make when starting a business that are too small to test, but are too risky to make without data.

How can I make creative decisions when I don't have enough data

You’re in a meeting with the CEO and your boss, as well as the rest of the executive team. They’re discussing whether the product is priced too low, and you offer a way to find out: Gabor Granger pricing analysis.

That’s interesting, so we basically survey people and ask them what they would pay
But not directly like “what would you be willing to pay”
Instead it’s like “would you buy this product if it was $10?”
Then we keep increasing or decreasing the price depending on if they say yes or no?
Yeah I’m willing to give that a shot: it’ll be better than just guessing!


Gabor Granger pricing is a type of economic analysis that attempts to identify the optimum price for a good or service. It is named after Zoltan Gabor, who developed the concept in the 1930s. It works by asking a customer if they would buy the product at one price, then if they say yes asking if they’d buy at a higher price, if they say no asking if they’d buy at a lower price. This keeps going until the prices are exhausted, and we can draw a demand curve out of the aggregate responses. This method is designed to avoid asking customers “How much would you pay?” because most people don’t know off the top of their heads. However if you show them a price and ask “Would you buy this product for $10?”, they will get an instant reaction and be able to decide if it feels high or low. 
There are a number of advantages associated with Gabor Granger pricing analysis. Firstly, it is relatively simple to understand and use. Secondly, it takes into account a wide range of factors that can affect prices, such as costs, demand and competition. Third, it can assist in setting prices that are competitive, yet still provide a good level of customer satisfaction. Finally, it is a good option for businesses that are looking to maximise profits, or decide on the tradeoff between profit and volume. 
There are also a number of disadvantages associated with Gabor Granger pricing analysis. It can be time-consuming to carry out and may not always produce accurate results if people don’t answer truthfully about their willingness to pay. Gabor Granger pricing analysis is most appropriate in situations where there is a need to price a product or service with not much competition, as it doesn’t take into account competitor price changes and therefore is prone to causing bidding wars. It is important to remember that Gabor Granger pricing analysis is only one tool that can be used to price goods and services. There are a number of other pricing methods available, and it is important to select the most appropriate option for each individual situation.

Pricing Analysis

00:01 Hey, let's learn how to do G or Grainer pricing analysis. So this is the end result that we're trying to get to, and what we're trying to do is draw this to man curve and see at what price should we set the price of our product in order to maximize the amount of profit that we make.
00:18 So I'm gonna walk you through how to do this, but first, just take a look at how the survey is designed.
00:23 So a Ga Granger survey is a little bit complicated. You usually need some specialist survey software, or you need to use custom variables and something like Typeform.
00:34 But essentially what you need to do is ask at the starting price, would you purchase this product for $11? If they say yes, then you increase and say, Would you, you know, purchase at 13?
00:44 And then you kind of keep going and test the different prices. You could also say, if they say no, would you purchase at eight?
00:52 If they say yes, would you purchase at 10? You know, so basically it's kind of, you know, starts at a certain point and then goes up or down depending on what they answer.
01:02 So this is a, you know, once you have the survey data, you can draw the pricing curve so the demand as in how many people are willing to buy at that price.
01:12 And then also the, you can figure out the profit as well, which is what we're gonna do. Cool. So once you've done this survey then this is the kind of data you'd have.
01:26 Like, so you'd have respondent ID on the side, then you'd have different prices.  I've done from $10 to $50 because this is a this is for the brand Ultra which is a men's grooming brand.
01:40 And you know, we can see respondent one said yes at $10, yes, at 15, yes, at 20, and then yes, at 25.
01:49 But then they stopped saying yes after that. So it was all zeros afterwards. Whereas say respondent four only said yes at $10 and they said zero for the rest.
01:59 So then you can randomize this as well to make the survey more it can kind of randomize the starting point to make it better.
02:08 Because one problem you might have is if you always start at 10 then you're anchoring them towards a low figure.
02:15 Or if you always start at 50, then, you know, then maybe you're kind of anchoring them too high and they're gonna say a price that they actually wouldn't buy it.
02:24 But whatever the kind of statistical issues with this, really, you're just looking for a smooth curve at the end. And, and this's gonna give you a decent range a respectable range for you to be in.
02:36 Cool. So you have all the survey data, in this case surveyed 94 people. And then all you need to do is just add up the ones essentially, right?
02:46 So I do it in average and an average of, of any column that's either a one or zero average would just be the percentage that were ones, right?
02:55 So 93% of these were ones was only 5% of People said 50. So then you just need to make some assumption about the market size.
03:05 In this case. We've assumed there are a hundred thousand people in the market. And what this is telling us is that 93% of people so like, yeah, it's actually 92.5 percent of people would become customers if we priced it at $10.
03:20 Which is great, you know, you wanna capture the whole market, right? But it's not so great because actually you know, there are, you know, there are options with higher revenue and and we actually making a loss here.
03:32 So the way you work that out is that the revenue is just the number of customers, tons the price, right?
03:39 So, you know, lower price, you need more customers in order to make the same revenue. And then the profit, we've just made some assumptions for this business.
03:47 So they have to work out the profit, we take the revenue, and then we subtract this they've got in this case it's a million dollars in fixed costs, which is important because, you know, whatever happens if they don't sell any units they're gonna lose a million dollars.
04:05 So yeah, that's a more realistic way to look at it rather than just margin because you know, if you're a startup, you have like a, you know, you have to pay for a warehouse to house all these products, right?
04:16 Like you have to pay for staff and that you, the staff still need to get paid no matter how many units you sell.
04:22 So you usually have some fixed costs that you have to factor in, and that necessitates you selling at least a minimum number of units in order to kind of get to the right level.
04:31 And then you have the variable costs. So in this case, it only costs two bucks to make the product. So you know, we're just multiplying the number of customers by two bucks and the, these two things into place.
04:41 So you can see that we lose money on the $10 even though we have 92,000 customers with with $15 we have 82,000, so we lose 10,000 customers.
04:55 We're gaining a lot in revenue. So now we're making 1.2 million in revenue, and that's enough to cover our fixed investment of a million.
05:03 And also the unit costs. So we're actually turning a small profit here. And and then you know, don't worry about this line.
05:10 I'll explain that in a second. But then it kind of keeps going on, and these are basically the estimates. And you can see that you know, this is the revenue maximizing point but you're not making very much profit.
05:22 Whereas if you increase the price to say, $25, you're only making half of the market by your product. But you know, and, and your revenue is lower, right?
05:31 Like it's $20,000 lower than the $15 option. But that might actually be better to have less market share and higher profit.
05:39 So think about this, like Apple, you know, Apple could price their phones cheaper and they would get a lot more people buying iPhones, but they would make less profit.
05:47 And, and right now they make the most profit out of any of the other phones in the industry. So they've kind of chosen a price point that's higher up here.
05:56 And You can see $125,000 in profit. Now, if we keep going it's actually gonna get worse because yes, we're making more per unit but we're selling less units, right?
06:09 So we're losing a lot of market share here, this share here. So you know, we're getting like a hundred K in profit by selling 39,000 units.
06:18 We're getting 78 k in profit by selling 82,000 units. So, you know, I mean, it's really up to you in terms of like what the business goals are and how much risk you wanna take, you know, things like that.
06:29 But it might actually be better to have a higher price product and sell less units because it maybe than you know, that's easier business to run or, you know, whatever.
06:38 So like a lot of luxury or boutique businesses operate these, you know, higher price and you know, higher price levels but you need to sell a certain amount of units to cover those fixed costs.
06:50 So for example if you, if you only sell 30% of the market you're only just making a million dollars in revenue and that's, that barely covers your fixed costs and it doesn't cover, doesn't left anything over, left over for the $2 per unit.
07:05 So, so you actually lose money $17,000 in this case, and that kind of keeps going on. So yeah, you could price it 50 bucks, but you know, only 5% of people wanna buy it that rate, right?
07:15 So you can actually lose, you know, a quarterly, three quarters of a million dollar dollars here. So that's a terrible choice.
07:23 So it just shows that, yeah, like yes, you can choose your pricing and you can be strategic. And once you map this out, you can make a more informed decision.
07:31 Really, it's a market that decides, you know, what you can get. Like, you can't just say, Well, my product's worth 50 bucks.
07:37 That's really for the market to decide. Cool. So now what is this line at the bottom here? Well, what we've done is we've drawn the curve here, see this curve, this trend line.
07:49 And the way that we've done that is if we go to edit chart and then we go to customize series or on data series one.
07:56 So that's that if you click on trend line, and there's a slightly different way to this in Excel, but you know, works is the same you can stick a trend line on and then you can say use equation and then show the R Square.
08:09 And what that does for you is basically just plots the nearest fit to all the, all the points here. So you can see some of the points are slightly off that one, slightly above, that one's slightly lower.
08:18 And this is a in this case we used a a polynomial. So there's like different options but polynomial polynomial fits best should do that polynomial fits best cuz it lets you kind of do this curve up and down.
08:36 Cool. So the, in this case, the, the Oscars was 0.95, A perfect fit would be one. So this is a very good fit.
08:45 And then and then that gives you the equation to derive the, the profit, right? And so it predicts given a price what profit you're gonna get.
08:55 And the way it does that is just this complicated formula, but you don't have to worry too much about it.
09:00 Basically it's saying you start with the value of minus 73,000 or sorry, minus 736,000 7 68. So then you add 7,020 yeah, so 70,278 multiply by x and x the price.
09:20 So if it was $10 and it'd be 70,278 times $10. So it's about 700,000. And then you also add minus one for, for two times x to the power of two.
09:37 And that's basically what gives this kind of coving up and down chip. So you don't need to worry too much about the statistics here you know, just know that that's what that's doing.
09:46 But all I've done here is just copy this function across, right? So is minus 7 36, 7, 6 8, and then plus 70 2 78 times B one B one is the price plus minus 1 4 42 times B one the price again to the power of two.
10:03 And there we go. Now we have our prediction. So we could do this for any price, but we've done this across all the prices here.
10:10 And the reason why you wanna do this is you see there are some lines are, you know, dots are above and below.
10:14 And really what we're trying to do is smooth this out. Like there's some statistical noise here, right? So we're just kinda smooth this out and find the optimum price, and it is at the $25 mark.
10:25 So making, we, we would predict we're making 118,000 in revenue. And the reason why the smoothing out is important is that because, you know, of the survey data, the $15 price mark, it's actually like higher, higher than we would expect.
10:38 So we might look at this and say, Hey, we're still gonna make this 78 grand in profit, but actually the the function or the prediction is that we lose 7,000.
10:47 So we're playing with fire there if we're gonna, you know, stick with this ROS estimate. And and instead we would kind of wanna stick at least, you know, somewhere in this range.
10:57 If you wanna optimize, you know, slightly more to revenue, then you could be at the lower end of this range.
11:01 You wanna slight optimize slightly more to, to margin then you could be at the upper end of this range. You know, but you know, it's giving you a place to choose your price.
11:12 Cool, Hopefully that was helpful.

Why are fixed costs important to take into account with regards to profit?

What might be the rationale for maximizing revenue over profits?

Why might Gabor Granger analysis not be a good fit for industries with intense competition?

Are you charging enough? Most businesses guess at pricing and move on, but that approach is going to leave money on the table. How do you set the right price to maximize profits?

What price maximizes profit?

Building a marketing mix model is not just about finding the right parameters, but also the hyperparameters used to transform your data to account for adstocks and saturation.

You've been looking for an excuse to learn how to use evolutionary algorithms for building models automatically, and now this might be your chance. You hear on the grapevine that a colleague is struggling with their model.

Yes you heard correctly
We're struggling with the GoolyBib model for Italy
I'm not satisfied with the model as it stands – the error is over 10%!
This could be a good use case for automated hyperparamter optimization
Doing it manually is getting us nowhere fast!
I'd appreciate your help

Every model has parameters: these are the variables used to make predictions when the model is built. For example your model might say that Facebook Ads drives a cost per purchase of $5. How the model arrives at that conclusion is dependent on Hyperparameters: parameters that control how the model learns the right answer. To take our Facebook Ads example, our cost per purchase might get worse at higher spend levels due to saturation, or there might be a carryover effect where spend today affects performance tomorrow. 
These transformations of the data are key to building an accurate model, but it's hard to know ahead of time what values are correct. The manual way of figuring this out is to choose a value for a parameter, for example an adstock level of 20% (20% of your spend from today has an impact tomorrow), and then move that value up or down to see how it impacts model accuracy. If you increase it to 30% and accuracy of the model goes up, you know that's a better fit. The problem is that each parameter affects the behavior of every other parameter in the model, so it can be hard to find the right value.
So there have emerged several strategies for dealing with this task of Hyperparameter optimization. The brute force approach is a grid search: try every possible value until you find the right one. This isn't always possible however, because there may be more potential combinations than there are stars in the universe! Random search lets you set a limit of how long to look for the right values, randomly guessing within a budget. The method we're using today is an evolutionary approach, which learns in a more efficient way than random guessing. It deploys a population of potential solutions, then kills off the worst ones (with the highest error) and replaces them by mutating the best  performers in the batch. With hundreds or thousands of iterations we can efficiently arrive at the best estimate for even a very complex set of parameters.

Hyperparameter Optimization

GoolyBib Italy model
Hi,
Thanks for offering to help: please see model attached. 

We have been choosing the values for theta (adstocks) and beta (saturation) manually and this is the best combination we've got to so far.

Best,

Statistician and Management Consultant

To optimize the whole, we must sub-optimize the parts.

Marketing campaigns tend not to have a linear relationship with sales. As you spend more money on a channel, the efficiency decreases because you have to outcompete a greater number of other advertisers for ad inventory. In order to model a non-linear variable in a linear model, we must transform the data. The Power saturation curve we're using simply takes the advertising spend to the power of beta. But what is beta? It's a Greek letter, because by convention mathematicians use them to identify parameters in a formula. 
We don't know what the correct value of beta is – it's our job to estimate it. The way this is done is by trying a lot of beta values from ~0 (completely saturated) to 1 (linear) and see which value best improves the model. If we see the accuracy increase from 0.3 to 0.4, but then decline again at 0.5, we know the true value could be around 0.4.
We do a similar job for adstocks, to estimate the percentage of advertising impact that's felt the next day after spend. For example if we choose 0.2 then 20% is felt the day after, 20% of that the next day (4%), and so on. The process is the same: calculate the accuracy of the model at each level of adstock, and see what it does to accuracy. Of course this can get complicated to do across multiple variables, as if you change one it changes the whole model, and the odds of ending up with the best possible combination are low.

Theta and Beta

For models with up to 5 or 6 parameters, it can be feasible to adjust the model manually and find a good enough solution. However this task quickly becomes impossible with more parameters because the number of potential combinations increases exponentially.

Finding numbers for Greek letters

Rather than manually choosing the value for each parameter, let's get Nevergrad to do it for us. It uses an evolutionary algorithm to build hundreds or thousands of models to find the right parameters for us.

After digging through the Nevergrad documentation and using some marketing mix modeling code you had from another project, you have something ready to run.

Nevergrad is a Python package created by the Meta (Facebook) team, for gradient-free optimization using techniques. It works using differential evolution and particle swarm optimization, which optimizes a problem by iteratively trying to improve a candidate solution with regard to a given measure of quality. 
This process is called evolutionary, because it is based on the ideas behind evolution in nature. The simplest evolution strategy operates on a population of size two: the current point (parent) and the result of its mutation. Only if the mutant's fitness is at least as good as the parent one, it becomes the parent of the next generation. Otherwise the mutant is disregarded.
Such methods are useful because they make few or no assumptions about the problem being optimized and can search very large spaces of candidate solutions. Because it does not use gradients like other optimization methods, it can be used on optimization problems that are not even continuous, are noisy, change over time, etc. This is perfect for machine learning where the actual model might be a black box, where changes in one part of the model can give non-linear responses in another part of the model. The downside of evolutionary algorithms is that they are computationally intensive, so take a while to run, and do not guarantee an optimal solution is ever found. 

Nevergrad Optimization

Hi, I'm going to walk you through how Nevergrad works
I'm gonna take the file here, and just going to make a copy, so we have our own copy in drive
First you want to install Nevergrad
So if you just click into that cell and press shift + enter, or you can click play here
That's just downloading the library and installing in on the machine Google is giving you for colab.
Now we just need to load in all of the libraries that we're going to use
Just shift + enter, or click the play button.
And these are some functions we're going to use as well
We need to authenticate Google drive so we can import the data
So if we just run that cell, then it'll give you a link to click through, and you just want to authorize, then copy and paste
Hit enter
Now if we run this cell
Its going to pull in the data from this Google doc, and it's going to display it here as well
So we know that we've got the right data, its the number of days, then facebook spend, radio spend, tv spend and then revenue
Now let's plot the data so we see how the data is trending over time
So we see here it's a plot of revenue, and then we see all the facebook, radio and tv spend underneath
Now in order to run Nevergrad we need to do a little bit of setup
First we need to get all the spend columns, so the media variables are if 'spend' is in the column, and then we just need to create the X and Y variables
Y is revenue, that's what we're trying to predict
Then X is what we're trying to predict it with
This is an adstock formula, it helps us take into account that there will be some lagged effect of advertising
And Nevergrad is going to help us find the right adstock values
It's going to try lots of theta values here
And it needs that function in order to transform the data in each run
Same for saturation, this is diminishing returns its the same thing, its just a power transformation, with just one variable
You can have more variables, if you do something more complex, like a weibull adstock or a hill saturation curve.
Now we need one big function to model everything together
So it needs to take in the facebook spend theta, facebook spend beta, and then same thing for each other channel
And then it just pulls it together, and then creates the adstocks, and the saturation, so it does adstock first, and saturation after that, which is best practice
Finally it creates a new data frame, basically an excel spreadsheet
then creates the X and Y values, and splits them into X train, X test, and X train X test
So this is a train test split that lets us check the accuracy, with data we haven't seen before, in the model
This prevents overfitting of the data
Finally we have a linear regression model
Although you could replace this with whatever model you want to use, if you want to use Ridge regression or something different, and then it fits the data, and gives us predictions, we then calculate the accuracy metrics
So normalized root mean squared error, is just an accuracy metric that gives a percentage, that penalizes really high percentage values, more than mape 
The mape is the mean absolute percentage value, and that's just a general percentage you were wrong each day
We also have the R squared as well
Then we kind of wrap this function and get all of the results, but only return the mape value
This function is what nevergrad is going to use to optimize to
So here we're optimizing to the mape
But we could swap this out for nrmse or r squared if we wanted to
Now we just need to make these scalar values
Don't worry too much about this, but we're essentially just setting the bounds of what the parameters could be, and then we're going to run the optimizer here
The optimizer, this is actually Nevergrad doing its thing
We're passing in the instrumentation into the parameters, and then we're giving it a budget, and we're saying this is how many times we want you to run a trial
And the trial is basically the algorithm choosing what kind of parameters to set or to try next and then run that trial to see if those parameters worked
Then we just want to log the error values, so we can do a chart afterwards of how quickly it approached the lowest mape, 
so that's what we're doing here
we're just registering the callback
every time it tells us the answer for that trial
ok and we're going to set a timer, so we just see how long it takes to run
this is the optimizer function, this is where its actually minimizing the error
this is like where all of the work is actually happening
so when you run all of this
oh I have an error
main function is not defined
if you ever get this, its because you probably ran things out of order
so I'm just going to go back and run this one here
then run this one again
there you go, you can see it took 1.8 seconds to do those 100 trials
and these are the values that it came back with for facebook beta and facebook spend theta
So these were the parameters for the adstocks and saturations, that got us the best accuracy out of all of the models they tried
You can also kind of push those parameters into the function that you created in order to get the MAPE
So it's 0.076 so it's 7.6%
Then you can see if you plot
So just run that so you can see if it works
It's going to be a little bit different each time, because its an evolutionary algorithm, so there's some random elements in there and it wont' always be the exact values
Just going to run this
We can see here that we started off with a very high mape, it's 18%, and then even spiked up to 20%
Then over time it kind of evolved towards the better values
It got a lot lower, to 0.08 and then stayed around there for most of the 100 trials
Then if we just run the build model function, its going to return us the mape value and the nrmse, r squared value, the model and also the result
So we just run that and its going to give us those values
We can also plot the accuracy
So if we plot the figure, and we take the revenue and the prediction from the result, which is the data frame, then we are showing that
We're seeing here the revenue is in purple, vs the red prediction
That's how you run nevergrad in order to find the right parameter values




Use Nevergrad

What are the benefits of using Nevergrad vs other optimization libraries?

Describe in your own words the steps taken in the Google Colab notebook to use the Nevergrad library

That's great to hear
8% error is much better than we had on our manual model
Looking forward to seeing if we can improve it further by dialling up the number of models

This approach works if you're running up 2,000 models, but it takes a while and hits an error if you set the budget too high. To reach 10,000 models we have to parallelize the operation, meaning apply multiple 'workers' to solving the calculation, rather than calculating each iteration one at a time.

Better than random guessing

We need to make some modifications if we want to scale up the number of models we're building. It will help to parallelize the operation to allow multiple calculations to happen at once.

We have a model that works – we're already getting better accuracy. Let's modify our code so we can run a much higher number of models.

Nevergrad offers parallelization right out of the box, you just need one other library called 'futures' to make it work. Futures lets you create a ThreadPool which manages the multiple workers you create to do the calculations. These workers can carry out tasks asynchronously, as if they were their own mini computers, and then pass back the information to the main function. We run this within a 'with' statement to ensure they're cleaned up properly and shut down after everything finishes executing. 
You can have as many workers as you have cores available for processing on your computer, which you can find with the os library `os.cpu_count()`, though it makes sense to leave 1 core for anything else your computer needs to calculate. For demonstration purposes we're going to just use 2 cores, which is the minimum needed to calculate 10,000 models. We set this in the optimizer on the first line with `num_workers=2`. The following file has the updated script we'll use.

Parallelizing Nevergrad

Copy the code from the last model to show the output of the parallelized code. What is the MAPE for the next model? Was it significantly better?

The parameter for TV spend beta changed from 0.45 to 0.10 in the new model. How would we interpret that?

What does the chart showing mape over time tell you? What does it look like Nevergrad was doing? Is this behavior what we'd expect?

Thanks for sending the results across - very interesting
I thought dialling up the models would get us a better result
However seeing that it jumped around a lot and tried lots of options gives me confidence that we really did end up with the best allocation
Thanks again for your work on this
We can reuse the script with modifications for every new model we need to optimize!

Scaling to 10,000 models

It's tedious to try different diminishing returns manually to find the best one. Facebook's Nevergrad package can try thousands of models for us!

Let's try 10,000 models

Ultra’s Offices – Quarterly Strategy Meeting

The hot topic being discussed is SEO: your blog content is good quality but not yet ranking on search engines. How do you build authority and start ranking on the first page of Google?

I think the quality of our blog content is great
But right now nobody is seeing it
Of course we all know SEO takes time
However if we don’t show tangible progress soon we won’t be able to justify the level of investment we’re at right now
The key issue has to be links
In my last company none of our content ranked until we got over 2,000 backlinks from a post that went viral
The rising tide lifts all boats…
We went from 200 to 20,000 visits a week
Of course we can’t expect to just go viral here with our skincare brand, that’s a bad strategy
But we can start building links
Let’s start with identifying broken links on competitor’s sites and redirecting them to us
Should be an easy place to start

In order to rank on Google you need authority. Google’s PageRank algorithm is somewhat of a popularity contest: if websites who have a lot of authority link to your website, they bestow authority on you, and your ranking increases. Of course the links still have to be relevant to the content that you produce. They also have to be legitimate: if our link profile looks unnatural like we’re buying links we can get penalized and de-indexed.
So we need to build authority in a legitimate way, ideally without spamming a bunch of people, or taking up a lot of time with websites that aren’t relevant. The cleanest, simplest and most useful way to do this, is to identify broken links to competitor websites. This occurs when other websites have linked to your competitor but the link is no longer working, it’s leading to a 404 page. This is an opportunity to contact the author and let them know, in the hopes that they’ll reciprocate and link to you as well, perhaps even replacing your competitor’s link with a link to your product instead.

Broken Links

Hey, let's do some broken link building. Uh, so we're going to use <inaudible>, uh, for this analysis. Uh, you can really do this in most of the SEO tools and potentially even, um, you know, use social operators to do this in Google.
Uh, but, um, I'm just going to show you a trust, cause that's what I use. Uh, I think you can pay $10 or something for a trial and, and do kind of the analysis that you need and then cancel.
Uh, so, uh, we are doing this for a skincare brand called ultra start-up and they compete with, uh, this competitive set here.
Uh, so, uh, they've done some surveys of the potential customers and they're finding that a lot of people use, uh, Harry's skincare, um, uh, solutions or they use, uh, Jack Black, which is the market leader in the U S um, and then they also use melamine guts, uh, because this is a very trendy skincare, which say usually encounter when they travel, uh, to different hotels.
Um, so, uh, we're going to go through these and figure out, uh, whether they have broken links, uh, because anyone that's looking to them, uh, could potentially be looking to us as well, especially if we notify them that one of their links is broken.
Cool. So, um, let's go to Harry's. Um, this is for the U S market. So you're going to copy your out, um, wherein within <inaudible> in the site Explorer, just going to put that in there and then hit enter.
Uh, so this is basically bringing up all the data on Harry's. You can see that they have a very high, um, you know, relevance in terms of, uh, domain, like they have very high authority.
Um, but, uh, the main thing we want to go to here and nature estimates is pretty simple to get to best by links.
Um, this is one of the best, uh, pages you could look at. It kind of shows you what pages have the most backlinks to them.
Uh, but, uh, the important thing for this link-building strategy is we can filter by HTTP code here. So we can basically say, um, four for not found.
Uh, and that's going to tell us all the different pages within Harry's that have a lot of links that, uh, haven't been, uh, found.
Um, so, um, we can, uh, you know, potentially search within the results of we're looking for something specific. We can export it into Excel and do some filtering.
Uh, but, um, I just like to kind of rank it based on, um, the Euro rating, cause it's a good proxy for like the most important URLs.
And then I just scroll down until, uh, there's not that many links coming through. So, uh, here they have a, you know, the social mission, which I guess say, give up on, uh, it's not found on the website anymore, our story.
So some of these are like, uh, mistakes. You can see, this is like the France page. It's got France, France.
Uh, so maybe there's some errors and that's why that's for a forum. Uh, but here's something interesting. Uh, they did a post for, um, I guess for, uh, pride week.
Um, and I, it's not showing up on the website anymore. So if we go to the way back machine, internet archive, this is sometimes a store, uh, different pages on the internet.
So maybe we get lucky and kind of see it hit. Okay. So it looks okay how one record and, uh, we can sometimes see what the website is.
So although it doesn't always load all of the, uh, all the images, uh, so let's just see what this page was just going to wait for that to load.
Hmm. It's taken awhile. Uh, it doesn't look like this living anything. Okay. Well, we can get kind of guessed what it is shave with pride set.
Um, we can see if there's another page on the left side That talks about this. Here you go. So they do have like a shave with pride set.
Now this is specific product, but I guess this was a blood post about this product and it's a pretty visually appealing one looks pretty cool.
Uh, so, um, so now we have an opportunity, I think because, um, we have here, um, there's 40 referring domains, uh, to this page that are going to a broken link.
So that's a user experience issue for the people who read about this. Um, there's also, uh, you know, from those 40 demands is 62 different links.
Um, so I like to kind of click through here and just see what they are and what you're seeing here.
Um, this is the referring page, the article that referred to them. Um, and then this is the anchor texts, like the texts that was highlighted and the URL, which is the shared with pride set.
So, um, we can see that this is about awesome packaging designs. That's kind of interesting. Um, this is the corporatization of pride month.
Maybe we don't know the year in here. Uh, that's kind of a negative article, um, you know, but like we would want to appear in packaging design reviews.
That's pretty cool, especially if we have a good package, uh, in terms of, uh, the, um, you know, the, the, uh, uh, the, the skincare products that we, that we created.
Um, there's also here 15 awesome limited edition product collaborations. Um, and then, uh, some article hit. Um, so there's some opportunities here we can go through.
Um, and, um, you know, we can contact these people. So if we click through to here and see the article, um, we can see, you know, we search the page for Harry's.
Um, we can see what they're read about. Um, and it looks like they're kind of listing different ones. So we have Harry's who gives us scrap.
So, um, this might be the sort of article that we want to get included into. And, um, if we, um, if we have a package that, um, in this case, uh, you know, fits the bill, we can contact the author of this post.
And so this is massly, um, Alrighty. And, um, you can usually click through and kind of, you know, find that Twitter or, um, you know, the, your, uh, their email.
Um, so, you know, you can just search for, uh, dialogue. Um, so that's like a little bit more about, uh, this is her personal website.
Uh, so she's on my crack as a PR kind of, uh, platform. So, um, yeah, ZoomInfo has a contact details.
It looks like. Uh, so, uh, yeah, that's, um, that's interesting, right? That's how you kind of go through, um, and then you can contact her and say, Hey, I noticed that, uh, the page you're linking to, um, isn't, isn't working anymore.
Uh, but, um, you know, I think actually, you know, we, we have a new package, uh, that you might want to check out that we might want to include.
Um, let me know if you want me to send you, um, you know, some of the products for free, just so you can check it out and, uh, you know, give, give it to a friend and see what they think, because it's a men's product in this case.
Uh, so, um, you know, that's a really good way to build that relationship, even if they don't link to you directly here, I, they might, you know, tweet about you or, um, you know, they might, uh, include you in a different article that they're working on.
Um, and, and you can't predict any of this. You also can't like specifically ask them, like, uh, pay them to do this, uh, because that goes against Google's guidelines.
Um, you know, but, uh, I think, you know, this is an opportunity to find someone who is looking to a competitor that is running about the space, um, that, um, you know, potentially will be influential and willing to, uh, to link through to you, uh, in the article or in a few trout skills.
So that's really what we're trying to find. Okay. So that's, that's one opportunity, uh, let's move on from Harris and see if we have anything for Jack Black.
This is the school, this is the leader here. So this is telling those that total, um, backlinks is good to hear best buy links again, um, and then filter it down to four, four.
Okay. So no, uh, for a force for their websites. So this is pretty good for them, uh, bad for us.
I mean, so they, uh, you know, whoever is managing our website, uh, you know, it hasn't made some of these messy changes.
Um, so they haven't taken off any of the pages in their site. So that's good. I would also look internal.
Um, it looks like none of the internal links are for, for either. Um, so that's, that's a, you know, that that's going to happen like, uh, relatively frequently.
Um, there, isn't always an opportunity to kind of have to search for the next one we're going to look at is not an, a goats.
Uh, this is the really trendy one, and we're on best by links here. So let's filter it down for, for, for, and here we go.
There's a lot of two TCS, so they have this face, grapefruit face cleaner. Um, so, uh, that has, uh, some, uh, it looks like they've changed some of these, uh, the URL structures here.
So, um, you know, him, men's journal fashionista.com, Jacob, Jacob, and Sebastian. Um, so it looks like there's a lot of these companies that have, um, you know, looking through to this and to hear that you're washing your face or wrong, et cetera.
Now in here we go. So they just have a few links to the bottom, so we can go to them and say, Hey, by the way, that link is broken and also, you know, did you want to look at our product, um, and, and see, see if you like it, if we have a face cleanser and then, um, and then maybe they'll add us as the fourth one here.
Uh, so I think that's, uh, that's like pretty straightforward. This is like ideal. This is exactly what we wanted. This is a lot less complicated than the, uh, than the pride week, uh, opportunity.
Um, the, uh, the other thing I think is useful to do is just see if they actually have that product elsewhere on the site.
Maybe they just change the URL, I guess. So they do, um, this is, this is useful to know just because, um, if they, if that product still exists, uh, then it's, um, I think like good practice to give them the updated link, uh, to the, uh, you know, remember, like you're not trying to do this at scale you're, should we get good quality links from influential people and build that relationship?
Um, so if you think about it that way, it's worth the extra effort to say, Hey, here's the correct link for that grapefruit face cleaner, because he now doesn't have face in the Euro, whereas before it did have face, um, and now it has the different fluid ounces in hairs off.
So, uh, that's, that's very useful. Um, I think that the, um, if, if the, this product doesn't exist, uh, then that's an opportunity for you to change the messaging and say, Hey, they didn't actually sell that anymore, but we think we have a good replacement.
So think about that. Uh, it's, it's, you know, trying to spot these opportunities. Um, but it's not really about scale.
It's more about quality. We're trying to find, um, you know, relationships with people who are LinkedIn to similar content and therefore, um, you know, this is just an opportunity to reach out.
Um, it's not like the end of the relationship. Um, so try and invest as much as you can into each one.


What part of Ahrefs do you need to go to in order to find broken links from a competitor?

What are the benefits of broken link building?

How would you pitch a broken link backlink opportunity?

How do we build authority

The difference between a website that gets 200 visits a month and 20,000 is backlinks. It’s night and day: if you don’t have authority on a relevant topic, Google won’t rank you anywhere near the first page.

GoolyBib Headquarters – Website Planning Meeting

Your CEO just read news online that Google was going to deprecate Optimize, their A/B testing tool, and is worried: his Q2 plans depend on that tool!

Even if the rumors aren’t true, they’re spreading because there’s something to it
Google hasn’t updated Optimize in years
They won’t offer a free tool forever
So let’s use this as an excuse to migrate off
What can we do to execute our testing plan if 


As the internet marketing world changes, so do the tools and solutions used to optimize websites and campaigns. One of the biggest changes in coming years might be a move away from Google Optimize. Google Optimize has long been the go-to solution for setting up A/B tests and other experiments, and is used on 500,000+ websites. But with Google making changes and discontinuing the service, many are left wondering what to do. 

While Google Optimize might be going away, you can still set up A/B tests and experiments with the same degree of accuracy and ease in another free Google tool - using Google Tag Manager. Google Tag Manager (GTM) is another free service from Google that allows you to add, update and manage tags on your website. It’s a powerful tool that can be used for a variety of tasks, including setting up A/B tests and experiments. 

So how do you set up an A/B test in Google Tag Manager? Let’s break it down. 

The first step is to create a new tag in GTM. You can do this by clicking the ‘New Tag’ button in the top right corner of the screen. You’ll then be taken to the tag creation page where you can choose the type of tag you want to create. For an A/B test, you’ll want to select the ‘Custom HTML’ tag type.

Once you’ve selected the tag type, you’ll need to enter the code for your A/B test. This is where you’ll use JavaScript to randomize visitors, and then apply filters to conditionally show each version. For example, you could use JavaScript to randomly assign visitors to either the ‘A’ or ‘B’ version of the page. 

You can also use GTM’s built-in variables to create the filters for each version. For example, you could use the ‘Page Path’ variable to filter for visitors on a specific page. Or you could use the ‘Referrer’ variable to filter for visitors from particular sources.

Once you’ve finished creating your A/B test tag, you can then set up triggers to determine when the tag should fire. GTM has a variety of built-in triggers, such as page view, click, form submission, etc. You can also create your own custom triggers if you want to be more specific. 

Once the tag is created and the triggers are set, you can then click the ‘Submit’ button to save your changes. After this, your A/B test will be up and running. The only thing you won’t have is the custom user interface that let you click and edit to make changes to the page, however that was only useful for simple tests anyway.

Test Implementation

0:01 All right. How to ab test when you can't use Google Optimize. So this rumors right now that Google Optimize is going away.
0:09 And that sucks because one of the cool things about it was it was free, easy to use and it let you do interesting stuff.
0:16 The webpages, without knowing how to code luckily you only need to know a very little amount of code in order to replicate it.
0:24 So this is maybe even a good idea to do in general. I found with these ab testing tools, there's like a flicker.
0:32 So when you land on the page, it's like not fast enough. Cause that's to load all of its crazy library, and then it makes the changes.
0:38 So I like to do this anyway. So say you wanted to change this, right? So one of the cool things you could do in Google Optimize and then optimize them for it.
0:49 So you could like click on it in the tool and then change it to whatever you want. You know, growth hackers, whatever, and it would change.
0:58 And then you'd go, great, this is what I want to test. And you'd hit the button. Well, you know, that's not possible anymore.
1:04 And obviously if you do this in the browser and you refresh the page then you know nothing's gonna happen because it's just gonna reload.
1:13 But you can still kind of plan your tests out this way. So if you say you wanted to change the button color you inspect then you can see that this has an idea of intent indicator.
1:26 You can actually click on it and say copy selector, that's useful. And then you can, you know, again, you only need to know a little bit of code if you just type in document and then call it like query sector.
1:47 And then piece that in. Then you, oh, did I do something wrong? Oh, query. Oh yeah, you didn't need this section.
2:00 Oh, here you go. Then you get your button. And you know, here it is. So if you want to kind of change that button, you can actually just make changes directly here.
2:10 So you can say like, what's this, you know, style you background color, color to be like green. Why not? It looks terrible, but there you go.
2:26 If that's what you wanted to do you could also change it and say, I want the text to be free access.
2:34 And maybe that's the ab test that you, you know, your heart desires you wanna run. So stupid example, but like, that's what you want to do, right?
2:42 So how do we actually make this persist? Well, the good thing is we have Google Tag Manager and Google Tag Manager can help us make changes in JavaScripts on our website without needing to deploy anything to our website.
2:55 We don't need to bother developers to do this, and we probably should tell them, right? It's not gonna bother them to tell them that you're changing something.
3:03 It'll bother them a lot actually, if you change things and you don't tell them. So just be careful here. Cool.
3:10 So how do we make this actually run on the website? Well, we can go to tags and we can set up you know some code.
3:22 So you know, you can actually don't even need all this stuff, right? Here's some code I made earlier. You know I, when I'm testing the stuff, I, I like to just go have an alert, which is like, it's loaded, right?
3:37 So you can just see that whether it worked or not pretty easily. So here's the tag, right? And don't worry about the triggers and stuff right now.
3:47 But if, if you wanted to, you can just like save this and then when you hit preview, you go through to your website.
3:56 If you load that page, then when that loads should say, yeah, look, it's loaded, right? And then you know, we've made this change here, the green button.
4:09 Cool. So that's not an AB test, so that's just making a change. But how do we make an AB test?
4:14 So if you look in here, I have the version B, right? What does that mean? So I'm only triggering it based on the version.
4:23 So if we look at this, it says on page views, but not all page views, just some page views. And the only one I find on the version when the page views the version matches, this is using regular expressions.
4:35 Don't worry too much about that slot. And then either slot, 6, 7, 8, 9, 10. So if you wanted to do this, you could do something similar.
4:45 You set this up. Now how does it know what this version is, right? Where does that come from? Well that is is just coming from if we go to variables and if we go down to custom variables we made a little custom variable for this.
5:11 So let's just look at here, yeah, version. And this is the code that we use to do this. So we have to randomize the visitors for an AB test, and that's what this part is doing.
5:25 It's giving you a random number times 10 slots. So that's gonna give us, you know, slots zero through nine. And that will assign a random number on every page, you to everyone.
5:38 But then we're not gonna get the same number again. So if I visit another page or if I refresh page, I'm gonna get something different.
5:44 So how do I persist it? Well, that's what the local storage thing is. What that's doing is, is first looking in local storage, which is just in the browser, you know, is there a version for this person and it's you know, gonna get the version and then pass that along.
6:01 And if there isn't a version, then it's gonna create one and then store it in local storage for me. So if we go back to this page, we right click inspect and then we go through the application.
6:15 We can see we are in slot nine. Now what if I change this to slot two? Lemme just refresh the page.
6:24 Ah, not in the test. Cool, let's inspect again and let's change it back to slot nine and then refresh, and then refresh.
6:43 Mess up. Slot nine. Yeah, I think I broke it. Oh, yeah, here we go. <laugh>. So it opened up a different page.
7:00 Yeah. So now I am in the experiment, right? Cool. So that should give you a really good idea of how this works.
7:09 It just randomizes the users, stores it in local storage, and then if they're in slots, 6, 7, 8, 9, 10 then you know, you can show them a version.
7:20 And and that's really all you have to do. You can obviously run as many experience as you want, like you could you know instead of having this trigger for version B, right?
7:32 You can actually have you know, a multiple triggers. You can have version A, B, C, D, E, F, G up to 10, you know and and, and then you could run multiple different experiments or like multiple different versions.
7:46 It's really up to you. But hopefully this gives you a good mental model for how that works. The other thing to think about is in terms of tracking you probably want to add, like, you know, add something to your, like if you're doing GA for I am you know, then you probably want to add something.
8:12 I think, oh yeah, no, it's not here. It's user properties, right? And user property. I'm just gonna make this version oh, yeah, sorry, I got this wrong to, oh, what did I do?
8:32 That should be version, sorry. So that's gonna input the value of the version for that user. And then I'm just gonna call this version.
8:44 So it's a property name. And if you do that now when you're in Google Analytics four, you be able to filter down to those users based on their version and then report On it afterwards.
8:56 So you know, this is a simple way to do that in any analytics package. So you know, there's always like a user level property you can set.
9:05 Cool. And that's how you reporting your OB test now. So you can just filter down and say like, people who got to that page and then what version were they in, and then whether they did, you know, any conversions afterwards.
9:16 Cool. Hopefully that was helpful.

Multiple Choice: What does the tutorial describe as a free alternative to Google Optimize for AB testing? 

How can you track the results of an AB test in Google Analytics?

What does the tutorial suggest you do to make sure the changes persist?

What is the purpose of using a random number times 10 slots in the tutorial?

What code do we need in order to make changes to the button?

I’m hearing rumors that Google Optimize is being deprecated (classic Google). If that happens it looks like over half a million websites are going to be without a solution that doesn’t cost $$$ enterprise money. 

Can we run tests without Google Optimize?

We're in the late evening at GoolyBib Headquarters, and our CEO, Gustav Gieger, has tasked us with finding a way to use Inpainting to edit and manipulate images using Dolly, Stable Fusion, and Mid-Journey. It's a tall order, but we're confident that with the right parameters and control net, we can segment anything we need to get the desired results.

We need to find a way to use Inpainting that allows us to make complex edits quickly and efficiently. 
Let's figure out how we can leverage control nets and parameters to segment images and get the results we want.
We need to start by testing different strategies and hypotheses.
Let's come up with a plan that will help us find the right combination quickly, without leaving a potential winner on the table.

In painting is a technique used to remove or replace unwanted elements in an image. It is a powerful tool used in digital art and photography to create stunning visuals. In this blog post, we'll be exploring the basics of inpainting, the different types of inpainting, and some advanced techniques to help you get the most out of your inpainting projects.

Inpainting is a process whereby unwanted elements in an image are removed or replaced. This is usually done by “erasing” the unwanted elements and then “filling in” the deleted area with new content. This content can be anything from a new image or texture to a different color or pattern. Inpainting is commonly used in digital art and photography to enhance an image, correct errors, or to create a totally new look.

There are two basic types of inpainting: image inpainting and pattern inpainting. Image inpainting is used to remove or replace unwanted elements in an image, whereas pattern inpainting is used to create unique patterns or textures. Image inpainting is typically done using software such as Adobe Photoshop or GIMP, while pattern inpainting is done using specialized software.

When it comes to inpainting, there are a few advanced techniques that can help you get the most out of your project. For example, you can use control net inpainting for more precise results. This technique involves using a single click to define the area to be inpainted, which then creates a mask that can be used to more accurately inpaint the desired area. You can also use segment anything inpainting, which allows you to quickly identify the area to be inpainted. This technique is especially useful if you are inpainting an object or person in a photo.

Finally, you can use the latest inpainting techniques such as stable fusion and mid-journey. These powerful tools utilize AI to generate highly realistic results. They also allow you to inpaint both inside and outside the frame, which can be useful when adding new elements to an image.

In conclusion, inpainting is a powerful technique that can be used to enhance, correct, or create totally new images. There are two basic types of inpainting — image inpainting and pattern inpainting — and a variety of advanced techniques, such as control net inpainting and segment anything inpainting

Advanced Inpainting

All right. We're going to talk about in painting. This is a Dolly to. And just going to upload an image. It's probably the easiest way to in paint. So it would just take her here's a. 

An image that I. 

Have uploaded. I'm going to click edit. And this is from the great Gatsby. And maybe instead of. This. A specific dress. Maybe we want different dresses search. To roughly erase. These parts here. And you don't have to be really strict and exact because AI is magic. 

And it does a lot of the work for you, which is nice. But yeah, here we go. So I have. 

Erased a lot of the image. And then and this is all within the generation frame. And she's going to put in here. A. Let's say a. I see. 





As you. 

Really. Address. 



Let's see. Great. 

All right. You can see the tip here is telling you. Once you to describe the whole image, not just like containing a Flamingo if we just put, containing your Flamingo as the prompt, that would not work very well. But if you put the whole image description and also what you need in here, then. 

That works much better as. And that's what I've done here. So I said, as you're in yellow, swirly flapper dress. Great Gatsby. And then just going to hit, generate. And it's going to basically just generate AI within. This edited out version here. And this can be done. Not just inside the frame, but also outside the frame, they call it out painting. 

We add another. Section two. And I'll show you that in a second, but it's basically the same mechanism. It's really just creating. Space creating white space and then generating within one of the things in painting does is. It takes the whole rest of the image into account, which is really helpful. 

And and then it also gives you multiple variations. This is the original image. And then here we go. We've added address this here and that here. And that's Shannon makes it look naked. That doesn't look very good. But it's not done a very good job. I think generally, because Dolly is not that good. 

Of an image generation model compared to the state of the art, like stable diffusion and mid journey. Mid journey, unfortunately doesn't allow him painting doesn't or it doesn't support it and do it right now. What I tend to use is several to fusion and I have this running. If you if you're going to use stable diffusion or use it through the several diffusion, a one, one. 

One in painting model. So I just have this running. If you need to install it, we have another course on that. The stable division web UI. And it has instructions for how to install but once you get it running, you can actually just use a local models. So this is a specific in painting model here. 

And we're just going to put in. I'm going to go to image. And upon in this image It's fun. And to do the same thing. So let me go in paint. Sorry. And then put that image. 

Here we go. 

And then you start drawing. 

And it's just going to get rid of. These parts of the dress. 

Okay. And then we'll just get rid of this. So the address of here. So you can see it's a very similar interface, except this is all open source and free, which is nice. So you can make a lot of mistakes if you need to. All right. So that is that. And then we're just going to put this same prompt in here and see how it does. 

So I would say, keep it simple at first and changing the parameters. Just see how it does, and then we can make changes to their practice based on whether the image completes well or not on that first go. 

Okay. So this is generating. 

And this is just generating one by default, but you could have it generated more here. If you like. Okay. So I think this actually already better than what we got from Dali, which is nice. So we haven't really done that much prompt engineering. The there's a few different things you can do. So one is we're not really getting much 

From this dress. So I'm just going to press command up. And this is going to basically emphasize it a little bit. I do that. 1.2 waiting. You can just do this manually as well by putting in brackets and putting a colon. So that's I think we're going to do that. Oh, sorry. I just missed out this pie here as well. Sorry. 

Let's improve that. And then we can also choose a few different things here. The mask blurs, how much we blur the image that mask mode. We want to him. We want to implant the things that are mast, not but if we were trying to generate the rest and keep the same dress, then we'd click and pay Namaste. 

The mass content parameters here. This does is you could either fill a witch. Generates, basically it fills in this gap with the same colors from nurse to image. So that's gives it a different starting point. Also original. So it starts filling in with the original image. And then have late noise. 

Which basically fills it with a lot of image noise. And that kind of makes it this is really useful if you want it to be very different from. What we had previously. The other thing you can do here is you can say, I want to inpaint the whole area, or just like only the mast. 

Area as we can try only max. And then the CFG scale. I'm going to put that up a little bit to eight. This is how much it follows the prompt. Versus being creative itself and then denoising, so it's at 7.5. I'm going to put it up at 8.50 6.85 and see how that does. That's going to do a de noising is basically how much shit. 

If you have a high denoising. Then it's going to add a lot of noise to the image before it diffuses hours. If you have low denoising, it's not going to add very much so essentially if you want it to be very similar to the original. Then you would use original mass content. 

A whole pitcher and low denoising. But if you if you want it really. A different, like here, we're trying to input like a whole new, different types of dress. Then I would say We only use these sorts of settings and it is a crap shoot. So like this I think is too similar to the original dress and that's a problem. 

Cool. I'm going to generate, but I'm also going to generate and the material for. 

Four images. And this is going to do them one at a time. If I wanted to generate for all the same time, that sucks up a lot of V Ram. It depends on your computer, but it might be quite slow. So just going to. Do them in batches. So just do one, one per batch. That's going to run. You can see that running here. 

We're going to see how that does. There's one other thing we can try if this doesn't work as well, which is something called control net in painting. And you can see here by the way as well that I'm using a specific in painting model. If you want to get that. The, I can show you how to get that while we're waiting. 



All the, Everything you need to know about automatic 11, 11 is in here. Super useful. So installation. 



Oh, I'm on the wrong one. Okay. 

Just going to find the link to the most. Here we go to his there in painting model. You could just Google it as well, but yeah, this is a, it was released by runway ML there originally. Yeah, it helped create a stable diffusion. And you can get the model file in here, files and versions. 

And then you just go. In here it's like in painting. Checkpoint file C K P T. 

All right. So this is running. 

And I think it's almost done. 

All right. First thing it does is give us a grid. So at least it's a yellow dress, or it's just kinda nice. There's some blue as your blue. Swirls here. Addresses and much better. There we go. This is like so much better than what we go from Dolly. It's actually form fitting. 

I think that's really good. That's much better. And then, we could, we might want to just. And do a bit more prompt engineering care or whatever it is. We needed. This is swirly, and then we could say maybe 

Vortex. Whirls. Inspire heavy. Brush strokes. 

Bye. 

Sorry. 

All right. Let's try this one more time. So you can either save images here. Or you can click save, or you can actually send it back to him, painting and start again. But also it's in your file system. So if you go into your folder, by the way, this is where you save the new models. If you download them, like I have the painting one in here. 

It's in models, stabled a fusion in painting. And then if you want to see any of the images you output, if you go in here it's this would be under image, images. And then this is today. And then that's the latest batch of images here. 

If you ever need to get them. That's where you'd find them. So this is running. 

Just going to let that run and then I'll show you two. To other more advanced concepts within painting. One is control net and the other is segment anything. 

And those will be in a different different module. A different course. 

Here we go. This looks much better. 

So that's like much closer to what I wanted. We've got some prompt engineering going on here. That's cool. But yeah, you can keep messing with that. If you want, you can change to the sampling steps, et cetera. The other the other things you can do down here. Control net and painting this we're not going to go into control net right now, but it's just a way to make the image match much closer to the original image. 

It works exactly the same. You just upload an image here and then you want to choose the in paint model. And then run that. And it should perform better. That, that should be, keep it more visually consistent with the rest of the. With the rest of the image, if that's what it, what you're trying to get across. And then a segment, anything it's just a way. I'll just quickly show you this. Same with anything it's like just a quicker way of finding the thing you wanna in paint. So I just clicked once on once in the dress on each side, and then once down here and then just going to preview the segmentation, see if it works. And what it should do is it should just mask out only those parts. Yeah, here we go. Yeah, here we go. This mask is worked. So that is That's just the dress, right? So it's picked out just the dress here. Which is very helpful. So that could make it much more precise. When you're doing this. And you choose the favorite mask and then you'd run it. And it. would just in paint in this area. That's much easier than you doing it yourself, right? All right? Hopefully that's helpful. There's a lot more advanced stuff. You can get him here. You can go much deeper with it. And it is like an intuitive process that takes a lot of experimentation to get really right. So just keep trying and looking at the parameters, if you don't know what they mean.

What does the CFG Scale parameter affect?

How can we find the images we output from the tutorial?

What are the two more advanced concepts mentioned in the tutorial?

How can the image generated by the AI be saved?

One of the coolest demos of AI in recent years is Adobe Photoshop's generative fill, however it doesn't work in many situations and the underlying model is weak.

I want something like Generative Fill from Photoshop, but I want it to work

GoolyBib Offices – Product Growth Standup Meeting

The CEO is at the product growth standup meeting. He's discussing the challenges of generating a long-tail keyword list for the company's programmatic SEO program. His latest obsession is AI.

We need to find a way to generate a long-tail keyword list for our programmatic SEO program, and I thought… why not get ChatGPT to do it?
My daughter showed me it in the elevator on the way up and it’s remarkable
Is anyone here using it? Do you think it can give us a good starting point for the baby recipe content site we’re building?
Look into it and find out for me

Programmatic SEO is a completely different approach to traditional content writing. Instead of hand-picking a few keywords with decent volume and low difficulty, you’re looking for an entire category of long tail keywords that will add up to something.

The key to successful programmatic SEO is to find the keywords that don’t show up in the keyword tools. This requires a deep understanding of search intent, user behavior, and how people talk about the topic at hand.

The best way to start is to look at the top 10 search results for a given topic. These are the pages that are currently ranking in the search engine results pages (SERPs). Take a look at the content and note the common words and phrases used.

Next, use a keyword tool to find related keywords and phrases. Look for keywords that have decent search volume but aren’t overly competitive. These are usually the long tail keywords that you’re looking for. You want to look for niche keywords, which are highly specific searches that are often overlooked, but they must be part of a wider pattern. 

For example you might not rank on “cheap flights” (head term) or “cheap flights to barcelona” (mid term), but there’s likely to be less competition and volume for “cheap flights to BCA”. More importantly, there’s a long list of airport codes, so having a page for every code can add up to a lot of search traffic. The category must be something people who buy your product would be interested in, otherwise the traffic won’t be worth anything.

Typically when you have a category that fits well, you then want to take that to an AI tool like ChatGPT to build out a larger list. ChatGPT can be prompted to add additional context from its training data and output it in table format, leaving you something you can easily parse into a spreadsheet.

Once you’ve gathered your long tail keywords, you can begin to create content around them. Create content that answers user questions and provides useful information. Be sure to use your target keywords throughout the content when generating the post.

Programmatic SEO involves a lot of research and analysis, but it’s worth the effort. By finding the right keywords and creating content around them, you can improve your search engine rankings and drive more organic traffic to your website.

Programmatic SEO

00:01 Hey let's walk through how to create your long tail keyword list when you're doing programmatic seo. So I'm using chat g p t for this by OpenAI.
00:13 You could also use G P T three you know, the powers chat g p t in the background as well.
00:19 So I'll, I'll show you that in a second. But my starting point here is usually chat, G P T. And the reason why I use this to start with is you could use keyword tools like ats.
00:33 I found in general that they don't have very good keyword coverage on like long tail keywords. So, you know, for example, I once built a, you know, website that listed all the different recruiters in tech.
00:48 And when I did keyword research I found that basically there was nobody you know, no traffic for these keywords.  But it's just because ats can't crawl the entire web.
00:58 The, the sample size isn't massive, right? Of, of people browsing the web, so they don't always get like the full picture.
01:07 And also I think in general anything that has a lot of traffic on one of those keyword tools is gonna be looked at so many times by competitors or, you know, other people are trying to own the space.
01:18 So you know, almost by definition, if it appears in there, then it's gonna be saturated pretty quickly. So what I try and do is come up with keyword ideas based on my own intuition, my experience, and, and also talking to customers to see what they search for.
01:34 So, you know, re reason we did the recruiter thing is we found that a lot of the people we talked to in tech said that they would google the name of a recruiter to check if they're legit before they you know, before they responded to them on LinkedIn.
01:50 So you know, we created a website by scraping LinkedIn found you know, all the recruiters names associated with the different companies they worked for and then also gave kind of a simple rating system.
02:02 So that's, that's one example. In the example today, we're gonna try and do this with baby food like recipes. So you know, I, I think like Chak now is where I start to generate ideas usually.
02:18 You know, rather than kind of using that to write the finished product, I think it's really good for ideation and kind of quickly checking whether something has interesting stuff.
02:27 Cool. So I'm just gonna say, you know, list you know recipes list of dishes that you can cook for. So this, this product is a baby bib.
02:47 And therefore they want to, you know, rank on terms that are related to babies. So there's certain things that their customers would be searching for but not directly commercial necessarily, right?
02:57 So your traditional SEO strategy, you would maybe write a blog post or, you know, write some content specifically targeted to specific keywords.
03:07 This isn't really what we're covering here. What we want to do today is like, you know, have a category of keywords, something that's like very, very you know broad in terms of like, it's, there's gonna be lots and lots of keywords in that category but the each individual keyword won't get that much traffic.
03:26 So it's called the long tail. And this is a good example that I came up with. So, you know, there's a list of different dishes that you can cook for babies.
03:38 We do have to be a little bit careful here because, you know, AI is not always correct and it warns you that of that you know, says you know, the babies shouldn't have specific foods.
03:53 You know, some of the recipes that we come up with might not actually be edible <laugh>, right? So we'd have to try to check that.
03:59 But I think it's very good for ideation as long as you have like a human checking it afterwards. Cool. So we have, we have a list of dishes here.
04:08 So are period fruits, period vegetables, pure meats, period legumes. So that's pretty cool. So it looks like there are certain dishes that we could potentially you know, do for babies.
04:22 I mean, I can bring up interest here and then just kind of show you what traffic those keywords get. So just piece that in here and search United States.
04:36 Okay, so these are the, the core keywords, like Turkey, you'll get <inaudible>. So it looks like some of these keywords have a lot of search traffic.
04:45 But it's saying yeah, here we go. So like, period meses. Yeah, so, you know basically like there's not that much.
04:59 Here we go. I think it's past it out a little bit funny. So let's just see, make sure this is correct.
05:19 Rentals. Yeah. So, yeah, it's still kind of keywords that they couldn't find. Here we go. So here's a good example, like oatmeal or rice cereal, like that keyword specifically doesn't have a lot of traffic but maybe ind individually it would.
05:35 So anyway, I mean, you can go this, this path, but I think that these tools are very good for finding like, which head terms Turkey for example has a lot of traffic, but if I say you know, Turkey recipe that's gonna have a, a decent volume for babies, like almost it says no traffic, right?
05:59 So this is kind of what I was saying. So here's our list. If we have like a basic list you know, but like, that's not all we need, right?
06:08 Like, we need a lot of other information if we're gonna create a programmatic SEO page, we can't just like have the word y'all get, right?
06:18 So what, how do you get kind of structured information out of out of open ai? So you know, what you can say is like if you said, you know, that you present this in a table with you know, the dish Difficulty and,   The ingredients, The dish,  Then it'll probably tend no, oh, no, sometimes it does, Sometimes It does say yes.
07:00 Okay. Yeah. So if you, if it does say no to you then one of the things you can do is like ask it to format the table in a specific way.
07:14 So like, you know, usually say like Create A list in the following format.  And it'd be like a Dishy ingredients.
07:32 And then you can also,  Like an example input, and there'd be Like period roots, it's, And then example, Output fruit Difficulty, Ingredients,  Ingredients would be apples, vanilla And Difficulty would be easy.
08:08 So you could kind of format it in that way. You know, that's that's like a pretty straightforward you know, that's like a pretty straightforward way to get structured data out.
08:21 In this case, it's actually given me you know, some good structured data in the table. You know, when we copy this and paste it, it's gonna be, it's gonna work out okay into Excel.
08:33 But just to kind of show you, I'm just gonna add like another thing on here. So these are all easy dishes.
08:42 Like maybe maybe we could also have like, time to cook and then and then have like a recipe. And then let's have like a health rating.
09:09 Cool. So, so we've got dish, and then we've got difficulty health rating. So let's had a health rating, so pure fruit to be five.
09:24 And then let me see ingredients. And then time to cook, say five minutes. Then recipe would be pause One, Pause pair, and, and that back slash n is like a new line.
09:56 And then we're gonna say two and four, five minutes. Cool. Let's make this five outta five, just so it knows that, like that's the healthiest.
10:12 Or you could also say like, very healthy, whatever. So let's see what it says. Sometimes if it, if it's wrong, then you know, you just want to stop it and then read it.
10:38 So maybe let's say let's create table in the format. Here we go. That's working. There we go. It's kind of working great.
11:15 Nice. Okay. All right. So now we've got a structure and we say like use the same structure but list 10, 10 dishes.
11:39 Perfect. So it is the good thing about chat GT is that you don't have to keep repeating the instructions. It understands what happened in the last, you know, few bits of the conversation which is cool.
11:54 So what we're gonna get out of this now is kind of basically a spreadsheet of the different data we're going to use you know, for these you know, for these recipes, right?
12:04 So like when we go to build our website you know, and, and write blog posts about these, et cetera then we'll have like a list to work from.
12:14 And you could just inject the recipe directly into the page. You could also use GPC three to then, you know, kind insert this data.
12:23 Just gonna stop generating.  Here we go. Cool. Hmm. I guess I'm having some issues. <laugh>.  Just few, five Takes a little bit of coaxing.
12:58 Here we go Now since you were working. Cool. So this is a great way to get structured data out of chat, G p T three.
13:19 And you know, once we have this you know, in this format, then we can kind of put it in Excel and, and use that you know, for our SEO site.
13:29 So these would be different fields in our cms you know, and then we would get a writer to, you know, write up more information to include.
13:40 So so that, that'd be really helpful to, to begin with. And it kind of gives us a good kind of list that A we know that, you know, the AI is aware of, and, and b is something that's obviously popular on the web, if the AI is aware of it.
13:54 And then three you know, it's kind of a quick and easy way to get this structured data. So we might remove some of these if we don't think they're very good.
14:02 We might add more whatever as we need to. The other thing we can do here is just kind of be targeted.
14:08 So we say like, these are all easy and healthy, Easy And healthy And you provide options. So sometimes issues, all right, so this is a good opportunity for us to kind of copy here and then just gonna, you know, create a new spreadsheet and then a, so I did a command or control shift v you know, command shift fee if you're on a Mac control shift fee if you're under pc.
14:52 Cool. So we have we have these now and chat gpt is running into some issues. Hopefully you aren't running into the same things but just to kind of show you that you can do the same thing.
15:04 Hopefully GT three isn't down <laugh> but let's see. Okay. All right. So we're gonna go into the playground here. And this is using the same kind of engine in the background.
15:19 What we're gonna do is we're gonna dial up the tokens to 2048 and then we're gonna say here we go.
15:36 Okay. So create a table in the following format. So I like a table of food of dishes for babies In A table in the flowing format.
16:02 Cool. So that is all you need to do. It's the same sort of thing as you know, chat gt. Oh, okay.
16:17 So it's continued here. So what we can do, there's two different things here. So one is so this is a little bit dumber in terms of the user interface.
16:41 So sometimes we need to kind of present a list first. Actually let's get from our spreadsheet. Let's get a list first.
16:51 And then we can say input, and then we're just gonna submit, and then here we go. So now it's, you know, giving us the output for this input, which is great.
17:09 So this is, you can see this is a types a lot faster than chat gt, so it can be a little bit better if you're doing a lot.
17:16 But then now we have this, we can kind of just build on it and then say, okay, gimme some more input, right?
17:25 So I'm gonna say stop token. Sometimes it gets confused and stops. So we're gonna but these hashes at the end, and then we're gonna tell it stop when you just get to this, because sometimes it just stops when it thinks it's finished the sentence.
17:49 And that isn't exactly like what you want <laugh>, right? Like we saw here, we were hitting it and it wasn't generating more ideas, right?
17:59 Here we go, Yogurt. Okay, so it doesn't work. List. I dunno, 10 more. Here we go. I got it. So this is the, the work with prompt engineering.
18:25 You know, it's sometimes takes a little bit of coaxing. I'm sure if you're watching this in the mid to late future <laugh>, this will be a lot easier to do and it'll kind of understand.
18:37 But you know, this is really like a mental model. You need to think of that, you know, you're just, just like working with a human.
18:44 You're trying to give it input in a way that it understands. Give it examples and then let it rip, right?
18:49 So here we've got, you know, another 10 here. Like we can just say, okay, you do 10 more, right? We can just kind of keep going.
19:03 Let's use our stop sequence and then it's gonna keep going. Hopefully,  Let's try. There we go. Yeah. So you do have to play around with it all a bit once you ha you know, have something that works, you can save it as a preset.
19:31 We also view the code and use that as a as a way to like you know, run this as an API call.
19:40 So if you wanted to kind of turn this into like some, you know, something that you can call externally from your code and, and kind of work into your product, then you can do that.
19:49 But yeah. Now we have a really good list here of oh, sorry. If I formatting, I have a really good list.
19:59 Oh yeah, here we go. So this is one last thing to show you. Sometimes the formatting doesn't work out great.
20:06 So in which case you're gonna go to data and then wanna split text to columns.  And rather than detect automatically now, let's just put the pipe in there.
20:18 Here we go. Cool. So now that, now that knows you know how to split those out Beth, why you want to do things as structured data, right?
20:29 Because then it's easier to pass and turn it into something like this. So by telling it to you know, split by the pipe, then the pipe isn't really ever used in normal text.
20:43 So you know, that's really gonna help you. You know, you're not gonna have any issues you know, with this sort of thing you know, like splitting it into different data.
20:55 So that's how, that's how that works. You know, you can pass this now to your writing team and say like, here are the different recipes I want you to write for, you know, dishes.
21:06 And, you know, then potentially have someone who knows how to cook you know, or a health expert, nutrition expert. Just review like the, the ingredients and recipes are correct because AI can sometimes hallucinate and, you know, put the wrong things together.
21:20 Cool. Hopefully that, that works. And that's kind of taught you a little bit how to work with AI in order to generate, you know, large sheets of data cuz you know, you can keep going with this and, you know, do a hundred or 200.
21:31 Alright, hopefully that was.

What type of keywords are beneficial to use when creating a programmatic SEO page?

What is one trick to get structured data from OpenAI when it doesn’t want to give you a table?

What is the main reason for using Chat GPT to generate keyword ideas?

What is an example of a keyword that might not have much search traffic individually, but in aggregate the category should give good volume?

What is an example of a dish that can be cooked for babies that wasn’t in our list?

Name one of the advantages of using chat GPT 3 to generate structured data.

Most people don’t realize programmatic SEO is a completely different game to traditional content writing. You don’t hand-select keywords with decent volume and low difficulty: you have to find a whole category of long tail keywords that will add up to something.

Generating a long-tail keyword list

MARKETING WING - CAMPAIGN PLANNING SESSION

Sally, our Head of Marketing, wants us to dive deep into analyzing landing pages and competitor landing pages using this powerful tool.

I'm really excited for today's session, everyone. 
Landing page analysis is such an important skill for us to have, especially as we're constantly striving to improve our campaigns and stay ahead of the competition. 
With GPT-V, we'll be able to analyze landing pages with precision and efficiency. 
Trust me, this skill is going to make a huge difference in our marketing efforts. 
Let's dive in and become landing page analysis experts!

Landing pages are crucial for any website as they play a significant role in attracting and converting visitors into customers. Analyzing landing pages, both your own and your competitors', is essential to gain insights into what is working and what can be improved. In this blog post, we will explore how you can analyze landing pages using GPT vision.

The first step is to set up the script by specifying the domain you want to analyze. This will be used as the base domain for comparison. The script uses functions to load a headless browser, create a page, and take screenshots. It also includes options to emulate different device types like mobile and desktop, ensuring a comprehensive analysis.

After taking screenshots of the landing pages, we convert them to base64 format for further processing. We then set up a LangChain pydantic model to receive responses from the GPT vision API. The vision model provides feedback on strengths, areas for improvement, and general recommendations for each landing page. This standardized output allows for easy integration into applications.

To demonstrate the analysis process, we provide a prompt to the model, acting as a marketing user researcher. The prompt includes the URLs of the landing pages and asks for a brief analysis of the screenshots, focusing on areas for improvement. The model generates a JSON schema following a specific format, providing valuable insights for optimization.

While the current script provides a solid foundation for landing page analysis, there are some edge cases and future improvements worth considering. One such case is handling cookie banners and pop-ups. Automating the process of clicking on these elements would enhance the analysis by accounting for their impact on user experience. Another improvement involves creating an X, Y coordinate grid to have complete control over the vision model's actions, allowing for greater autonomy and interaction with the website.

In conclusion, analyzing landing pages, including your own and those of your competitors, is crucial for optimizing user experience and conversion rates. The combination of GPT vision, and LangChain provides a powerful toolkit for conducting in-depth analysis. By leveraging these tools and considering potential edge cases, you can gain valuable insights into your landing pages and make informed decisions to enhance their performance.

Landing Page Analysis

 All right. So in this video, what we're going to have a look at is how you can analyze landing pages, your landing page, and competitors, landing pages, using GPC vision. We're going to use something called Piper tear, which is a puppeteer port for  And this allows us to run puppets here, browsers. , and to execute some commands on some headless browsers. , so there's a couple of different types of installations that were running in this Piper tear, stealth Piper tear. , and we're also going to using Lang chain as well. , we're also going to add in some use agents. 

Now use agents are a way for us to potentially spoof. To other websites and pretend that we're actually a real browser. , and then what you're going to need to do is when you run this script, make sure that you put your domain. In here, because this is what's going to be. Inside of the prompt later on. 

So I'm going to be using the home page. , for, for analysis, but the homepage of my website is still under this specific domain. Now once you've changed that domain. We have this function here, which is called a screenshot full page. And what it does is it loads a headless browser. It creates a page. 

And then if the, that the device type is mobile, it emulates a different view. Por. And, , sets the user agent for the mobile user agent and some other little bits, but if it's a desktop, , that's entered into this, , specific type of function. , as a parameter, then we set the viewport to be slightly larger. 

And we also, , , emulate , the viewport should be that size and also updating the user agent here.  And then we've also got a function which is meant to improve the stealthiness of your browsers. , and that comes from property, a stealth. , and then what we got is a bet that says we're going to wait until that the network is idle. 

And this means that if the browser is downloading resources after has visited the page, , then what's going to happen is, , you can specifically, , wait until the library source downloaded. And then what we do is we scroll to the bottom to ensure that all the lazy loaded images are loaded. , and then after that we wait around two seconds and then we scroll all the way back to the top. And then what we do is we take a screenshot of the entire page by changing the viewport so that the width is equal to the body dot scroll with. And the height is equal to the,  the scroll height of the document of body. 

We take a single screenshot and we save that screenshot.  , and so now what you do is you provide you a severe ELLs and you can see here. , I'm going to be analyzing is the home page. , and then I'm going to be comparing it against these two data engineering pages. Now, if your domain wasn't inside of this first one, so you can see my domain is understanding dates. So calm that it raised an Arabic, cause you do need to update that domain. 

That's at the top of the, , of the script and then we get some clean names and then we basically say, ah, let's create a screenshot. And then we set up an asynchronous function, which allows us to loop over every year row for the both desktop and mobile devices. And then you'll see, we get this kind of output here, where it says taking a screenshot of understanding data. 

And we've also got this other one do frame.co UK and FDM group.com. , and then what we do is we then decide, okay.  We've got all these individual screenshots and I can show you a couple of examples of those. So this is the desktop. This is the mobile. This is the desktop for DeFrain. This is the mobile for the frames. 

Quite large, as you can see. , and then we've also got the desktop for FDM group and the mobile for FDM group. , so they're pretty good. There's a couple of different edge cases that we're wanting to talk about, , in terms of how will you know how you'd want to improve the script in the future. , but for now, , we'll say that these images are good enough. 

, and so what we then do is we need to convert these images to base 64. , and we've got all the images in this folder. They'd been written out to this folder. And so we're going to use glob, which is going to allow us on the left-hand side to go and access all these individual PNG files. , and so we basically go through and , we'll go in and get all the screenshot files and we'll encode those into basically four encoded images. , and then after that, then we set up a Lang chain pedantic model. 

And the idea behind this is that we want to not only get the vision model to respond to us. But we want the vision model to respond to us in a standardized way that we could include , in a application. So this could be the backend of the application and we could provide a standardized output. Bye Jason. , and so what we've got is a feedback aspects of what the aspect is, the description and the recommendations. , and then you've got a website year, row. 

You've got them some strengths, some areas for improvement, which are a list of feedback aspects. And then we've got some general feedback there's optional and also some additional comments as optional, too. , and so what happens is,  we then set up a kind of interesting. Approach where we  say, , act as a marketing research, a user researcher, , you've received that screenshots for, , my website.  And there'll be some different websites. , and then, , please provide brief analysis of the screenshots and we must provide adjacent. , Jason schema, and this is this output Paso with the get format. 

Instructions allows us to, , convert the vision models, response to a standardize output. Now the vision model doesn't currently support Jason mode and it doesn't support function calling. So this is the only way that we can actually get Jason data out of GPC vision at the moment. , but you will see what we do is we also then say a message saying, here's my website pages. , and, , then we pretend to be the AI and say, thanks for providing your web pages and both desktop and mobile versions. , before analyzing them on its research at different websites to understand the competition. 

Can you please provide some information on the different websites? And then we provide it. All of the ones that aren't our domain. , in that specific a screenshot, basically fours dictionary. , and so then we get a response from the vision model. , and you can see the vision model gives us a range of strengths and some areas for improvement. , and so that's basically what we've got at the moment, , you've got this chat model and that takes the prompt into the model into the output pass. 

So we invoke the, , We invoked the chain. And then once we get that back, then you've got this pedantic model. Now, I think, , just to  talk on a couple of different edge cases that have propped up whilst making this, , I think the next steps would be, can we automatically, , click on cookie banners? And can we also click on any popups, , banner ads? , and could we also create an X, Y coordinate grid that will allow us to completely control? , the vision model. , because that's also going to be important from, , like these two ones. 

What we're really trying to do is figure out, can we reliably generate X, Y coordinates. , to automate actions. , the other way we could do cookies as by, , the popups is by specifically looking at all the HTML data and getting rid of any JavaScript. , and just looking at the pure, dome and then asking chat TPT, can you see any Dom elements that look worthy of being clicked on, , pop up elements? , but I suspect that, , an X, Y coordinate grid will allow us to like completely control the vision model. , and drive that to be a lot more autonomous in terms of interacting with the website and also clicking away banners before it takes screenshots. , so hopefully you found this useful. 

Yeah.  , there's a lot to go through, but the main premise is we are guessing lots of images, asynchronously. , from using a browser on your local machine. And then after that, then what you can do is you can use all those images to perform a competitor analysis or multiple landing pages. 

What is the purpose of using user agents in the script?

How does the script ensure that all lazy-loaded images are loaded before taking a screenshot?

Learn how to leverage computer vision models to analyse and optimize your landing pages with unprecedented efficiency.

5 Retention Courses

Courses - It's time to grow👇

Learn the skills to be a Data-Driven Marketer