Your boss is building a budget proposal and wants to know: what contribution will we get from organic? We need to figure out how to predict word of mouth.

Your boss messages you: this is probably about the budget proposal he's putting together. Let's see what he wants.

Hey, what’s the expectation on organic traffic for Q4?
If we can get a handle of how much traffic to expect from word of mouth, we can pair that with our paid ads forecasting and get a realistic picture of what’s realistic for setting Q4 targets.
Can you take a look at seeing how we can predict the contribution we can expect from word of mouth?

Before the invention of mass media, word of mouth was the only way people learned about new products. However recommendations from friends and family still influence 78% of purchase decisions, more than any other channel. The internet made going viral more accessible and more easily trackable, thanks to social media. However public social networks are just the tip of the iceberg. Private sharing via Whatsapp, Email or Slack account, known as ‘dark social’, accounts for as much as 69% of traffic according to Alex Madrigal of the Atlantic.
Word of mouth is becoming more and more critical thanks to Facebook, Google and Amazon’s dominance of ad budgets, and Apple’s recent moves towards user privacy, which have hurt our ability to track the performance of ads. We can't easily track this directly because these platforms don't pass a marketing source or referrer — they show up as 'direct' traffic in analytics platforms.
Since a significant portion of your traffic comes from word of mouth, you have a big hole in your performance forecasts unless you can predict it. If you can measure word of mouth you can calculate how much traffic you can count on from virality and make sure future forecasts are realistic, before you agree to performance targets.

Word of Mouth

Founder, Amazon.com

If you do build a great experience, customers tell each other about that. Word of mouth is very powerful.

I read something on Reforge the other day that might help
They claim you can predict word of mouth with a few simple calculations
Let me know if it works for you because I was planning to try it too

In 2014 Yousuf Bhaijee worked at Zynga with Tomas Pueyo (author of coronavirus articles with 50M+ views) where they created a simplified way of measuring word of mouth. They found a metric that was stable enough to use in forecasting and insightful enough that teams could figure out how to influence it. Yousuf teamed up with Mike Taylor (co-founder at Ladder and Vexpower) to apply this metric to products in a range of categories and found it to be highly predictive.
Most models of virality assume that new users invite X number of people on their first day or week, then the cycle continues - this is analogous to the spread of a disease where people are only infectious for a specific time period, and if the R0 is above 1 (the number of people each person infects) the virus grows exponentially.
The key insight behind the word of mouth coefficient, is that word of mouth is a function of the number of active users you have: every day you get a user to come back and use your product is another day where your product is top of mind, increasing their chance of making a recommendation to a friend. We’ve found returning users to be a reliable, realistic and stable predictor of new organic users. This is important because when making forecasts you’ll know many users to count on organic delivering for you, and will be able to justify higher budgets based on virality assumptions.

Word of Mouth Coefficient

What can we expect from organic?

How do we measure and predict word of mouth? If we can figure this out we can make more realistic forecasts, and figure out ways to move the needle.

Having learned about the word of mouth coefficient, you're excited to try and calculate it. Let's work through the steps.

To calculate Word of Mouth Coefficient (WoMCo) you take new organic users and divide them by returning users plus inorganic users. The WoMCo metric can be interpreted as how many new word of mouth users you get for each inorganic user, i.e. if you drive 10 people to the site via a paid ad campaign and your WoMCo is 0.1, you’re getting 1 new user for free from word of mouth. 

Calculating WoMCo

The key to finding a stable predictor of word of mouth is how you define the numerator: ‘organic’. For example direct traffic and google search traffic that lands on the homepage should count, but search traffic to product or blog pages should be categorized as inorganic. It's also important to think about how you define the denominator: 'returning + inorganic'. You get the best results by only including users likely to refer, for example users that have done at least one qualifying action, like add to cart or wishlist.

The first step is to define the denominator you'll use in your calculation. This is the segment of your userbase that you expect to refer new users. Typically it makes sense to start with a simple definition, like 'Returning Users'. This will give you a good starting point from which you can then refine further if the R2 isn't high. Usually you get better results from further refinement down to just users who have completed at least one qualifying action. For example a returning user who hit a paywall and bounce without becoming a member isn't likely to refer new users, but active subscribers who log in to read each week are highly likely to keep making recommendations. It doesn't have to be only returning users you include in the denominator: in many cases new users who complete a qualifying action are just as likely to refer.

Define the Denominator

Which of these segments are likely to tell friends, coworkers or family about the product?

The second step is to define the numerator you'll use in your calculation. This is the segment of your userbase that you expect to be from word of mouth. The simple definition is 'direct' traffic, i.e. any user which is unattributed as they have no marketing source. However due to user privacy rules and adblocking a lot of users who are actually driven by paid ads might end up in this bucket. It sometimes makes sense to narrow this definition down to just direct users who land on specific pages that indicate someone shared a link with them. Alternatively you might have more luck with broadening the definition to users who search for your brand term on Google. It's hard to see this in Analytics since Google no longer passes keyword information, so a good proxy for this is users from Google who land on the homepage - the rest of the pages on your site likely owe their traffic to SEO campaigns, not word of mouth. Finally the definition of 'new' is important. Did they perform a qualifying action or did they just bounce? Did they sign up (so you can tell they're a new user) or did they just not have a returning user cookie (due to adblocking, clearing cookies or switching devices)?

Define the Numerator

Which of these segments would be mostly driven by word of mouth?

Once you have your definitions for new organics and inorganics, and have pulled the data, you can plot the WoMCo metric over time to see how stable it is. You can also identify its predictive accuracy by calculating the R2 value, which will tell you how much you can rely on forecasting using this metric. We've provided sample data for this next exercise, and you'll need to pivot that data to get it in the right format (one column per segment, one date per row), in order to calculate your word of mouth coefficient.

Plotting WoMCo

Create a pivot table that shows one column per segment, with a row for each week. Then use the resulting table to calculate word of mouth coefficient. Explain your process.

Plot a stacked area chart on the left axis of each segment that adds up to all users (Direct, Brand, Returning, Inorganic). Then add a line chart (make the original chart a combo chart) on the right axis that shows WoMCo over time. Is the metric relatively stable over time? What noticeable periods are there?

It's possible to find the correlation of two variables without doing any math: simply add a trendline and choose the option to add the equation to a scatterplot chart, and you'll have the R2, slope and intercept calculated for you.

Calculate the correlation between new organics (direct + brand) and returning users by creating a scatterplot and adding a trendline with the equation. What is the R2?

The coefficient from the trendline equation (the number multiplied by x) is the inverse of the average word of mouth coefficient. So for example if the coefficient is 9.54, that means we get 1 word of mouth user for every 10 returning users. The inverse (1/9.54) is 0.10, which is the number of word of mouth users we get per returning user, which was the metric shown on our line chart.

This is great
Knowing we get 0.1 new users for each returning one is the last thing I needed to forecast budgets
It's amazing how stable this metric is: we should start tracking it for anomalies and overall trends.
This is a potential game changer!

Calculating Word of Mouth Coefficient

A significant portion of your users come via 'dark social' i.e. they show up as 'direct' in Google Analytics. Is it possible to measure word of mouth?

How do I measure virality?

We're going to do an initial beta test and launch to 20,000 users at a university campus
That'll buy us time to fix any bugs and iterate on what's working
We'll have 1,000 initial test users who are on our email list, and from there we expect to spread virally through the school
Can you make a few assumptions and build a viral growth model I can present to the board?
Once we know what to expect, we can track our progress against your model

It's easy to find the concept of viral growth appealing: who doesn't want to grow their product via word of mouth and save on advertising costs? In reality true virality is rare: often what we call 'viral' is really the result of a large influencer sharing to their audience. True virality is when each person that uses the product tells more than one person about it, recruiting new users via word of mouth or social media. 
If you achieve viral growth your product will grow exponentially – like a biological virus – as 1 user gets 3, and 3 get 9, 9 get 27, then 81, 243... fast forward 10 cycles and that 1 initial user has 'infected' over 20,000. Keep growing at this rate for 12 more cycles and you've reached 10 billion users, more than the population of Earth. Of course in reality exponential  'hockey-stick' growth doesn't continue forever, it's just the first part of an S-Curve. Growth flattens out as the audience becomes saturated, and most people who get invited have already joined.
To calculate virality a handful of metrics are important. The number of invites each user sends out is key: this doesn't have to be actual invite messages, they could be simply telling their friends about a new app, or using it in a prominently visible way. Next is the conversion rate from users being exposed to the product and actually signing up as a new user. Finally the carrying capacity is what gives us our S-Curve, as we can use it to adjust the conversion rate downwards as the audience becomes saturated.

Viral Growth

I am going to show you how to model viral growth. So this is what we're going to end up with. I'm just going to create a new tab, a which will be here, and we need to put a few different assumptions into the model first. So the assumption. The kind of inputs that you hard-code into the model the convention is you want to do these in blue.
So that it's pretty clear, like what the assumptions are. And then you want to reference them all in one place. So that whenever someone wants to change the model or kind of update some of the assumptions or they can do that here rather than you know, doing it elsewhere and then like, you know, potentially changing it at one place and then somewhere other places referencing it.
So Put these different assumptions in here. So I'm just going to copy and paste them across. So I'm sorry, let me paste them across. Cut that. Okay, cool. So these, this is where the the numbers are going to go. All right. So what assumptions do we have? We have carrying capacity that is.
Size of the audience. This is really important because we have to be realistic about this. If we assume that our audience is infinite than vitality is exponential. So if I tell three people about the app and then they tell three people. Yeah, it's going to triple that each time. So eventually you would have more people than the population of the entire earth.
You know, it's like a viral infection, right? Like COVID and obviously there's a spread that way that eventually reach herd immunity as the people who. I've heard about the app or could have been invited to the app, have already been told about it or in the case of COVID people who've already been infected way less likely to get infected again, or if they've been vaccinated.
So that's what we're gonna handle with carrying capacity. Are we going to say that we have 20,000 uses? And researches that number is that's just how many people were in Harvard when Facebook watched. Right? So that, that went viral in Harvard, as I'm just using that as a placeholder average invitation.
So this is how many people get invited to the app, each period by by the the, the number of uses as they're using the app. 'cause I'm like Facebook was the retell seven friends in 10 days as someone just going to use that as an example. And then a conversion rate is off the people who get told how many of them actually convert and start to use the app.
So what I'm assuming here is if you tell seven people about Facebook maybe only 10% are going to. So it might've been much higher invitations actually, but we can play around with these numbers once the model is built and then the initial user base, I'm just going to say in this case you have a thousand users and this is the seed audience.
These are the people that are spreading the word, the initial people you invited to your app. Okay. So now we have the assumptions, it's time to build the model. So we're just going to model out. Can I copy across this, I'm going to have different time periods. The convention is a do it you know, length ways like this are horizontally.
But you can do it basically if you want, if you prefer. I'm interested in to copy these labels. And so we need the saturation and we need the conversion rate, which will adjust based on the saturation. And I'll explain that in a second. And then we'll calculate how many new conversions we get for each user using the app.
And then we have the new users that, that generates each cycle then adds to the cumulative users and then we calculate the monthly growth. Cool. So for saturation we want to make the saturation at first it's going to be 0% because in the first period it's not saturated. You know, w we don't have uses.
And then the second period, it's going to be the cumulative users divided by the. Carrying capacity and we're going to make this an absolute reference and that's F so. I right now, it's just going to say zero out, but we'll deal with that in a second. So that, that would be the saturation will drag that formula across and it'll populate in the second the conversion rate.
That's going to adjust based on the saturation. So the simple formula here is it's going to be one minus the saturation. And then it's going to multiply that by the conversion rate. So at first is 10%, but then as we drag it along, oh, sorry, I need to make this an absolute reference as say, this C6 I'm this effort, or you can just put the dollar signs in there.
So when we drag it across, it's going to say. Still carry with us. So it's right now, it's just 10% initially, but, but it we'll see when we update the bottom this is going to fill in conversions per user is the average invitations, which is going to be an absolute reference multiplied by the conversion rate.
So that's how many conversions would get per user per period. And then the new uses is going to be. How many uses we initially have the initial user base multiplied by the conversions per user. So we're going to get 700 new users in this period. And then this case, we're not going to drag it across from him.
For the next set are we're going to take the cumulative users. So how many people were using it last period? I know in a most pipe that by the conversion user, I would say if someone was using it, you know, if around had a hundred people using it, the last period, I then 0.7 multiplied by a hundred to be like 17 new people.
But this is going to populate in the second year. Now, now we have the final thing here. Cumulative uses as a cumulative uses can be the initial user base. So that's one plus uses that we brought then for this one, it's going to be the cumulative uses plus the new users here. And then we're just going to drag that across.
And so that is how many cumulative users we've got total. And now it just do some formatting. So I'm gonna select this and then say number and then kind of select these and make these numbers as well. And then just decrease the number of decimal places. And then, and then we can see that our cumulative users rises.
So. Okay now monthly growth that is simply you know, how many newses we got divided by what the initial user base I'd say this is 70% initially. And then over here, it's going to be the new users divided by this here. So you can see actually, if. Bring this across. Then over time initially it kind of matches the conversions per user.
But over time that becomes less important. And it's about how many users we have as. Eventually the growth does slow, which is what you'd expect. Okay. So now let's create a chart so we can visualize a list. And one thing to note before Curtis chart is you can see the saturation rate went from 0%, 9%, 14%, 22.
Eventually it got 98% saturated by the end. And therefore the conversion rate has dropped. And the reason why we're doing. Is to make it more realistic. So that as the audience gets more saturated, the conversion rate drops. If nobody's heard of the app and I tell seven people, I then maybe 10% of them will sign up.
But if if everyone has heard of the app in the audience in the carrying capacity then maybe 0% would sign up. Okay. Let's make a chart. So we're going to slough that row and then we're going to go insert. There we go. Okay. Cumulative users over time. As you can see that with this saturation rate, we're getting to 20,000 users and then it's done.
And what's interesting is even if we increase the conversion rate, we never get past 20,000 users because that's the carrying capacity. That's the full amount of audience. If you think about viral infections this is why we quarantine essentially trying to bring down the carrying capacity because if there are less people that you can spread the virus to because you know, you close the borders of your country then you know, then, then.
You're going to hit a wall and infections will drop off. So it's the same sort of thing here. The, the audience that we that we have the audience is relevant for the product is the limiting factor to how many people can spread virally to you can see, even if we change the average number of the patients, that's going to make a difference.
But then if we increase the carrying capacity to 200,000 because we. You know, a much bigger audience, I, then we can see over time. We get some really good viral hockey stick growth, and then it drops off here as well. So we just change these assumptions back. And if we have the original assumptions of average invitations of seven and conversion, we have 10%, or you can see that, you know, our charts just going to look hockey stick because we haven't gotten to a saturation yet we're only 43% saturated by.
If we want the the horizontal access the way we do it is we select the data rain. Just going to select this row here. And then we have the periods on him. So when we look at our initial model or we can see that by period, a growth is significantly slower. Parents six is like when the change over happens, right when we get to 50% saturation.
Okay. Hopefully this was useful and enjoy modeling viral growth.

How quickly do we get to 10,000 cumulative users, with an initial userbase of 1,000, 7 avg. invitations per user, a 10% conversion rate and carrying capacity of 20,000?

How many users would we have after 10 cycles if we kept all assumptions the same except increased carrying capacity to 100,000?

What happens if you don't factor in 'carrying capacity' to your viral growth model?

Why spend money on ads when we can just go viral? Thousands of failed startups assumed their product was so good users would tell their friends about it. Virality doesn't work that way: it has to be engineered by understanding, modeling and optimizing all the inputs.

We need to go viral

You’re talking to your CEO and product team who are eager to find out what issues to prioritize on the product roadmap to improve people sharing via word of mouth.

Have we looked at the product reviews?
We could do some sort of analysis
Extract the most common issues and see what patterns there are
Then we know what to prioritize


Product reviews are important for every marketer to read. Any customer who has taken the time to leave a positive or negative review is doing you a favor by doing your research for you. However it can be time consuming to read every review for your product. You might have hundreds or maybe even thousands of reviews, and each review could be anything from a few lines or thousands of words. Often you just want to extract the relevant information from each review and summarize any patterns you find.
This used to be a highly manual process, but now has become relatively trivial with the advent of AI models like GPT-3 from OpenAI. These large language models (LLMs) excel at lots of tasks, but in particular they can handle arbitrary classification tasks with very few examples given. Using GPT-3 in Google Sheets can make the process far simpler to get results across a wide range of inputs, and get back structured results for use later.

Summarize Product Reviews

00:01 Hey, let me show you how to use G P T three in a Google sheet in order to categorize product reviews.
00:09 So the, if you're not familiar with G PT three it's an AI from the company open ai where you can send it some texts and then you get a response back.
00:19 And normally you have to do that through an api, like send, you know, use code to send the data or you have to kind of like log into the system and use it.
00:29 But that's kind of limiting because quite often I wanna be able to do this scale, like in the spreadsheet or something.
00:36 So what I've developed is basically a Google Apps extension that gives you the function for GT three as like a Google Sheet function.
00:46 So if you wrote like one to write a story once, one time then you could post equal as GT three.
00:56 You can see the function here has a little description. So you just need to input basically the prompt. And then the max tokens that's an optional one.
01:10 You'd have to worry about that. And then the temperature also optional. For now, let's just input this text once upon a time and then close it.
01:19 And it's just gonna complete that text, so it's gonna load, It only takes a few seconds usually. And there we go, once upon a time in a village called Bird Cottage in the Village of Water back West Yorkshire, England.
01:33 Okay. Right? And, you know, if we, you know, wanted to do this a bunch of times, you know, we could just drag this formula down and then now it's made, you know, all these API calls for us to GT three, and it's gonna come up with different different completions.
01:50 So that makes the usage, I think, of GT three much cleaner and easier. And yeah, anyone non-technical can use it.
01:58 So there's a little bit of a setup, which we're gonna go through in second, But first let me show you the actual kind of issue that we're trying to solve.
02:06 Okay? So we have a bunch of predator reviews. You know, this is for a smart a smart baby product. And you know, it helps check for newborn's breathing.
02:16 And and, and you know, like this these are kind of real product reviews. And what, you know, we have 30 here, but you could imagine having hundreds of these, right?
02:27 And we want to really get a sense of like, what are the main issues that people are finding with the product?
02:32 Like, what can we improve for our product development? So what I've done is I, you know, this is a specific review here and then I've just manually tagged it with, like, this person talked about alarm failure, talked about connecting to the sensor.
02:47 They talked about bugs in the app, and they talked about it being expensive. This person talked about the alarm volume, like it should be louder.
02:56 And this one didn't mention any issues. So five star review, they were really happy.  So I just put none. And we're using these as examples to pass to GT three because you know, if I just say, you know, GT three if I just say, you know, tell me the issues mentioned in this product review and then I, you insert the proof review.
03:27 So put amand and then click into here. So let's kinda make this all one piece of text. Let me space there, and then, and then I wanna close it out and say issues.
03:42 That's called a zero shot completion. Meaning we're not giving you any examples, We're just saying, you know, we're just talking to it like a, we were a human, right?
03:50 We're saying, just tell me what are the issues here, and it will do a reasonable job. You'll see. So yeah, it says, app seems to be a bit buggy.
03:58 It has no way to telling their batteries low. There's some repetition here. But in, in general one problem with zero shot is that it doesn't really give you back the structure of data that you want to be useful.
04:13 So you can see, like, my data is much more structured and there's specific labels. So instead of sending it that way we have a set up a template.
04:23 So we've set up a specific you know, list of examples and structure for the data. So this is a really useful prompt engineering tactic.
04:33 So we're asking a question here. We want it to extract issues from the fall product reviews and then we've got like a review, and then issues review, and then issues review, and then issues.
04:45 So we've given it three examples of the ones that I put in here.  And and when when I've, when we've done that, then it'll actually kind of understand, like, almost like, kind of training out on what type of response that we want.
05:00 So this this is what we're doing here. So we're, we're basically taking the you know, the template and then we're adding you know, and review and then and issues here.
05:14 So that, that is the, that is the prompt that we're passing. And that gets us much, much better results. So you can see here that you know, we found a lot of good issues and, and there, you know, some of them are not in like, full structure that we'd like, but this has saved us a lot of time.
05:30 And we can kind of look in here you know, some, some sometimes, like when you're evaluating your prompt you kind of wanna evaluate it like any model.
05:40 So you know, are there any false positives? So did it imagine an issue that the user didn't actually mention? Or are there any false negatives?
05:50 So there was an issue and it didn't pick up on it. So typically I would look at some of the nons when I'm testing this.
05:56 So I'd say like, this one is a non easy to use, annoying only connects to one phone. Okay? So that would be, you know, that's something that it missed, right?
06:05 So that's a, that's a false that's a, a false negative. It said there were non, but it was wrong about that.
06:13 And then you know you know, I, I don't think there'll be that many false positives here cause he's relatively conservative.
06:22 But like, let's see, this guy so they do they mention battery life? Yeah, draining battery. Do they mention connectivity? Yeah, connection drops out.
06:31 Do they mention synchronization? Yeah. So I, I imagine with this type of categorization, we're gonna get more you know, false negatives than false positives.
06:42 Although sometimes it does make up stuff, so you have to be careful. But, you know, this is saved as a bunch of time, and we're really looking at this data in aggregate anyway.
06:50 So you know, the way you'll use this in real world is you might have, you know, hundreds of thousands of these reviews and then drag this across, and then just pick up on the general themes and kind of categorize 'em from here.
07:03 Cool. So so that's that's important. Now one thing you might notice is we have some extra numbers here. So in the function, we ha we've also passed in these optional prompts, max tokens and temperature.
07:16 So what are these? So max tokens, it is basically how many tokens to return. So just to kind of give you an example here yeah, if you'll say hello.
07:28 And then if I put GT three and then put the prompt, if I, if I put max tokens of like five, it's gonna give a really short answer because and it's gonna cut off, right?
07:47 Because token is about four characters roughly, and therefore it's, you know, run out of space to respond. But if I put in, you know, ma max tokens of 50 it's gonna gimme a much longer response.
07:59 And the reason why this is important is that this changes changes the cost of, yeah. So here we go much, Okay, gimme some code here, <laugh>.
08:09 So it changes the actual cost, right?  Because you you pay by the number of tokens for T P three, it's very, very cheap.
08:17 It's something like 2 cents 2 cents an API call on average. But you know, you have to be careful. It's like, one thing to really think about is that whenever you drag this formula, or if you change something and then it recalculates, you're paying for here, you know, 30 API calls or like 27 API calls every time.
08:38 So it could add up pretty quickly if you, what you're doing. So I would say test it on just a few, and then, and then when you're really sure, then, then drag everything down.
08:47 Cool. The other thing we didn't explain is the temperature. So you could say hello, my name is Dave, and then gimme a prompt, put that into the and then let's just, let's just say max token is a hundred.
09:09 That's what we're using as default. And then if we put a temperature of nine that rate high. So that's gonna increase the randomness of the response, just waiting for that.
09:27 Okay, So what was this saying? Okay, <laugh>, I name Dave, This is my jam as it seem, at least it seems I've written the duster a few times.
09:39 <laugh>, okay. Yeah, pretty crazy response, right? So that's a very, very random response. It's a very creative response. And and then if you kind of dial it down to one you're gonna get a very boring response, a very, you know, predictable response.
09:53 So it depends on yeah, here go, I say I'm recovering alcoholic and drug addicts, right? So this is like, when it's looked in the data set, like this is probably the most popular pairing, right?
10:03 My name is Dave and I'm a alcoholic, right? <laugh>. So so this is you know, this is a very a very kind of predictable repetitive response.
10:16 You know, something that's like very highly likely to show up in the data. You know, and the data set here is like the entire internet basically.
10:24 But like as 0.9 would be something creative, something very divergent from the data set. So have a think and, you know, test a few different options out.
10:32 In in this scenario, I found that you want it to be like slightly creative, but not too much. And therefore I kind of have it at 0.4 and I found that that works better, but it's really trial and error in no hard and faster rules to this.
10:47 The default is 0.7. Cool. So that's how that works. If you you know, in terms of setup there's a couple of things you need to do.
10:57 So one is click here and set, Create an account for open ai. Get your API key and paste it in here.
11:04 My API key will be deleted once once I share this with you. So this function won't be working anymore. But what, what I will do is I'm gonna have function here and I'm just gonna delete this equal times.
11:22 So if you need to copy this function then yeah, if you need to copy this function then all you need to do is copy it and then put equals here.
11:33 And the reason why I'm gonna put it on the side here is I'm just gonna take this and then press draw, check down, and then control C to copy, and then control shift v the paste without function, right?
11:47 And formatting. So that just paces the values. And what I've done now is that now there are no you know, there are no responses back, right?
11:56 You know, that, that it's like, it's not gonna change now because GT 83 isn't running, it's just the text value that we got before.
12:05 So I recommend you do this as soon as you've got results that you like, because the worst thing you could do is like accidentally you know, delete <laugh> the function or, you know, or, or change something, and then it all reruns and you might not get actually like this predictable answer back.
12:19 There's a lot of, you know, randomness and entropy in, in, in what it sends you back. Cool. Okay. So this is how it works.
12:26 You could really use this for anything. You could use this for translation. You could use it for, you know, writing ad copy really whatever you want.
12:34 Now that you have this function, if you're interested in how this actually works then you can see there's this GT three run, GT three.
12:44 Actually if you're running into issues, just click here and click run GT three. And that should correct it. But but what this is, is this is actually being added by a Google app script.
12:54 So go to extensions and then get to App Scripts. You can see how this works. So, and, you know, feel free to copy this and use it.
13:04 This is how you would make a Google app script function. What I'm doing here is just adding the menu here, like run g PT three.
13:12 I've code have function that just calls the api with the prompt max tokens in the temperature, and then a few other hard coded settings.
13:22 And then gets the response. We send our API key alongside in the headers. That's how it knows that it's us.
13:28 And then, and that's why you never wanna share the API key cuz they could just rinse your account and spend a ton of money.
13:34 Then it makes the uses your URL app which basically makes the API call for you to the completions engine for open api.
13:45 And then when it gets it back, it takes the top choice and then returns the text. So, you know, you could also edit this if you like the, this you know, like I'll put this out open source so you guys can mess around with it if you want.
13:59 Cool. So that's how that works. You know, if it only works in this sheet. So when you, when you come in here, you wanna just make a copy and then it'll copy the script as well or you know like make a copy, copy of the script.
14:15 And if you wanna kind of edit the script, you can come in here and the Google app scripts and then copy it and kind of make changes if you like.
14:22 Cool. Okay. Hopefully that was helpful.


What’s the advantage to using GPT-3 in GSheets?

Why is it useful to do this sort of analysis of customer reviews?

What does the ‘temperature’ mean when sending a prompt to GPT-3?

It’s amazing how long it has been since most marketers last read a customer review. They’re missing out on a goldmine of insight for figuring out why customers don’t recommend your product to their friends.

What do our customers think?

EXECUTIVE SUITE – STRATEGY DISCUSSION POST-LUNCH

You're in a strategy discussion and need to learn how to effectively evaluate your generative AI based applications more effectively.

Alright team, we've got a lot on our plate today. 
Our boss wants us to dive into the world of evaluating generative AI. 
We're going to learn about different techniques for evaluating customer valuation metrics for AI models, including programmatic rule-based evaluation, LLM based evaluation, and human-based evaluation. 
We'll also learn how to calculate accuracy for each method and get some tips for optimizing the evaluation process. 
So grab your thinking caps and let's get started!

Generative AI has revolutionized various industries by enabling the creation of realistic and customized content. However, evaluating the performance and quality of generative AI models can be challenging. In this course, you’ll explore different evaluation techniques and their trade-offs to help you build effective customer valuation metrics for generative AI models.

The first evaluation technique we will discuss is programmatic rule-based evaluation. This approach involves creating custom rules to determine the quality of the generated output. For example, in the context of generating email content, you can evaluate the length of the email or check if it contains specific keywords. The advantage of this technique is its quick feedback loop and cost-effectiveness. However, it may not be suitable for tasks that require nuanced evaluation.

The second technique is LLM (Language Model)-based evaluation. Here, instead of relying on custom rules, another LLM is used to evaluate the generated output. This approach is useful for tasks that are too nuanced for rule-based evaluation, such as evaluating the helpfulness or readability of text. Multiple outputs are generated by the LLM, and another LLM is used to rate each output. While this technique provides more accurate evaluation, it is slower and more costly due to the need for human intervention.

The third technique is human-based evaluation, where human evaluators rate the generated output. This approach is ideal for high-stakes tasks where false positives or false negatives can have significant consequences. However, human-based evaluation is time-consuming, costly, and requires a labeling step. It provides the highest level of accuracy but may not be feasible for large-scale evaluations.

To illustrate these evaluation techniques, we provide code examples using different scenarios. In the programmatic rule-based evaluation example, we demonstrate how to set up evaluation metrics using Python libraries like pandas, numpy, and TQDM. We define custom rules, such as the minimum length of a social media blog post or the presence of specific keywords, and evaluate the generated output accordingly.

In the LLM-based evaluation example, we show how to use one LLM to generate multiple outputs and another LLM to rate each output. We discuss the importance of ground truth labels and how synthetic data generated by a powerful model can be used for evaluation.

In the human-based evaluation example, we generate images using OpenAI's GPT-3.5-Turbo and ask human evaluators to rate the quality of the images. We demonstrate an interactive approval system that allows evaluators to provide feedback on each output.

Evaluating Generative AI

What are the three different types of methodologies mentioned for evaluating LLM responses?

What are some advantages of programmatic rule-based evaluation?

Which evaluation strategy is best suited for tasks that have high customer risk and where false positives or false negatives can cause consumer damage?

Which of the following is NOT mentioned as a way to optimize LLM outputs?

Discover cutting-edge techniques for elevating the accuracy and effectiveness of Large Language Models and Image Generation Models. Our guide delves into innovative evaluation metrics, providing insights to enhance model reliability and drive impactful results.

Maximizing LLM & Image Model Performance: Advanced Evaluation Strategies	

In the upsert office, Sally asks everyone to gather round.

I've seen some AI-generated content that's really convincing, but it's important that we can't be making false claims in our content
AI can be incredibly powerful, but it's also prone to making things up. We need to be able to distinguish fact from fiction. 

Claim Detection: Unveiling the Truth with AI

In the era of AI-generated content, one common issue that arises is claim detection. AI has the ability to create content, but it also has the tendency to fabricate claims, making up quotes, statistics, and even entire events that never actually occurred. This weak point of AI, however, can be effectively addressed using AI itself.

Imagine this scenario: you have an article based on a real story from a reputable source, like The New York Times. You decide to put it to the test by asking an AI model to rewrite the story while adding a few fake elements. These fake elements include stating that a court has voided Elon Musk's pay package, attributing a fabricated quote to a lawyer, and making up a statistic about Musk's ownership percentage in Tesla. Now, the challenge is to detect these false claims.

To tackle this task, a diligent copy editor utilizes AI to detect any claims that need to be fact-checked. The editor uses a prompt that specifies the types of claims to look for, such as statistics, quotes, anecdotes, and instructions. By employing AI, the editor aims to quickly and inexpensively identify any claims that may need verification.

The AI model is then run with a low temperature setting to minimize the likelihood of it making up false claims. The results are promising. The model successfully identifies both the true claims and the fake ones that were intentionally inserted. This demonstrates the potential of AI in detecting claims and investigating their veracity.

To further enhance the fact-checking process, the editor employs a tool called TabT, which provides web search capabilities to AI agents. This tool enables the AI model to validate claims by performing a web search and presenting the results, including links, content, and relevance scores. This information proves invaluable in determining the accuracy of the claims.

To streamline the process even further, the editor utilizes the Q&A search feature within TabT. This feature allows the AI model to directly ask a question based on the claim and context. The model then receives an answer, eliminating the need for additional calls or human intervention. In this case, the claim that the court voided Elon Musk's pay package is found to be invalid, as recent shareholder voting reaffirmed his compensation package.

The power of AI in fact-checking lies in its ability to swiftly access up-to-date information. AI models can easily verify claims by cross-referencing multiple sources, eliminating the need for labor-intensive human fact-checking

Claim Detection

  Hey, you can talk about claim detection. Because this is a common issue in creating content with AI. It makes things up hallucinates and. It comes up with quotes that didn't actually happen. It makes up statistics. They're out of date. It's typically something that is a weak point of AI, but the interesting thing is that you can actually use AI to solve AI's own problems. 

Everything just becomes AI. What we have here is some article text. I made this it's an article based on a real article here in the New York times. I competent paste it into Chach empty, and I asked it to rewrite the story, but add a few fake things in here. 

The fake things. That it's added that it said that the court has not. Yeah. The court has voided his pay package. Elon Musk's pay package. And in reality, they didn't. We also added in this. Fake quote from Greg Varel home. This quote from the lawyer of the disenchanted shareholders, it's fabricated. 

And then we made up a statistic as well. So I'm just saying that we'd. He would own 20% of Tesla, an actual figure that wasn't quoted. Cool. Those are the fake things. We'll see if it finds those. It's a new of times their journalism is pretty good. You would expect all the other things to be true, but that's not true of all journalism either AI generated or human. 

So it's useful for us to be able to detect what claims are being made and then investigate. After again, using AI. Just gonna run that. And I have a basic prompt that I put together here as a diligent copy editor to detect any claims that need to be fractured. And I've given some specifics, a claim could be a statistic, a quote, an anecdote and instruction. So if it tells you this is how this function works in Excel, right? That can actually be a problem. 

Like it might make that up. I've seen that happen in the past. Or say any information that stated as fact which may be out of date incorrect or imagined, I'm asking it to attend Jason then unmanned, unmodified exit from the text. The claim, the type of claim being detected and some reasoning why the claim might need very caged verification. 

So a bit of chain of thought going on here, I'm using  Yeah, it makes mistakes. But it's very fast, very cheap. So I prefer for this type of task and then I'm just passing the article, text tip, and then the temperature I'm putting a 0.2 because I don't want it to make up the X ups. I've had that as a problem in the past, but no, should be good here. 

If we keep the temperature low. And then if we run that.  Can you see what accepts we get back?  

Okay. So here's an example of next up. For months, many tests. Investors are worried about how engaged he would be, blah, blah, blah. That's an And then reasonably the claim judge Deller of what it must pay package needs. Verification. We, that was one of our fake ones, right? It says, especially after a judge in Delaware voided his pay package. 

So it has correctly found that, it also found the statistics. She has a with 48 billion. So we need to look up the latest value of the shares. Yeah. You can see that. Picked up the percentage of Tesla. Here, which is pretty good as well. So I picked up a lot of the claims, both true ones and also the fake ones that we put in him. All right. 

We need to get that into actual Jason. So in order to do that we just basically remove this like beginning pot from.  Yeah. I see it says  triple tick, Jason. Get rid of that and get rid of this triple takes at the end. And then we have Python object that we can work with now. So that's the first claim. 

I'm using  to look up these research claims tabularly is an AI. Tool basically to social web, it gives us a web search. Capabilities to your AI agents. There are other options here, like you could use perplexities API has searched. And Google's AI has such as well. And then you could also use serve API, I think is another one that people use to this. Some of them have free credits, some of them don't but it's heavily I've I've been recommended by a few people. 

It has a thousand requests for free per month. It's worth setting it up and then you don't need a credit card or anything. Cool. His tablet client, you just import that it's going to run this. It's going to ask me for my API key. If I go into Temperley here, I'm just gonna regenerate this.  I'm going to copy it. 

You don't want to show anyone your API key. And pay some. And to. And then hopefully that run. Here we go. So here's the basic search. I've just asked it to validate his claim a formatted it slightly. So I just put the claim. And then the reasoning. Into a string. And then I put that into his, just says, validate this claim. You can see this is the career that I gave it a and you can see, this is the one about the pay package being voided. 

So that's the question I asked. And then these are the results. So it's done basically a Google search and you can see the content of those searches. So it has the links. So if you want to be able to provide citations and things, this is really great. It was really helpful. And you got the content here as well as the score in terms of how relevant they are.  Great. 

We have the context that's really useful, but I'm going to be even lazier here because I don't want to have to make another call and say, okay, based on this context, is this going? Correct? Yeah, maybe it depends on what application you're designing. But for me, I just want to check. And get an answer. 

So I'm going to use the Q and a search. Which is basically just another call in a underneath. It will do this such and then it will give you back the answer. So it says the claim that the judge in Delaware avoided on AMS pay package is not valid. Test to shareholders recently voted to reaffirm a $56 billion compensation package. 

So this actually just happened a couple of days ago. So it's really impressive that you were able to pull out this up-to-date information. And one of the reasons why AI makes fake claims is because they don't know what the up-to-date information is. Really. Powerful to be able to go and fact check. Your sources again with AI, right? 

So you don't have to have a human do this fact checking. When you want to run this across lots of different ones, you could do a four loop. Here I'm just looping through every single one, and then I'll have all my claims checked. You could also. Structure it in a different way so that you would get maybe. A, a, yes, no answer. 

For example, that'd be really helpful because then you could display Invalid claim in your user interface instead. If you do run into an issue. There's also error handling as well. So I've noticed that sometimes it's have really, it gets, I get this like bad request for URL sometimes. 

Or you might have rate limiting. Yes, sir. Maybe you use a multiple solutions and see which ones work best for you. But ultimately what you get here is you have the X up, you have the type of claim, so you can see what types of claims are being made. And then you would see the context. So here we go. 

There's no verified information. To support the claim that the judge voided the pay packet. Here, it's claims that she has a, with 48 billion with has his cousin. It's going to be verified.  It doesn't specify the exact value of the shares. What you could do now, if you were building an AI application, but this. Is. You can have another call to LLM for each one of these. And just say, okay, based on the context, do we need to alert the user? To check this claim, so maybe in user interface, you have some visual. Highlighting that you say, Hey these are the claims you would need to look at and rewrite potentially, or make them up-to-date. 

And you could also use this as a trigger. So you could say, send this back to  to rewrite this, except if there is. Some claim that is, that we have the context to fix. Yeah, there's lots you can do with this. This is a more of a mental model for you to think about using, for lots of different purposes. 

But in particular, I used this. for SEO generated content, just to make sure that it is accurate and it's not. Putting more fake information out there in the internet because that's one of the biggest bug beds people have with AI and hopefully you can help avoid it with this. 

What is the common issue in creating content with AI mentioned in the tutorial?

How can AI be used to solve its own problems?

How can AI be used to solve its own problems in content creation?

What is the purpose of using the TableClient in the tutorial?

How can AI help in fact-checking and verifying claims in content creation?

One of the biggest problems with AI is their tendency to make things up, dubbed hallucination, leading to inaccuracies in content marketing.

What if my AI makes something up?

Alright, team! Gather 'round in the breakroom for a lunchtime exchange. We've got an exciting tutorial today on a concept called role prompting. It's all about instructing language models, like our very own GoolyBib AI, to adopt the perspective of a specific celebrity or job role to generate responses in a desired style. 

Imagine being able to channel the genius of Elon Musk or the creativity of Steve Jobs in our AI's output. That's what role prompting can do! We'll learn how to use this technique to influence our AI's responses and make them more aligned with the style we want.

But hold on, that's not all. We'll also dive into evaluating the model's responses using an evaluation prompt. This helps us determine if the AI's output matches the desired characteristics of the chosen role. It's like putting our AI's responses through a quality check!

So grab your lunch, get comfy, and get ready to act like a genius with role prompting.

I can't wait to see how role prompting can take our AI to the next level. It's like tapping into the minds of the greatest thinkers and innovators. \n
This technique will give us the power to shape our AI's responses and make them truly genius-like. \n
By using evaluation prompts, we'll ensure that our AI's output matches the style and characteristics we're aiming for. It's like having a quality control system for our AI's brilliance. \n
I'm excited to see how this tutorial will help us unlock the full potential of our language model. Let's act like geniuses and revolutionize the way we interact with our AI!

Role Prompting: Unleashing the Power of Language Models

Language models have come a long way in recent years, thanks to advancements in artificial intelligence. These models, such as OpenAI's GPT-3, have the ability to generate human-like text based on given prompts. However, sometimes we want more than just generic responses. We want the language model to adopt a specific persona or style, and that's where role prompting comes into play.

Role prompting involves instructing the language model to take on the persona of a particular celebrity, domain expert, or even a job role. By doing so, we can elicit responses that align with the desired style or expertise. For example, we can ask the model to respond to a math question as if it were a math teacher, or to generate product names as if it were Elon Musk brainstorming.

The power of role prompting lies in its ability to provide subjective preferences. Different organizations or individuals may have different stylistic requirements for their content. Some may prefer a writing style reminiscent of Hemingway, while others may want a more playful and innovative tone, like that of Elon Musk. With role prompting, we can cater to these preferences and get the language model to generate text that aligns with the desired style.

To demonstrate how role prompting works, let's take a closer look at an example. In the provided transcript, the speaker discusses the process of role prompting using GPT-3. They show how the system can be set to a specific role, such as Elon Musk, and then provide prompts related to that role. By giving the model examples of what Elon Musk would say or do, we can guide it to generate responses that match his persona.

The transcript also highlights the importance of providing synthetic examples of what we want from the language model. In the case of role prompting as Elon Musk, the speaker created fake product names that Musk might invent or name. By incorporating these examples into the prompts, the language model can better understand the desired style and generate responses that reflect it.

However, role prompting alone is not enough. We need a way to evaluate whether the generated text truly captures the intended style or persona. For this, the transcript introduces an evaluation prompt. This prompt asks an LLM (language model with knowledge) of Elon Musk's style and naming conventions to evaluate the product names generated by GPT-3. The evaluator provides a percentage score based on how closely the names resemble Musk's style.

By running the evaluation

Role Prompting

 Fencing is one of those common techniques. It's really easy to do. What it is just asking the LN to play the role of a specific celebrity or someone that knows about that is a domain expert. You could also just ask it to play a specific job role. So yeah, as a math teacher, answer this question. And there is some evidence that actually does improve the intelligence of the model slightly to say. You can get a better math scores when you tell it to act like a math teacher, but the way that I tend to use it is really just to get it in the right style, because some things are subjective about L M. Responses and, some organizations want something in the style of Hemingway, whereas another organization might one as something in this solid. You need blights it? 

No. Whatever your preferences are. There's no good answer or right answer. But with role playing, you can at least get to a good subjective preference. So let me show you an example. Now. We have in here, the system pro the role setting and then the normal user prompt hair. And I'm just going to show you how those things come together. 

For the normal prompt. We're just passing in here, brainstorm a list of product names for a shoe that fits any foot size. That's the task that we wanted to do. Whereas with the roll prompt, we are telling it, you are. Elon Musk and you're brainstorming names, new products. I know you've given some examples and this is what I think is necessary to make rural prompting work in. And the shed because. Previous, he used to be able to just say you are. Elon Musk, whatever. 

And it would do a decent job or at least a better job. But what I'm finding now is that role prompting doesn't make that much of a difference unless you give it some examples of what Elon Musk would, I would name these products. So I made up some fake products. Had. So this is a refrigerator. It dispenses bad. 

And then I just lazy kinda called it beer, fridge, X, like space X. So not a beer fridge, like not a fine through. Which had products that Musk has invented or named. So that's that's really helpful, I think is if we can even make some synthetic examples of what you want and put them in that. And then you can get it into the type of character that you want with the Elon Musk Example here. 

So I'm just going to run this and you'll see.  That if we look at the role-play Anset. We're getting something much like Elon Musk. The universal fit shoe company. Fit all ex. Whereas with the previous example, we were just asking it without all the, all of these things that we're just getting. Adaptive fits, dried universal fit flexi foot. 

So I'm not getting Musk like names. So that's role prompting in a nutshell. But that's not where it ends, right? Like we need some evaluation metric. And typically for this type of evaluation metric, you need. An LLM to check essentially whether as a judge, whether you are falling ill, must style or not. 

So we have another prom tear, which has, Is it language model with knowledge of illness, style and naming conventions evaluated. Fucking product names. And determine if they sound like they're credited by ILA MUSC.  So we have the product names and then we need to provide the evaluation as a percentage. And then we have some templates here that we've said like product X or the product company or not the product. I said, I find that this really helps. If you're giving you some examples, just like adding examples to the previous prompt, where they helped as well. And what we're responding with here is instead of the product description and product names, we're responding with an explanation. 

And then the Musk likelihood. Now the reason I do it this way round by the way, is that if you ask it to provide an explanation first or some analysis or thinking first, it does a better job of getting an accurate percentage or a reliable percentage that you know, will come back again. And again, if you run it multiple times,  And so this is like a small chain of thought implementation.  All right. 

So we're going to run that. Which is going to see. What we're doing. Is running the.  Role play names with descendant names and just seeing  which ones are more Musk role-play names to really well. We have 90%, a hundred percent likely to be Elon Musk. We also have the explanation. 

So if we don't agree. And we can just check this and see. I think that, that works really well. And then if we look at the standard names, we've got a 60%, 50%, 70%. So we can really tell the difference. Like these names still sound like Musk, according to this. Because it's playful suggested innovation. Yeah. 

Like some boldness, a wound school nature often found a Musk's branding. Yeah, so yeah we could optimize maybe the the percentages hit by changing this prompt. And, obviously this is a prompt as well, so we need to optimize this prompt to. But it's usually much easier to optimize an evaluation prompt. Because you can. Rewrite the percentages yourself and say this should have been 20%, 

and then you could test again. This evaluator and then get to a prompt that gets it right and starts accurately repeating the 20% rule. Cool. So anyway, now you have this, you could AB test your original prompt provision. Now you can evaluate the must names. And then go back and just check whether it's actually following that role or not. 

What is the purpose of role playing in language models?

How can role prompting improve the performance of a language model?

What is the significance of evaluation metrics in assessing the effectiveness of role prompting?

How can role prompting be used to improve the quality of responses from a language model?

How can evaluation prompts be optimized in role prompting?

The funny thing about AI is that you can make it smarter just by telling it to act like a genius.

Act like a genius

UPSERT HEADQUARTERS – THE DAY BEFORE END OF QUARTER

You’re in a boardroom at UPSERT Headquarters and your boss, William Winters, Brand Strategist at Whipple, is asking you to figure out a way to protect the company’s data from Open AI. You’ve been researching the LM Studio and think it might be the answer.

We need to keep our data safe from Open AI.
I've looked into LM Studio, and it seems like it could be the answer.
But we need to decide how to use it.
What are our options? Do you have any ideas on how we can protect our data?\

LM Studio is an open source model created by Matter that is used to run models on a local computer. Despite being a smaller model than Chatpetite, LM Studio is slower and doesn't usually follow instructions as well as GPT-4 does. However, a new model created by Mystro 7 is getting closer to the results that one would normally get from GPT-4. 

For those looking to use LM Studio, it is important to understand the technical aspects of the model as running into errors is something that is common. Despite this, the text generated by LM Studio is still decent and with some editing, can be turned into something worthwhile. 

What makes LM Studio so unique is that it was created with the intention of creating narrative and character development. This is done by having two paragraphs per section. This allows for a unique fantasy style that is not found when using other models.

Despite the capabilities of LM Studio, it is still not as good as the free version of Chatpetite. This is why many people are hoping that the model will get better, as having something that is as good as GPT-4 would be extremely beneficial to those who want to create interesting stories. 

Overall, LM Studio is a great model that is able to generate interesting stories and narratives. Despite its shortcomings, if used properly, LM Studio can create some unique stories. It is important for those using the model to understand the technical aspects as well as the capabilities of LM Studio in order to have the best experience. With some practice and patience, LM Studio can be used to generate some great stories.

LM Studio

 Okay meet. Lama this is an open source model from matter. They release it which is really great because they spend all the money training it and then just released to open source. That means I can actually run this somewhere on computer and then there are different. Sources online where you can run this, like on hugging face or replicate. But I really wanted to run it. At my own computer, I'm using something called L M studio. Which you can download is just a Mac app. And and then once you've got that, Set up, you can download anything from hugging face. 

Like this is a downloaded the Lama to chat 13 B. Model and you can prompt it just like you would  So just going to paste in my prompt. And see how it does. What I've found in general is that obviously running it on your local computer is a little bit slower. So even though this is a smaller model than chatting petite that the response has actually come out relatively slower. The other thing is that it doesn't follow the instructions usually as well as GPT four, does. Does. These models, these episodes model is getting closer to what you would get from the free version of challenging.  And in fact, there's a new model out by Mistral which the Mystro seven. The model, which is actually pretty good. And pretty close to chat cheap T levels if you're comparing it to the free version, but nobody has managed to get to the actual results that you would normally get. From GPD for. 

So while this is generating, let's familiarize ourselves with what we came up with jazzy beauty. You can see. Wrote some pretty interesting stuff here. And, we have two paragraphs per section. And it's a unique fantasy style. With actual narrative and character development. Let's see if we can get that We run into an error. Sorry. You do run into this sometimes, unfortunately running these models require some technical ability in some cases. 

But I think in general, I would say these models aren't quite there yet. They're not as good as what you get from chat, GBT, even the free version. We're all hoping that these models will get better now because it would be very cool to have. Something that's as good. 

Now you can see here that the text isn't bad, it's come up with some interesting stuff here. It has, pretty decent introduction. But again, I prefer T's version.

What is the name of the open source model used in the tutorial?

What is the name of the software used to run the model?

What is the name of the open source model from matter?

What is the difference between running the model on a local computer versus online?

What is the name of the new model released by Mistral?

We're just now getting to the point where OS models are as good as ChatGPT (the free version), except they have one major advantage: no data sharing with OpenAI!

I don't want to share my data with OpenAI

Alright, team, here's the situation. We've been tasked with making some creative decisions for our coffee shop interiors, but we don't have enough data to work with. Our boss wants us to use memetic analysis to analyze and tag images of coffee shop interiors using AI. This will help us identify patterns and trends in the industry, and inform our design decisions. So, let's roll up our sleeves and get started!

I know it's frustrating that we don't have enough data to make these decisions, but memetic analysis can really help us out here. \nBy analyzing and tagging images of coffee shop interiors, we can uncover valuable insights that will guide our design choices. \nThis process will allow us to understand the trends in the industry and make informed decisions that resonate with our target audience. \nSo, let's dive in and see what we can discover through memetic analysis!

Memetic Analysis: Uncovering Insights with AI and Vision

In today's digital age, understanding consumer preferences and market trends is crucial for businesses to stay competitive. This is especially true for industries like coffee shops, where interior design and ambiance play a significant role in attracting customers. But how can businesses gain insights into their competitors' designs and make informed decisions about their own establishments?

One innovative approach is using memetic analysis, a method that combines artificial intelligence (AI) and computer vision to analyze images and identify patterns. This technique can help businesses gain valuable insights into the typical interiors of their competitors and make data-driven decisions about their own establishments.

The first step in conducting a memetic analysis is to gather a large dataset of images from various coffee shops. Manually analyzing these images can be time-consuming and inefficient. However, with the advancements in AI and computer vision, it is now possible to automatically tag these images using machine learning algorithms.

To scrape a large number of images from Google, a Google image scraper tool can be used. This tool automates the process of collecting images based on specific keywords, such as "coffee shop interiors." Once the images are downloaded, they can be analyzed using the AI-powered Google Cloud Vision API.

The analysis begins with inductive coding, where two different images are randomly selected and compared. The images are converted into base64 encoding and sent to the API for analysis. The prompt asks the AI to compare the two coffee shops using inductive coding and identify similar and different features and design elements. The response from the API provides valuable insights into the shared and unique attributes of the two coffee shops.

The next step is deductive coding, which involves analyzing each image against a set of predefined labels. This step is more time-consuming and expensive, but it provides a comprehensive understanding of the design elements present in each coffee shop. To speed up the process, asynchronous coding can be utilized, allowing multiple API calls to be processed simultaneously.

The results of the deductive coding are stored in a dictionary, which includes the labels and corresponding coffee shop images. These labels can be customized based on the specific design elements or features that the business wants to analyze, such as natural light, wooden elements, or suspended lighting.

Once the analysis is complete, businesses can gain valuable insights into the design choices of their competitors. By comparing the frequency of different labels across the dataset, businesses can identify popular trends and common design elements. For example, if a high percentage of coffee shop images include patrons,

Memetic Analysis

  Okay. Say you're designing a coffee shop and you want to figure out what the typical interiors for your competitors. So you want to understand what type of furniture they using, whether they have plants whether there's a chalk, a chalkboard on the wall for the menu. Like this. 

And you wanted to do that analysis and see how often a certain things come up versus not. So one interesting way to do this. Is instead of manually looking through a bunch of images. You could tag them automatically with AI because aegypti vision is actually pretty good at this now. What you need is you have to scrape. Google images, which is actually the hotpot. But thankfully there is a tool that does this. 

What do you say if you just search a Google image scraper? Yeah, literally just. That's just such a Google image of scraper. And that was the Fest result here. But however you get the data. It doesn't really matter, but I'll show you how it works. I.  I just, this is in the main.py. So when you download this to your computer from get hub then you have this curd you just put in here. What are the different keywords that you want to search for? 

I downloaded a bunch and you can see. Got a bunch of, coffee shops in Beijing. Got a cough coffee shops in the room. And then also New York, I'm willing to compare them potentially. So I'm going to just run an abbreviated. Part of this analysis I would say by the way, it takes a while to down the them say I downloaded a bunch and it doesn't always work. 

So you can see, I didn't get 80 every time we got 36 and Sydney. But once you literally just run Python, main dot PI. And you should see these images. Stop appearing in hand. But that's not the part I'm going to teach you because that is someone else's code. And I'll let them teach you that in that read me. But what I'm gonna teach you is what's due when you have the data, right? 

This is a script for the Jeep T vision API. And you just need to run. These libraries here, but I have all the code written here. I had to pull in. What are the different coffee shops, right? As you can see how many photos you got just looking at the first five here. See, I've got 4,500 just from New York. 

Got a bunch of Sydney. Et cetera. So. What I'm going to do is I'm just going to.  Just limit just to Let me just a little bit to Sydney.  

Just for testing because a visionary pair is expensive. If you're running a bunch of times, it also takes a little bit of time. So I'm just gonna run it on these 33 images and in Sydney.  All right. And there's two pastas. One is inductive coding. And then the second is deductive coding. 

Inductive coding is when you're looking at these and you look from, for patents and you're comparing the Migo. Okay. That one has a coffee. You know that one has a chalkboard for the menu. And then there's other ones that have chalkboard for many use as well. So maybe that's something that I want to hear again, is another chalkboard. 

So I've noticed that pattern. And now that I know there's a pattern. I want to find that everywhere. So there's another truck board. There's another chalkboards. So that's deductive coding, inductive coding is coming up with the patterns, comparing and contrasting to find what's different or what's similar. And then deductive coding is then going and finding those patents everywhere else. So we're going to do those two things here with a GP vision. All right. 

The first thing we're doing is inductive coding, and we need to be able to send an image to opening eyes. So the way that you send an image is with the base 64 encoding. This literally just takes an image and then turns it into some texts. And that you consent along with your API call.  And this could does, is it just looks in the folder and then it just chooses two images at random. From the different folders that we have. 

So we just have the one folder right now, but if we have multiple folders that. It would look in all of them. Oh, I did mention, I forgot to mention one thing when using Google images scraper you'll probably need to download. I had to download this Mac. 64 Chrome crim for testing. 

When it scraping these images you need that in that. Here I've just put us in folder called WebDriver and then it opens that up. That's, I've just mentioned that because I ran into an error. All right. So back to the inductive coding. So choosing an image pairs, choosing two different images at random. And then it's comparing them. So it's getting the base 64 string, and then it's using this prompt here compared these two coffee shops using inductive coding. What features attributes, elements of design are similar, different. 

And then I'm asking for a structured response here. That's just how you pass it into GPC vision, by the way. You just. To client chat completions. And then you pass the prompt. But then you also pass in the two images as well in the same content string. And I'm just using the user prom tear, but you could also have a system prompt at the top. Cool. 

So let me just run this, see what we got. So it's selected to these images at random. Number 30. 25 and number 31.  And you can see, it takes a little bit of time for the image response is this is opening eyes, biggest modal. Here we go. So now we've got some labels the there's some things that they share. 

So coffee shop a and B. Both have modern designs. They both have wooden elements. But only cough shot B has natural light. And own the coffee shop a has a spacious interior. So that's really interesting. Now we have some labels. We have a dictionary of labels. We can run that and then save it. So if we look at. We should have hit some clean levels so you can see. 

Yeah, there's a bunch of labels here. For these different things and it's rewritten them. We've got the actual kind of folder that they're in the label. And then the phone name.  Cool. So now you have your level dictionary and that's useful in itself because you could run that multiple times across multiple images. 

So we just compared to those images, you could compare every single image against each other. It just takes a little bit of time. That you can run again as well and get a bigger dictionary of data. Now it comes to the deductive coding. And this is the one that's more time consuming, more expensive. 

The way, the reason why I've set this up as asynchronous is. Because otherwise it takes a very long time to run because you're going through every single image and saying and every single label and going, is this label in this image, does this coffee shop have wooden elements? Does this coffee shop? 

How bought-in elements. Does this coffee shop have wooden elements, right? And then you go through all the other labels about, does this coffee shop have suspended lighting and does this coffee shop have a seating areas? There are different tricks for doing this faster or slower. But for me, I found if I do an asynchronous. It means I can get all the labels at once for a single image. And they all run quite quickly. So I runs a little faster. So again, this is here, we're just passing one image in and we're just passing the user prompt, which is much simpler. 

If the label applies to this image, reten one, otherwise reten zero on the output one, a Sierra.  All right. If we get a one, then we are going to return the label. And if we don't, then we're not going to attend on. So this. This does this, the asynchronous code would just set up a bunch of tasks. 

So we run it for every image and then we wait until after all the tasks had done for that image. It's processing all the labels at the same time. And open AI can handle multiple API calls at the same time. What will happen is instead of waiting for each one to come back, it's going to process all of them said time. 

We're going to get through asset. If we're doing 10 labels, we'll get through all 10 at the same time, rather than waiting. If one takes a minute. We're not waiting 10 minutes. We're getting all of them in one minute. So that's the benefit of asynchronous. Cool. And then we're just going to return the labels. 

If the. If the label wasn't there.  Great. And then actually running the deductive code and we just pull open the cleaned labels, Jason, that we have before. And then we have the clean labels. Which which loads to Jason from that dictionary we had, and then it finds the values. 

I've also limited it to just five labels per coffee shop. Otherwise, it's going to take a bit of time.  Cool. All right. So let me just run this. And the processing coffee shop Sydney, you can see that all three of those came back to this in time. But some of them are taking a little bit longer. But we're not waiting for one to finish it before we got the other. 

Cool. So here we go. So now we're processing coffee, shop Sydney.  

And let's see how quickly they come back. Yeah, this is Sydney 30. Okay. Now we're processing Sydney 10. Getting hanging light fixtures. So this is only coming back. As positive if it does have that label, if that makes sense.  

Cool. That's how. Movantik analysis works with vision it's really powerful. Once you figure that you can apply this to pretty much anything. Like any time you have a lot of images, like maybe in your ads or on your website. You can do some analysis to figure out what labels are up. And you can define the labels as well. 

You could sell it. What sorts of things you're looking for. You could give it a D list of things. However you want to do it, that's fine too. You can just add new stuff to this. This clean labels thing, right? You could just add in some new.  New ideas, right? 

Like I want to. I wanna find, this is customers queuing, right? I want to find out if there's a barista that has a tattoo, right? Like you could just add that as another option, copy. This one has Tattooed.  

Barista.  

All right. And there's already funding. If the barista is present in this bit, then we're going to see if they have tattoos, visible tattoos, cool. That's how this works. And then what you're going to get back is you're going to get some, a file name like this. This is just a big list of all the different labels, right? 

So it says coffee shop, New York 42 has the label of street view. This one has the label patrons present, right? And then you can do some analysis after this. To basically figure out okay. How many time, how many does it, how many labels does this apply to you? 

So how many of the coffee shops has this label? I'll give you an example. I'll show you an example here, cause that's still processing. Cause it's going through 33, images is an example from New York. Then I ran earlier, so you can get a list of all the labels into a data frame. This is like an Excel spreadsheet. And then you can print out what percentage of the total images have these different labels? 

So here we're finding that 41% of the patrons that 40% of the coffee shop images, show patrons present. So to me you want to really pay attention to the ones that are on almost everything. Because that's almost like a rule of the game. If you don't have patrons present in your coffee shop image, that's probably really bad. And you definitely want to at least have a stance on that. 

Like maybe have a good reason for not having patrons present in the image of your coffee shop. When you're uploading that image to Google my business. But you should at least address that have the strategic choice here. Similarly, for things that are not very popular you also might want to think about these as a differentiator. 

So exposed brick interior, you could argue that, that's not very fashionable anymore, and that's why only 10% of people had it. Or you could say, this is going to be fashionable. I can see that this is a trend as, or I'm going to stop using this right. That's how. You do this type of analysis. 

Obviously what it doesn't tell you is like which choices to make and what designs to have. But if you're not a designer like me and you want to be a little bit more data to make your decision, this can help inform your decision to give you a map of. What is what are the different creative decisions you need to make? 

When you're doing something designed or visual. All right. Hopefully that's useful. This is my little hack for, being a data guy, but also playing in the world of branding. Hopefully you find it useful too.

What is one potential benefit of using AI to tag images in a coffee shop analysis?

What is the difference between inductive coding and deductive coding in the context of analyzing coffee shop interiors?

What is the purpose of using AI to tag images in the coffee shop analysis?

What is inductive coding in the coffee shop analysis?

How can the coffee shop analysis help inform design decisions?

There are millions of decisions you need to make when starting a business that are too small to test, but are too risky to make without data.

How can I make creative decisions when I don't have enough data

You’re in a meeting with the CEO and your boss, as well as the rest of the executive team. They’re discussing whether the product is priced too low, and you offer a way to find out: Gabor Granger pricing analysis.

That’s interesting, so we basically survey people and ask them what they would pay
But not directly like “what would you be willing to pay”
Instead it’s like “would you buy this product if it was $10?”
Then we keep increasing or decreasing the price depending on if they say yes or no?
Yeah I’m willing to give that a shot: it’ll be better than just guessing!


Gabor Granger pricing is a type of economic analysis that attempts to identify the optimum price for a good or service. It is named after Zoltan Gabor, who developed the concept in the 1930s. It works by asking a customer if they would buy the product at one price, then if they say yes asking if they’d buy at a higher price, if they say no asking if they’d buy at a lower price. This keeps going until the prices are exhausted, and we can draw a demand curve out of the aggregate responses. This method is designed to avoid asking customers “How much would you pay?” because most people don’t know off the top of their heads. However if you show them a price and ask “Would you buy this product for $10?”, they will get an instant reaction and be able to decide if it feels high or low. 
There are a number of advantages associated with Gabor Granger pricing analysis. Firstly, it is relatively simple to understand and use. Secondly, it takes into account a wide range of factors that can affect prices, such as costs, demand and competition. Third, it can assist in setting prices that are competitive, yet still provide a good level of customer satisfaction. Finally, it is a good option for businesses that are looking to maximise profits, or decide on the tradeoff between profit and volume. 
There are also a number of disadvantages associated with Gabor Granger pricing analysis. It can be time-consuming to carry out and may not always produce accurate results if people don’t answer truthfully about their willingness to pay. Gabor Granger pricing analysis is most appropriate in situations where there is a need to price a product or service with not much competition, as it doesn’t take into account competitor price changes and therefore is prone to causing bidding wars. It is important to remember that Gabor Granger pricing analysis is only one tool that can be used to price goods and services. There are a number of other pricing methods available, and it is important to select the most appropriate option for each individual situation.

Pricing Analysis

00:01 Hey, let's learn how to do G or Grainer pricing analysis. So this is the end result that we're trying to get to, and what we're trying to do is draw this to man curve and see at what price should we set the price of our product in order to maximize the amount of profit that we make.
00:18 So I'm gonna walk you through how to do this, but first, just take a look at how the survey is designed.
00:23 So a Ga Granger survey is a little bit complicated. You usually need some specialist survey software, or you need to use custom variables and something like Typeform.
00:34 But essentially what you need to do is ask at the starting price, would you purchase this product for $11? If they say yes, then you increase and say, Would you, you know, purchase at 13?
00:44 And then you kind of keep going and test the different prices. You could also say, if they say no, would you purchase at eight?
00:52 If they say yes, would you purchase at 10? You know, so basically it's kind of, you know, starts at a certain point and then goes up or down depending on what they answer.
01:02 So this is a, you know, once you have the survey data, you can draw the pricing curve so the demand as in how many people are willing to buy at that price.
01:12 And then also the, you can figure out the profit as well, which is what we're gonna do. Cool. So once you've done this survey then this is the kind of data you'd have.
01:26 Like, so you'd have respondent ID on the side, then you'd have different prices.  I've done from $10 to $50 because this is a this is for the brand Ultra which is a men's grooming brand.
01:40 And you know, we can see respondent one said yes at $10, yes, at 15, yes, at 20, and then yes, at 25.
01:49 But then they stopped saying yes after that. So it was all zeros afterwards. Whereas say respondent four only said yes at $10 and they said zero for the rest.
01:59 So then you can randomize this as well to make the survey more it can kind of randomize the starting point to make it better.
02:08 Because one problem you might have is if you always start at 10 then you're anchoring them towards a low figure.
02:15 Or if you always start at 50, then, you know, then maybe you're kind of anchoring them too high and they're gonna say a price that they actually wouldn't buy it.
02:24 But whatever the kind of statistical issues with this, really, you're just looking for a smooth curve at the end. And, and this's gonna give you a decent range a respectable range for you to be in.
02:36 Cool. So you have all the survey data, in this case surveyed 94 people. And then all you need to do is just add up the ones essentially, right?
02:46 So I do it in average and an average of, of any column that's either a one or zero average would just be the percentage that were ones, right?
02:55 So 93% of these were ones was only 5% of People said 50. So then you just need to make some assumption about the market size.
03:05 In this case. We've assumed there are a hundred thousand people in the market. And what this is telling us is that 93% of people so like, yeah, it's actually 92.5 percent of people would become customers if we priced it at $10.
03:20 Which is great, you know, you wanna capture the whole market, right? But it's not so great because actually you know, there are, you know, there are options with higher revenue and and we actually making a loss here.
03:32 So the way you work that out is that the revenue is just the number of customers, tons the price, right?
03:39 So, you know, lower price, you need more customers in order to make the same revenue. And then the profit, we've just made some assumptions for this business.
03:47 So they have to work out the profit, we take the revenue, and then we subtract this they've got in this case it's a million dollars in fixed costs, which is important because, you know, whatever happens if they don't sell any units they're gonna lose a million dollars.
04:05 So yeah, that's a more realistic way to look at it rather than just margin because you know, if you're a startup, you have like a, you know, you have to pay for a warehouse to house all these products, right?
04:16 Like you have to pay for staff and that you, the staff still need to get paid no matter how many units you sell.
04:22 So you usually have some fixed costs that you have to factor in, and that necessitates you selling at least a minimum number of units in order to kind of get to the right level.
04:31 And then you have the variable costs. So in this case, it only costs two bucks to make the product. So you know, we're just multiplying the number of customers by two bucks and the, these two things into place.
04:41 So you can see that we lose money on the $10 even though we have 92,000 customers with with $15 we have 82,000, so we lose 10,000 customers.
04:55 We're gaining a lot in revenue. So now we're making 1.2 million in revenue, and that's enough to cover our fixed investment of a million.
05:03 And also the unit costs. So we're actually turning a small profit here. And and then you know, don't worry about this line.
05:10 I'll explain that in a second. But then it kind of keeps going on, and these are basically the estimates. And you can see that you know, this is the revenue maximizing point but you're not making very much profit.
05:22 Whereas if you increase the price to say, $25, you're only making half of the market by your product. But you know, and, and your revenue is lower, right?
05:31 Like it's $20,000 lower than the $15 option. But that might actually be better to have less market share and higher profit.
05:39 So think about this, like Apple, you know, Apple could price their phones cheaper and they would get a lot more people buying iPhones, but they would make less profit.
05:47 And, and right now they make the most profit out of any of the other phones in the industry. So they've kind of chosen a price point that's higher up here.
05:56 And You can see $125,000 in profit. Now, if we keep going it's actually gonna get worse because yes, we're making more per unit but we're selling less units, right?
06:09 So we're losing a lot of market share here, this share here. So you know, we're getting like a hundred K in profit by selling 39,000 units.
06:18 We're getting 78 k in profit by selling 82,000 units. So, you know, I mean, it's really up to you in terms of like what the business goals are and how much risk you wanna take, you know, things like that.
06:29 But it might actually be better to have a higher price product and sell less units because it maybe than you know, that's easier business to run or, you know, whatever.
06:38 So like a lot of luxury or boutique businesses operate these, you know, higher price and you know, higher price levels but you need to sell a certain amount of units to cover those fixed costs.
06:50 So for example if you, if you only sell 30% of the market you're only just making a million dollars in revenue and that's, that barely covers your fixed costs and it doesn't cover, doesn't left anything over, left over for the $2 per unit.
07:05 So, so you actually lose money $17,000 in this case, and that kind of keeps going on. So yeah, you could price it 50 bucks, but you know, only 5% of people wanna buy it that rate, right?
07:15 So you can actually lose, you know, a quarterly, three quarters of a million dollar dollars here. So that's a terrible choice.
07:23 So it just shows that, yeah, like yes, you can choose your pricing and you can be strategic. And once you map this out, you can make a more informed decision.
07:31 Really, it's a market that decides, you know, what you can get. Like, you can't just say, Well, my product's worth 50 bucks.
07:37 That's really for the market to decide. Cool. So now what is this line at the bottom here? Well, what we've done is we've drawn the curve here, see this curve, this trend line.
07:49 And the way that we've done that is if we go to edit chart and then we go to customize series or on data series one.
07:56 So that's that if you click on trend line, and there's a slightly different way to this in Excel, but you know, works is the same you can stick a trend line on and then you can say use equation and then show the R Square.
08:09 And what that does for you is basically just plots the nearest fit to all the, all the points here. So you can see some of the points are slightly off that one, slightly above, that one's slightly lower.
08:18 And this is a in this case we used a a polynomial. So there's like different options but polynomial polynomial fits best should do that polynomial fits best cuz it lets you kind of do this curve up and down.
08:36 Cool. So the, in this case, the, the Oscars was 0.95, A perfect fit would be one. So this is a very good fit.
08:45 And then and then that gives you the equation to derive the, the profit, right? And so it predicts given a price what profit you're gonna get.
08:55 And the way it does that is just this complicated formula, but you don't have to worry too much about it.
09:00 Basically it's saying you start with the value of minus 73,000 or sorry, minus 736,000 7 68. So then you add 7,020 yeah, so 70,278 multiply by x and x the price.
09:20 So if it was $10 and it'd be 70,278 times $10. So it's about 700,000. And then you also add minus one for, for two times x to the power of two.
09:37 And that's basically what gives this kind of coving up and down chip. So you don't need to worry too much about the statistics here you know, just know that that's what that's doing.
09:46 But all I've done here is just copy this function across, right? So is minus 7 36, 7, 6 8, and then plus 70 2 78 times B one B one is the price plus minus 1 4 42 times B one the price again to the power of two.
10:03 And there we go. Now we have our prediction. So we could do this for any price, but we've done this across all the prices here.
10:10 And the reason why you wanna do this is you see there are some lines are, you know, dots are above and below.
10:14 And really what we're trying to do is smooth this out. Like there's some statistical noise here, right? So we're just kinda smooth this out and find the optimum price, and it is at the $25 mark.
10:25 So making, we, we would predict we're making 118,000 in revenue. And the reason why the smoothing out is important is that because, you know, of the survey data, the $15 price mark, it's actually like higher, higher than we would expect.
10:38 So we might look at this and say, Hey, we're still gonna make this 78 grand in profit, but actually the the function or the prediction is that we lose 7,000.
10:47 So we're playing with fire there if we're gonna, you know, stick with this ROS estimate. And and instead we would kind of wanna stick at least, you know, somewhere in this range.
10:57 If you wanna optimize, you know, slightly more to revenue, then you could be at the lower end of this range.
11:01 You wanna slight optimize slightly more to, to margin then you could be at the upper end of this range. You know, but you know, it's giving you a place to choose your price.
11:12 Cool, Hopefully that was helpful.

Why are fixed costs important to take into account with regards to profit?

What might be the rationale for maximizing revenue over profits?

Why might Gabor Granger analysis not be a good fit for industries with intense competition?

Are you charging enough? Most businesses guess at pricing and move on, but that approach is going to leave money on the table. How do you set the right price to maximize profits?

What price maximizes profit?

Building a marketing mix model is not just about finding the right parameters, but also the hyperparameters used to transform your data to account for adstocks and saturation.

You've been looking for an excuse to learn how to use evolutionary algorithms for building models automatically, and now this might be your chance. You hear on the grapevine that a colleague is struggling with their model.

Yes you heard correctly
We're struggling with the GoolyBib model for Italy
I'm not satisfied with the model as it stands – the error is over 10%!
This could be a good use case for automated hyperparamter optimization
Doing it manually is getting us nowhere fast!
I'd appreciate your help

Every model has parameters: these are the variables used to make predictions when the model is built. For example your model might say that Facebook Ads drives a cost per purchase of $5. How the model arrives at that conclusion is dependent on Hyperparameters: parameters that control how the model learns the right answer. To take our Facebook Ads example, our cost per purchase might get worse at higher spend levels due to saturation, or there might be a carryover effect where spend today affects performance tomorrow. 
These transformations of the data are key to building an accurate model, but it's hard to know ahead of time what values are correct. The manual way of figuring this out is to choose a value for a parameter, for example an adstock level of 20% (20% of your spend from today has an impact tomorrow), and then move that value up or down to see how it impacts model accuracy. If you increase it to 30% and accuracy of the model goes up, you know that's a better fit. The problem is that each parameter affects the behavior of every other parameter in the model, so it can be hard to find the right value.
So there have emerged several strategies for dealing with this task of Hyperparameter optimization. The brute force approach is a grid search: try every possible value until you find the right one. This isn't always possible however, because there may be more potential combinations than there are stars in the universe! Random search lets you set a limit of how long to look for the right values, randomly guessing within a budget. The method we're using today is an evolutionary approach, which learns in a more efficient way than random guessing. It deploys a population of potential solutions, then kills off the worst ones (with the highest error) and replaces them by mutating the best  performers in the batch. With hundreds or thousands of iterations we can efficiently arrive at the best estimate for even a very complex set of parameters.

Hyperparameter Optimization

GoolyBib Italy model
Hi,
Thanks for offering to help: please see model attached. 

We have been choosing the values for theta (adstocks) and beta (saturation) manually and this is the best combination we've got to so far.

Best,

Statistician and Management Consultant

To optimize the whole, we must sub-optimize the parts.

Marketing campaigns tend not to have a linear relationship with sales. As you spend more money on a channel, the efficiency decreases because you have to outcompete a greater number of other advertisers for ad inventory. In order to model a non-linear variable in a linear model, we must transform the data. The Power saturation curve we're using simply takes the advertising spend to the power of beta. But what is beta? It's a Greek letter, because by convention mathematicians use them to identify parameters in a formula. 
We don't know what the correct value of beta is – it's our job to estimate it. The way this is done is by trying a lot of beta values from ~0 (completely saturated) to 1 (linear) and see which value best improves the model. If we see the accuracy increase from 0.3 to 0.4, but then decline again at 0.5, we know the true value could be around 0.4.
We do a similar job for adstocks, to estimate the percentage of advertising impact that's felt the next day after spend. For example if we choose 0.2 then 20% is felt the day after, 20% of that the next day (4%), and so on. The process is the same: calculate the accuracy of the model at each level of adstock, and see what it does to accuracy. Of course this can get complicated to do across multiple variables, as if you change one it changes the whole model, and the odds of ending up with the best possible combination are low.

Theta and Beta

For models with up to 5 or 6 parameters, it can be feasible to adjust the model manually and find a good enough solution. However this task quickly becomes impossible with more parameters because the number of potential combinations increases exponentially.

Finding numbers for Greek letters

Rather than manually choosing the value for each parameter, let's get Nevergrad to do it for us. It uses an evolutionary algorithm to build hundreds or thousands of models to find the right parameters for us.

After digging through the Nevergrad documentation and using some marketing mix modeling code you had from another project, you have something ready to run.

Nevergrad is a Python package created by the Meta (Facebook) team, for gradient-free optimization using techniques. It works using differential evolution and particle swarm optimization, which optimizes a problem by iteratively trying to improve a candidate solution with regard to a given measure of quality. 
This process is called evolutionary, because it is based on the ideas behind evolution in nature. The simplest evolution strategy operates on a population of size two: the current point (parent) and the result of its mutation. Only if the mutant's fitness is at least as good as the parent one, it becomes the parent of the next generation. Otherwise the mutant is disregarded.
Such methods are useful because they make few or no assumptions about the problem being optimized and can search very large spaces of candidate solutions. Because it does not use gradients like other optimization methods, it can be used on optimization problems that are not even continuous, are noisy, change over time, etc. This is perfect for machine learning where the actual model might be a black box, where changes in one part of the model can give non-linear responses in another part of the model. The downside of evolutionary algorithms is that they are computationally intensive, so take a while to run, and do not guarantee an optimal solution is ever found. 

Nevergrad Optimization

Hi, I'm going to walk you through how Nevergrad works
I'm gonna take the file here, and just going to make a copy, so we have our own copy in drive
First you want to install Nevergrad
So if you just click into that cell and press shift + enter, or you can click play here
That's just downloading the library and installing in on the machine Google is giving you for colab.
Now we just need to load in all of the libraries that we're going to use
Just shift + enter, or click the play button.
And these are some functions we're going to use as well
We need to authenticate Google drive so we can import the data
So if we just run that cell, then it'll give you a link to click through, and you just want to authorize, then copy and paste
Hit enter
Now if we run this cell
Its going to pull in the data from this Google doc, and it's going to display it here as well
So we know that we've got the right data, its the number of days, then facebook spend, radio spend, tv spend and then revenue
Now let's plot the data so we see how the data is trending over time
So we see here it's a plot of revenue, and then we see all the facebook, radio and tv spend underneath
Now in order to run Nevergrad we need to do a little bit of setup
First we need to get all the spend columns, so the media variables are if 'spend' is in the column, and then we just need to create the X and Y variables
Y is revenue, that's what we're trying to predict
Then X is what we're trying to predict it with
This is an adstock formula, it helps us take into account that there will be some lagged effect of advertising
And Nevergrad is going to help us find the right adstock values
It's going to try lots of theta values here
And it needs that function in order to transform the data in each run
Same for saturation, this is diminishing returns its the same thing, its just a power transformation, with just one variable
You can have more variables, if you do something more complex, like a weibull adstock or a hill saturation curve.
Now we need one big function to model everything together
So it needs to take in the facebook spend theta, facebook spend beta, and then same thing for each other channel
And then it just pulls it together, and then creates the adstocks, and the saturation, so it does adstock first, and saturation after that, which is best practice
Finally it creates a new data frame, basically an excel spreadsheet
then creates the X and Y values, and splits them into X train, X test, and X train X test
So this is a train test split that lets us check the accuracy, with data we haven't seen before, in the model
This prevents overfitting of the data
Finally we have a linear regression model
Although you could replace this with whatever model you want to use, if you want to use Ridge regression or something different, and then it fits the data, and gives us predictions, we then calculate the accuracy metrics
So normalized root mean squared error, is just an accuracy metric that gives a percentage, that penalizes really high percentage values, more than mape 
The mape is the mean absolute percentage value, and that's just a general percentage you were wrong each day
We also have the R squared as well
Then we kind of wrap this function and get all of the results, but only return the mape value
This function is what nevergrad is going to use to optimize to
So here we're optimizing to the mape
But we could swap this out for nrmse or r squared if we wanted to
Now we just need to make these scalar values
Don't worry too much about this, but we're essentially just setting the bounds of what the parameters could be, and then we're going to run the optimizer here
The optimizer, this is actually Nevergrad doing its thing
We're passing in the instrumentation into the parameters, and then we're giving it a budget, and we're saying this is how many times we want you to run a trial
And the trial is basically the algorithm choosing what kind of parameters to set or to try next and then run that trial to see if those parameters worked
Then we just want to log the error values, so we can do a chart afterwards of how quickly it approached the lowest mape, 
so that's what we're doing here
we're just registering the callback
every time it tells us the answer for that trial
ok and we're going to set a timer, so we just see how long it takes to run
this is the optimizer function, this is where its actually minimizing the error
this is like where all of the work is actually happening
so when you run all of this
oh I have an error
main function is not defined
if you ever get this, its because you probably ran things out of order
so I'm just going to go back and run this one here
then run this one again
there you go, you can see it took 1.8 seconds to do those 100 trials
and these are the values that it came back with for facebook beta and facebook spend theta
So these were the parameters for the adstocks and saturations, that got us the best accuracy out of all of the models they tried
You can also kind of push those parameters into the function that you created in order to get the MAPE
So it's 0.076 so it's 7.6%
Then you can see if you plot
So just run that so you can see if it works
It's going to be a little bit different each time, because its an evolutionary algorithm, so there's some random elements in there and it wont' always be the exact values
Just going to run this
We can see here that we started off with a very high mape, it's 18%, and then even spiked up to 20%
Then over time it kind of evolved towards the better values
It got a lot lower, to 0.08 and then stayed around there for most of the 100 trials
Then if we just run the build model function, its going to return us the mape value and the nrmse, r squared value, the model and also the result
So we just run that and its going to give us those values
We can also plot the accuracy
So if we plot the figure, and we take the revenue and the prediction from the result, which is the data frame, then we are showing that
We're seeing here the revenue is in purple, vs the red prediction
That's how you run nevergrad in order to find the right parameter values




Use Nevergrad

What are the benefits of using Nevergrad vs other optimization libraries?

Describe in your own words the steps taken in the Google Colab notebook to use the Nevergrad library

That's great to hear
8% error is much better than we had on our manual model
Looking forward to seeing if we can improve it further by dialling up the number of models

This approach works if you're running up 2,000 models, but it takes a while and hits an error if you set the budget too high. To reach 10,000 models we have to parallelize the operation, meaning apply multiple 'workers' to solving the calculation, rather than calculating each iteration one at a time.

Better than random guessing

We need to make some modifications if we want to scale up the number of models we're building. It will help to parallelize the operation to allow multiple calculations to happen at once.

We have a model that works – we're already getting better accuracy. Let's modify our code so we can run a much higher number of models.

Nevergrad offers parallelization right out of the box, you just need one other library called 'futures' to make it work. Futures lets you create a ThreadPool which manages the multiple workers you create to do the calculations. These workers can carry out tasks asynchronously, as if they were their own mini computers, and then pass back the information to the main function. We run this within a 'with' statement to ensure they're cleaned up properly and shut down after everything finishes executing. 
You can have as many workers as you have cores available for processing on your computer, which you can find with the os library `os.cpu_count()`, though it makes sense to leave 1 core for anything else your computer needs to calculate. For demonstration purposes we're going to just use 2 cores, which is the minimum needed to calculate 10,000 models. We set this in the optimizer on the first line with `num_workers=2`. The following file has the updated script we'll use.

Parallelizing Nevergrad

Copy the code from the last model to show the output of the parallelized code. What is the MAPE for the next model? Was it significantly better?

The parameter for TV spend beta changed from 0.45 to 0.10 in the new model. How would we interpret that?

What does the chart showing mape over time tell you? What does it look like Nevergrad was doing? Is this behavior what we'd expect?

Thanks for sending the results across - very interesting
I thought dialling up the models would get us a better result
However seeing that it jumped around a lot and tried lots of options gives me confidence that we really did end up with the best allocation
Thanks again for your work on this
We can reuse the script with modifications for every new model we need to optimize!

Scaling to 10,000 models

It's tedious to try different diminishing returns manually to find the best one. Facebook's Nevergrad package can try thousands of models for us!

Let's try 10,000 models

Alright team, let’s dive into today's tutorial on creating consistent characters in AI-generated images. We'll explore using a character reference URL for consistency and in-painting to vary scenes with the same character. We’ll also look at a blending hack, though it might not be available forever. Let’s get started and elevate our creations!

I'm not sure how creating consistent characters will really benefit our fizzy drink company. \n
I mean, isn't it more important to focus on our product and marketing strategies? \n
But I guess if it can enhance our brand image or help us in storytelling, it might be worth exploring. \n
I'm curious to see how this tutorial will change my perspective. \n
Let's give it a shot and see what we can learn!

Consistency is key when it comes to creating memorable and believable characters in storytelling. Whether you're writing a book, a screenplay, or a video game, ensuring that your characters remain consistent throughout different situations and scenarios is essential for maintaining their authenticity. In this blog post, we'll explore some techniques and tools that can help you achieve consistent characters in your creative projects.

One effective method for creating consistent characters is demonstrated in the transcript above, using OpenAI's DALL-E model. The transcript highlights how you can generate different variations of a character by using different prompts. However, to maintain consistency, a specific parameter can be added to the prompt, known as the "C dot E F character reference." By adding this parameter along with the character's URL and specifying a desired consistency level, the model can generate variations of the same character that remain consistent with each other.

Additionally, the transcript introduces another technique that utilizes in-painting to create consistent characters. By taking an initial image of the character and manipulating it using in-painting tools, you can create multiple variations of the same character in different scenarios. This technique allows you to cut out specific parts of the image and replace them with new backgrounds or outfits, resulting in a consistent character that can be used in various situations.

One of the advantages of using the in-painting technique is that you can generate multiple images of the same character, each with different expressions, poses, or settings. This can be particularly useful when mapping out different scenarios that your character needs to be in. For example, if you're writing a detective novel, you can create images of the character in various crime scenes or interrogation rooms. By having a range of consistent character images, you can effectively illustrate the character's development and experiences throughout the story.

Furthermore, the transcript suggests that you can use the generated character images as training data for your own stable diffusion model. By refining the generated images and iteratively improving upon them, you can create a more customized and consistent character. This can be especially valuable if you want to have more control over the generated variations and ensure that they align with your creative vision.

It's worth noting that while the techniques mentioned in the transcript primarily rely on AI models and tools, there are other traditional methods for developing consistent characters. These include carefully crafting character profiles, defining their personalities, motivations, and backstories, and ensuring that their actions and dialogue remain consistent with their established traits. These traditional methods can work in conjunction with AI-generated images to create well

Consistent Characters

 Okay, I'm going to show you how to do consistent characters in mid Jenny. So this is an example of a character I wanted to replicate. Like I've got a picture of her and the party just generated this, using this prompt. But say for example, I was writing a story about. She was one of the main characters I'd want to show her in different situations, maybe in the dentist office or at a birthday party. 

Whatever it is. And if we use the same prompt again, we're just going to get different results. And see all of these like very clearly different people, even though they're quite similar. So how do you keep them consistent? So mid Jenny actually added one way to do this. 

So if you click a copy of the link and then let's put in the same prompt again. Actually, yeah, that'd be grab the prompt.  

Just say 

At a dentist office.  

All right. A.  

Okay. So that is a new prompt. And obviously we just run that. We're going to get like a very different person. But no, to keep it similar, just going to click copy link.  And then the parameter you need to add here. Is , 



C dot E F character reference and then paste in the URL. Then you want to add dash. 

You want to say C w. And let's just say. It's, this is from zero to a hundred and so we want to keep it at that's how consistent you want it to be? Let's keep it like 90 on it to be very consistent. All right. Now let's pace that and see. How it works.  And what a man Jenny should do here is keep that character relatively consistent. Over time. 

So that would be really helpful.  

But obviously, while that's running, I'm going to tell you about a different technique that I use. That doesn't rely on this proprietary feature. It uses in painting instead. So I'm going to. And to show you that in a second. Now let me see. Is this look like the same person is. Pretty similar. 

So this is her.  

And yeah, you can see a condo looks the same. So a little bit wrong though. I think to be honest and look, she's wearing the same sequent dress. I didn't want it to be wearing that. I want it to be. In a dentist office. So we'll see if we can address that.  All right. How do I do it? 

What I do, my trick is I want to pan out. I'm going to click pan.  Submit.  And that's going to add some extra bits to the right-hand side.  And we're going to do that twice in order to get a two to one aspect ratio.  

And you can see here that we've added some extra background side.  

And then we need to upscale one of them. Does really matter which one.  And the moon to pan that, again.  

When you get here is basically. Enough space for two of the same character.  

Okay, let me just upscale one of these.  I'm just doing the one with the least amount of noise and the. Background. Okay. So now we have our character. We have, a consistent background on this side. All right. What we can do now is we can do a very region.  



And this is where the magic happens. 

So you can do this actually with any AI tool that offers in painting. So we're just going to select.  This sort of. Basically cutting out this part of the image. So the right-hand side of the image. And instead of. Doing one image with a background on the right hand side, are we going to change it? 

So we're going to say. Sorry. Two images.  Is side by side. The same.  



And you say At an 1975. Priority.  

And then we're going to say.  

This.  Hey. Your sweater.  

Alright.  

And then we're going to get rid of this stuff. We don't need that.  All right. And we've got the aspect ratio two to one.  So let's see if that works and this is not always consistent by the way. Sometimes it does go a little bit crazy, but that should give us the best chance of success. So how does this work? When you're in painting you're basically erasing part of the image and then prompting it for what you want. They image new part of the image to be well. The. Here we go. 

So what we're doing is we're taking instead of this, a wide angle, A shot. We're cutting it in half, or we're saying it's just two pictures of the same. Person right.  

And you can see here that, okay. It didn't get the dentist office saying properly, but it is the same person, right? It's very consistent, which is good. And she is wearing a blue sweater. So that is pretty good. And now all of these have our eyes closed. So one of the things I'm going to try now is I can go back to a very region. Did I say one it's still cut out. You're saying let's just try. The. She's Chinese five for potty left. At wearing the blue sweat 

Staring straight at the camera.  

Alright.  

All right, the, see if that fixes the problem, you're back into any normal kind of prompting techniques that you're going to use to do this. But the key is that. We're in painting. So there's two. Pictures of the same character now. The reason why this is useful is now we can basically cut out. If we drop this into Figma or Photoshop or whatever, We can cut out the different images that we want. 

So we can do this a bunch of times and then get Maybe 10 20 images that the same woman. And there's a few different things. We can do that, but one is we can. Map out all the different scenarios that we need her to be in. But the more interesting thing is that you could then use that as training data to train your own stable diffusion model on this specific character. 

And that's something I've done. In the past as well. You can see that if some of these are better than others you might take a bit of an iterative process here. Like maybe you need to get her with her eyes opened first, keep the sequined dress and then try and work on. Maybe a different background and they'll try and work on a different outfit. And it is a bit of an issue that you're active process, but the really cool thing here is that now you have a method for getting to different. Perspectives or two different behaviors or situations from the same character. 

And that character is consistent. And this works really well. Now the other thing actually I've found.  In general is that you can't do this with uploaded images. There is a little bit of a hack. They have this blend functionality. And if you blend two images together, they're similar. 

You do get a generated version in the same style. If you did need to hack it, I don't know how much longer that. But if he didn't need to hack it and wanted to get a character that you didn't generate that as like an upload character, then you could do that. But for now they've disabled. The ability to in paint, uploaded images. But yeah, hopefully that's helpful for you guys. 

And it gives you a lot of different opportunities to create more interesting character development in your stories.

What is the purpose of using the "C dot E F character reference" parameter in the tutorial?

What technique does the tutorial suggest using instead of the proprietary feature to achieve consistent characters?

How does in painting work in maintaining consistency in characters?

How can the generated images be further utilized in character development?

What is a limitation of the in painting technique mentioned in the tutorial?

There is a random element to diffusion models, so you get a different result every time you run the same prompt, making it hard to keep characters in a story consistent.

Every time I prompt it generates a different person

Ultra’s Offices – Quarterly Strategy Meeting

The hot topic being discussed is SEO: your blog content is good quality but not yet ranking on search engines. How do you build authority and start ranking on the first page of Google?

I think the quality of our blog content is great
But right now nobody is seeing it
Of course we all know SEO takes time
However if we don’t show tangible progress soon we won’t be able to justify the level of investment we’re at right now
The key issue has to be links
In my last company none of our content ranked until we got over 2,000 backlinks from a post that went viral
The rising tide lifts all boats…
We went from 200 to 20,000 visits a week
Of course we can’t expect to just go viral here with our skincare brand, that’s a bad strategy
But we can start building links
Let’s start with identifying broken links on competitor’s sites and redirecting them to us
Should be an easy place to start

In order to rank on Google you need authority. Google’s PageRank algorithm is somewhat of a popularity contest: if websites who have a lot of authority link to your website, they bestow authority on you, and your ranking increases. Of course the links still have to be relevant to the content that you produce. They also have to be legitimate: if our link profile looks unnatural like we’re buying links we can get penalized and de-indexed.
So we need to build authority in a legitimate way, ideally without spamming a bunch of people, or taking up a lot of time with websites that aren’t relevant. The cleanest, simplest and most useful way to do this, is to identify broken links to competitor websites. This occurs when other websites have linked to your competitor but the link is no longer working, it’s leading to a 404 page. This is an opportunity to contact the author and let them know, in the hopes that they’ll reciprocate and link to you as well, perhaps even replacing your competitor’s link with a link to your product instead.

Broken Links

Hey, let's do some broken link building. Uh, so we're going to use <inaudible>, uh, for this analysis. Uh, you can really do this in most of the SEO tools and potentially even, um, you know, use social operators to do this in Google.
Uh, but, um, I'm just going to show you a trust, cause that's what I use. Uh, I think you can pay $10 or something for a trial and, and do kind of the analysis that you need and then cancel.
Uh, so, uh, we are doing this for a skincare brand called ultra start-up and they compete with, uh, this competitive set here.
Uh, so, uh, they've done some surveys of the potential customers and they're finding that a lot of people use, uh, Harry's skincare, um, uh, solutions or they use, uh, Jack Black, which is the market leader in the U S um, and then they also use melamine guts, uh, because this is a very trendy skincare, which say usually encounter when they travel, uh, to different hotels.
Um, so, uh, we're going to go through these and figure out, uh, whether they have broken links, uh, because anyone that's looking to them, uh, could potentially be looking to us as well, especially if we notify them that one of their links is broken.
Cool. So, um, let's go to Harry's. Um, this is for the U S market. So you're going to copy your out, um, wherein within <inaudible> in the site Explorer, just going to put that in there and then hit enter.
Uh, so this is basically bringing up all the data on Harry's. You can see that they have a very high, um, you know, relevance in terms of, uh, domain, like they have very high authority.
Um, but, uh, the main thing we want to go to here and nature estimates is pretty simple to get to best by links.
Um, this is one of the best, uh, pages you could look at. It kind of shows you what pages have the most backlinks to them.
Uh, but, uh, the important thing for this link-building strategy is we can filter by HTTP code here. So we can basically say, um, four for not found.
Uh, and that's going to tell us all the different pages within Harry's that have a lot of links that, uh, haven't been, uh, found.
Um, so, um, we can, uh, you know, potentially search within the results of we're looking for something specific. We can export it into Excel and do some filtering.
Uh, but, um, I just like to kind of rank it based on, um, the Euro rating, cause it's a good proxy for like the most important URLs.
And then I just scroll down until, uh, there's not that many links coming through. So, uh, here they have a, you know, the social mission, which I guess say, give up on, uh, it's not found on the website anymore, our story.
So some of these are like, uh, mistakes. You can see, this is like the France page. It's got France, France.
Uh, so maybe there's some errors and that's why that's for a forum. Uh, but here's something interesting. Uh, they did a post for, um, I guess for, uh, pride week.
Um, and I, it's not showing up on the website anymore. So if we go to the way back machine, internet archive, this is sometimes a store, uh, different pages on the internet.
So maybe we get lucky and kind of see it hit. Okay. So it looks okay how one record and, uh, we can sometimes see what the website is.
So although it doesn't always load all of the, uh, all the images, uh, so let's just see what this page was just going to wait for that to load.
Hmm. It's taken awhile. Uh, it doesn't look like this living anything. Okay. Well, we can get kind of guessed what it is shave with pride set.
Um, we can see if there's another page on the left side That talks about this. Here you go. So they do have like a shave with pride set.
Now this is specific product, but I guess this was a blood post about this product and it's a pretty visually appealing one looks pretty cool.
Uh, so, um, so now we have an opportunity, I think because, um, we have here, um, there's 40 referring domains, uh, to this page that are going to a broken link.
So that's a user experience issue for the people who read about this. Um, there's also, uh, you know, from those 40 demands is 62 different links.
Um, so I like to kind of click through here and just see what they are and what you're seeing here.
Um, this is the referring page, the article that referred to them. Um, and then this is the anchor texts, like the texts that was highlighted and the URL, which is the shared with pride set.
So, um, we can see that this is about awesome packaging designs. That's kind of interesting. Um, this is the corporatization of pride month.
Maybe we don't know the year in here. Uh, that's kind of a negative article, um, you know, but like we would want to appear in packaging design reviews.
That's pretty cool, especially if we have a good package, uh, in terms of, uh, the, um, you know, the, the, uh, uh, the, the skincare products that we, that we created.
Um, there's also here 15 awesome limited edition product collaborations. Um, and then, uh, some article hit. Um, so there's some opportunities here we can go through.
Um, and, um, you know, we can contact these people. So if we click through to here and see the article, um, we can see, you know, we search the page for Harry's.
Um, we can see what they're read about. Um, and it looks like they're kind of listing different ones. So we have Harry's who gives us scrap.
So, um, this might be the sort of article that we want to get included into. And, um, if we, um, if we have a package that, um, in this case, uh, you know, fits the bill, we can contact the author of this post.
And so this is massly, um, Alrighty. And, um, you can usually click through and kind of, you know, find that Twitter or, um, you know, the, your, uh, their email.
Um, so, you know, you can just search for, uh, dialogue. Um, so that's like a little bit more about, uh, this is her personal website.
Uh, so she's on my crack as a PR kind of, uh, platform. So, um, yeah, ZoomInfo has a contact details.
It looks like. Uh, so, uh, yeah, that's, um, that's interesting, right? That's how you kind of go through, um, and then you can contact her and say, Hey, I noticed that, uh, the page you're linking to, um, isn't, isn't working anymore.
Uh, but, um, you know, I think actually, you know, we, we have a new package, uh, that you might want to check out that we might want to include.
Um, let me know if you want me to send you, um, you know, some of the products for free, just so you can check it out and, uh, you know, give, give it to a friend and see what they think, because it's a men's product in this case.
Uh, so, um, you know, that's a really good way to build that relationship, even if they don't link to you directly here, I, they might, you know, tweet about you or, um, you know, they might, uh, include you in a different article that they're working on.
Um, and, and you can't predict any of this. You also can't like specifically ask them, like, uh, pay them to do this, uh, because that goes against Google's guidelines.
Um, you know, but, uh, I think, you know, this is an opportunity to find someone who is looking to a competitor that is running about the space, um, that, um, you know, potentially will be influential and willing to, uh, to link through to you, uh, in the article or in a few trout skills.
So that's really what we're trying to find. Okay. So that's, that's one opportunity, uh, let's move on from Harris and see if we have anything for Jack Black.
This is the school, this is the leader here. So this is telling those that total, um, backlinks is good to hear best buy links again, um, and then filter it down to four, four.
Okay. So no, uh, for a force for their websites. So this is pretty good for them, uh, bad for us.
I mean, so they, uh, you know, whoever is managing our website, uh, you know, it hasn't made some of these messy changes.
Um, so they haven't taken off any of the pages in their site. So that's good. I would also look internal.
Um, it looks like none of the internal links are for, for either. Um, so that's, that's a, you know, that that's going to happen like, uh, relatively frequently.
Um, there, isn't always an opportunity to kind of have to search for the next one we're going to look at is not an, a goats.
Uh, this is the really trendy one, and we're on best by links here. So let's filter it down for, for, for, and here we go.
There's a lot of two TCS, so they have this face, grapefruit face cleaner. Um, so, uh, that has, uh, some, uh, it looks like they've changed some of these, uh, the URL structures here.
So, um, you know, him, men's journal fashionista.com, Jacob, Jacob, and Sebastian. Um, so it looks like there's a lot of these companies that have, um, you know, looking through to this and to hear that you're washing your face or wrong, et cetera.
Now in here we go. So they just have a few links to the bottom, so we can go to them and say, Hey, by the way, that link is broken and also, you know, did you want to look at our product, um, and, and see, see if you like it, if we have a face cleanser and then, um, and then maybe they'll add us as the fourth one here.
Uh, so I think that's, uh, that's like pretty straightforward. This is like ideal. This is exactly what we wanted. This is a lot less complicated than the, uh, than the pride week, uh, opportunity.
Um, the, uh, the other thing I think is useful to do is just see if they actually have that product elsewhere on the site.
Maybe they just change the URL, I guess. So they do, um, this is, this is useful to know just because, um, if they, if that product still exists, uh, then it's, um, I think like good practice to give them the updated link, uh, to the, uh, you know, remember, like you're not trying to do this at scale you're, should we get good quality links from influential people and build that relationship?
Um, so if you think about it that way, it's worth the extra effort to say, Hey, here's the correct link for that grapefruit face cleaner, because he now doesn't have face in the Euro, whereas before it did have face, um, and now it has the different fluid ounces in hairs off.
So, uh, that's, that's very useful. Um, I think that the, um, if, if the, this product doesn't exist, uh, then that's an opportunity for you to change the messaging and say, Hey, they didn't actually sell that anymore, but we think we have a good replacement.
So think about that. Uh, it's, it's, you know, trying to spot these opportunities. Um, but it's not really about scale.
It's more about quality. We're trying to find, um, you know, relationships with people who are LinkedIn to similar content and therefore, um, you know, this is just an opportunity to reach out.
Um, it's not like the end of the relationship. Um, so try and invest as much as you can into each one.


What part of Ahrefs do you need to go to in order to find broken links from a competitor?

What are the benefits of broken link building?

How would you pitch a broken link backlink opportunity?

How do we build authority

The difference between a website that gets 200 visits a month and 20,000 is backlinks. It’s night and day: if you don’t have authority on a relevant topic, Google won’t rank you anywhere near the first page.

GoolyBib Headquarters – Website Planning Meeting

Your CEO just read news online that Google was going to deprecate Optimize, their A/B testing tool, and is worried: his Q2 plans depend on that tool!

Even if the rumors aren’t true, they’re spreading because there’s something to it
Google hasn’t updated Optimize in years
They won’t offer a free tool forever
So let’s use this as an excuse to migrate off
What can we do to execute our testing plan if 


As the internet marketing world changes, so do the tools and solutions used to optimize websites and campaigns. One of the biggest changes in coming years might be a move away from Google Optimize. Google Optimize has long been the go-to solution for setting up A/B tests and other experiments, and is used on 500,000+ websites. But with Google making changes and discontinuing the service, many are left wondering what to do. 

While Google Optimize might be going away, you can still set up A/B tests and experiments with the same degree of accuracy and ease in another free Google tool - using Google Tag Manager. Google Tag Manager (GTM) is another free service from Google that allows you to add, update and manage tags on your website. It’s a powerful tool that can be used for a variety of tasks, including setting up A/B tests and experiments. 

So how do you set up an A/B test in Google Tag Manager? Let’s break it down. 

The first step is to create a new tag in GTM. You can do this by clicking the ‘New Tag’ button in the top right corner of the screen. You’ll then be taken to the tag creation page where you can choose the type of tag you want to create. For an A/B test, you’ll want to select the ‘Custom HTML’ tag type.

Once you’ve selected the tag type, you’ll need to enter the code for your A/B test. This is where you’ll use JavaScript to randomize visitors, and then apply filters to conditionally show each version. For example, you could use JavaScript to randomly assign visitors to either the ‘A’ or ‘B’ version of the page. 

You can also use GTM’s built-in variables to create the filters for each version. For example, you could use the ‘Page Path’ variable to filter for visitors on a specific page. Or you could use the ‘Referrer’ variable to filter for visitors from particular sources.

Once you’ve finished creating your A/B test tag, you can then set up triggers to determine when the tag should fire. GTM has a variety of built-in triggers, such as page view, click, form submission, etc. You can also create your own custom triggers if you want to be more specific. 

Once the tag is created and the triggers are set, you can then click the ‘Submit’ button to save your changes. After this, your A/B test will be up and running. The only thing you won’t have is the custom user interface that let you click and edit to make changes to the page, however that was only useful for simple tests anyway.

Test Implementation

0:01 All right. How to ab test when you can't use Google Optimize. So this rumors right now that Google Optimize is going away.
0:09 And that sucks because one of the cool things about it was it was free, easy to use and it let you do interesting stuff.
0:16 The webpages, without knowing how to code luckily you only need to know a very little amount of code in order to replicate it.
0:24 So this is maybe even a good idea to do in general. I found with these ab testing tools, there's like a flicker.
0:32 So when you land on the page, it's like not fast enough. Cause that's to load all of its crazy library, and then it makes the changes.
0:38 So I like to do this anyway. So say you wanted to change this, right? So one of the cool things you could do in Google Optimize and then optimize them for it.
0:49 So you could like click on it in the tool and then change it to whatever you want. You know, growth hackers, whatever, and it would change.
0:58 And then you'd go, great, this is what I want to test. And you'd hit the button. Well, you know, that's not possible anymore.
1:04 And obviously if you do this in the browser and you refresh the page then you know nothing's gonna happen because it's just gonna reload.
1:13 But you can still kind of plan your tests out this way. So if you say you wanted to change the button color you inspect then you can see that this has an idea of intent indicator.
1:26 You can actually click on it and say copy selector, that's useful. And then you can, you know, again, you only need to know a little bit of code if you just type in document and then call it like query sector.
1:47 And then piece that in. Then you, oh, did I do something wrong? Oh, query. Oh yeah, you didn't need this section.
2:00 Oh, here you go. Then you get your button. And you know, here it is. So if you want to kind of change that button, you can actually just make changes directly here.
2:10 So you can say like, what's this, you know, style you background color, color to be like green. Why not? It looks terrible, but there you go.
2:26 If that's what you wanted to do you could also change it and say, I want the text to be free access.
2:34 And maybe that's the ab test that you, you know, your heart desires you wanna run. So stupid example, but like, that's what you want to do, right?
2:42 So how do we actually make this persist? Well, the good thing is we have Google Tag Manager and Google Tag Manager can help us make changes in JavaScripts on our website without needing to deploy anything to our website.
2:55 We don't need to bother developers to do this, and we probably should tell them, right? It's not gonna bother them to tell them that you're changing something.
3:03 It'll bother them a lot actually, if you change things and you don't tell them. So just be careful here. Cool.
3:10 So how do we make this actually run on the website? Well, we can go to tags and we can set up you know some code.
3:22 So you know, you can actually don't even need all this stuff, right? Here's some code I made earlier. You know I, when I'm testing the stuff, I, I like to just go have an alert, which is like, it's loaded, right?
3:37 So you can just see that whether it worked or not pretty easily. So here's the tag, right? And don't worry about the triggers and stuff right now.
3:47 But if, if you wanted to, you can just like save this and then when you hit preview, you go through to your website.
3:56 If you load that page, then when that loads should say, yeah, look, it's loaded, right? And then you know, we've made this change here, the green button.
4:09 Cool. So that's not an AB test, so that's just making a change. But how do we make an AB test?
4:14 So if you look in here, I have the version B, right? What does that mean? So I'm only triggering it based on the version.
4:23 So if we look at this, it says on page views, but not all page views, just some page views. And the only one I find on the version when the page views the version matches, this is using regular expressions.
4:35 Don't worry too much about that slot. And then either slot, 6, 7, 8, 9, 10. So if you wanted to do this, you could do something similar.
4:45 You set this up. Now how does it know what this version is, right? Where does that come from? Well that is is just coming from if we go to variables and if we go down to custom variables we made a little custom variable for this.
5:11 So let's just look at here, yeah, version. And this is the code that we use to do this. So we have to randomize the visitors for an AB test, and that's what this part is doing.
5:25 It's giving you a random number times 10 slots. So that's gonna give us, you know, slots zero through nine. And that will assign a random number on every page, you to everyone.
5:38 But then we're not gonna get the same number again. So if I visit another page or if I refresh page, I'm gonna get something different.
5:44 So how do I persist it? Well, that's what the local storage thing is. What that's doing is, is first looking in local storage, which is just in the browser, you know, is there a version for this person and it's you know, gonna get the version and then pass that along.
6:01 And if there isn't a version, then it's gonna create one and then store it in local storage for me. So if we go back to this page, we right click inspect and then we go through the application.
6:15 We can see we are in slot nine. Now what if I change this to slot two? Lemme just refresh the page.
6:24 Ah, not in the test. Cool, let's inspect again and let's change it back to slot nine and then refresh, and then refresh.
6:43 Mess up. Slot nine. Yeah, I think I broke it. Oh, yeah, here we go. <laugh>. So it opened up a different page.
7:00 Yeah. So now I am in the experiment, right? Cool. So that should give you a really good idea of how this works.
7:09 It just randomizes the users, stores it in local storage, and then if they're in slots, 6, 7, 8, 9, 10 then you know, you can show them a version.
7:20 And and that's really all you have to do. You can obviously run as many experience as you want, like you could you know instead of having this trigger for version B, right?
7:32 You can actually have you know, a multiple triggers. You can have version A, B, C, D, E, F, G up to 10, you know and and, and then you could run multiple different experiments or like multiple different versions.
7:46 It's really up to you. But hopefully this gives you a good mental model for how that works. The other thing to think about is in terms of tracking you probably want to add, like, you know, add something to your, like if you're doing GA for I am you know, then you probably want to add something.
8:12 I think, oh yeah, no, it's not here. It's user properties, right? And user property. I'm just gonna make this version oh, yeah, sorry, I got this wrong to, oh, what did I do?
8:32 That should be version, sorry. So that's gonna input the value of the version for that user. And then I'm just gonna call this version.
8:44 So it's a property name. And if you do that now when you're in Google Analytics four, you be able to filter down to those users based on their version and then report On it afterwards.
8:56 So you know, this is a simple way to do that in any analytics package. So you know, there's always like a user level property you can set.
9:05 Cool. And that's how you reporting your OB test now. So you can just filter down and say like, people who got to that page and then what version were they in, and then whether they did, you know, any conversions afterwards.
9:16 Cool. Hopefully that was helpful.

Multiple Choice: What does the tutorial describe as a free alternative to Google Optimize for AB testing? 

How can you track the results of an AB test in Google Analytics?

What does the tutorial suggest you do to make sure the changes persist?

What is the purpose of using a random number times 10 slots in the tutorial?

What code do we need in order to make changes to the button?

I’m hearing rumors that Google Optimize is being deprecated (classic Google). If that happens it looks like over half a million websites are going to be without a solution that doesn’t cost $$$ enterprise money. 

Can we run tests without Google Optimize?

We're in the late evening at GoolyBib Headquarters, and our CEO, Gustav Gieger, has tasked us with finding a way to use Inpainting to edit and manipulate images using Dolly, Stable Fusion, and Mid-Journey. It's a tall order, but we're confident that with the right parameters and control net, we can segment anything we need to get the desired results.

We need to find a way to use Inpainting that allows us to make complex edits quickly and efficiently. 
Let's figure out how we can leverage control nets and parameters to segment images and get the results we want.
We need to start by testing different strategies and hypotheses.
Let's come up with a plan that will help us find the right combination quickly, without leaving a potential winner on the table.

In painting is a technique used to remove or replace unwanted elements in an image. It is a powerful tool used in digital art and photography to create stunning visuals. In this blog post, we'll be exploring the basics of inpainting, the different types of inpainting, and some advanced techniques to help you get the most out of your inpainting projects.

Inpainting is a process whereby unwanted elements in an image are removed or replaced. This is usually done by “erasing” the unwanted elements and then “filling in” the deleted area with new content. This content can be anything from a new image or texture to a different color or pattern. Inpainting is commonly used in digital art and photography to enhance an image, correct errors, or to create a totally new look.

There are two basic types of inpainting: image inpainting and pattern inpainting. Image inpainting is used to remove or replace unwanted elements in an image, whereas pattern inpainting is used to create unique patterns or textures. Image inpainting is typically done using software such as Adobe Photoshop or GIMP, while pattern inpainting is done using specialized software.

When it comes to inpainting, there are a few advanced techniques that can help you get the most out of your project. For example, you can use control net inpainting for more precise results. This technique involves using a single click to define the area to be inpainted, which then creates a mask that can be used to more accurately inpaint the desired area. You can also use segment anything inpainting, which allows you to quickly identify the area to be inpainted. This technique is especially useful if you are inpainting an object or person in a photo.

Finally, you can use the latest inpainting techniques such as stable fusion and mid-journey. These powerful tools utilize AI to generate highly realistic results. They also allow you to inpaint both inside and outside the frame, which can be useful when adding new elements to an image.

In conclusion, inpainting is a powerful technique that can be used to enhance, correct, or create totally new images. There are two basic types of inpainting — image inpainting and pattern inpainting — and a variety of advanced techniques, such as control net inpainting and segment anything inpainting

Advanced Inpainting

All right. We're going to talk about in painting. This is a Dolly to. And just going to upload an image. It's probably the easiest way to in paint. So it would just take her here's a. 

An image that I. 

Have uploaded. I'm going to click edit. And this is from the great Gatsby. And maybe instead of. This. A specific dress. Maybe we want different dresses search. To roughly erase. These parts here. And you don't have to be really strict and exact because AI is magic. 

And it does a lot of the work for you, which is nice. But yeah, here we go. So I have. 

Erased a lot of the image. And then and this is all within the generation frame. And she's going to put in here. A. Let's say a. I see. 





As you. 

Really. Address. 



Let's see. Great. 

All right. You can see the tip here is telling you. Once you to describe the whole image, not just like containing a Flamingo if we just put, containing your Flamingo as the prompt, that would not work very well. But if you put the whole image description and also what you need in here, then. 

That works much better as. And that's what I've done here. So I said, as you're in yellow, swirly flapper dress. Great Gatsby. And then just going to hit, generate. And it's going to basically just generate AI within. This edited out version here. And this can be done. Not just inside the frame, but also outside the frame, they call it out painting. 

We add another. Section two. And I'll show you that in a second, but it's basically the same mechanism. It's really just creating. Space creating white space and then generating within one of the things in painting does is. It takes the whole rest of the image into account, which is really helpful. 

And and then it also gives you multiple variations. This is the original image. And then here we go. We've added address this here and that here. And that's Shannon makes it look naked. That doesn't look very good. But it's not done a very good job. I think generally, because Dolly is not that good. 

Of an image generation model compared to the state of the art, like stable diffusion and mid journey. Mid journey, unfortunately doesn't allow him painting doesn't or it doesn't support it and do it right now. What I tend to use is several to fusion and I have this running. If you if you're going to use stable diffusion or use it through the several diffusion, a one, one. 

One in painting model. So I just have this running. If you need to install it, we have another course on that. The stable division web UI. And it has instructions for how to install but once you get it running, you can actually just use a local models. So this is a specific in painting model here. 

And we're just going to put in. I'm going to go to image. And upon in this image It's fun. And to do the same thing. So let me go in paint. Sorry. And then put that image. 

Here we go. 

And then you start drawing. 

And it's just going to get rid of. These parts of the dress. 

Okay. And then we'll just get rid of this. So the address of here. So you can see it's a very similar interface, except this is all open source and free, which is nice. So you can make a lot of mistakes if you need to. All right. So that is that. And then we're just going to put this same prompt in here and see how it does. 

So I would say, keep it simple at first and changing the parameters. Just see how it does, and then we can make changes to their practice based on whether the image completes well or not on that first go. 

Okay. So this is generating. 

And this is just generating one by default, but you could have it generated more here. If you like. Okay. So I think this actually already better than what we got from Dali, which is nice. So we haven't really done that much prompt engineering. The there's a few different things you can do. So one is we're not really getting much 

From this dress. So I'm just going to press command up. And this is going to basically emphasize it a little bit. I do that. 1.2 waiting. You can just do this manually as well by putting in brackets and putting a colon. So that's I think we're going to do that. Oh, sorry. I just missed out this pie here as well. Sorry. 

Let's improve that. And then we can also choose a few different things here. The mask blurs, how much we blur the image that mask mode. We want to him. We want to implant the things that are mast, not but if we were trying to generate the rest and keep the same dress, then we'd click and pay Namaste. 

The mass content parameters here. This does is you could either fill a witch. Generates, basically it fills in this gap with the same colors from nurse to image. So that's gives it a different starting point. Also original. So it starts filling in with the original image. And then have late noise. 

Which basically fills it with a lot of image noise. And that kind of makes it this is really useful if you want it to be very different from. What we had previously. The other thing you can do here is you can say, I want to inpaint the whole area, or just like only the mast. 

Area as we can try only max. And then the CFG scale. I'm going to put that up a little bit to eight. This is how much it follows the prompt. Versus being creative itself and then denoising, so it's at 7.5. I'm going to put it up at 8.50 6.85 and see how that does. That's going to do a de noising is basically how much shit. 

If you have a high denoising. Then it's going to add a lot of noise to the image before it diffuses hours. If you have low denoising, it's not going to add very much so essentially if you want it to be very similar to the original. Then you would use original mass content. 

A whole pitcher and low denoising. But if you if you want it really. A different, like here, we're trying to input like a whole new, different types of dress. Then I would say We only use these sorts of settings and it is a crap shoot. So like this I think is too similar to the original dress and that's a problem. 

Cool. I'm going to generate, but I'm also going to generate and the material for. 

Four images. And this is going to do them one at a time. If I wanted to generate for all the same time, that sucks up a lot of V Ram. It depends on your computer, but it might be quite slow. So just going to. Do them in batches. So just do one, one per batch. That's going to run. You can see that running here. 

We're going to see how that does. There's one other thing we can try if this doesn't work as well, which is something called control net in painting. And you can see here by the way as well that I'm using a specific in painting model. If you want to get that. The, I can show you how to get that while we're waiting. 



All the, Everything you need to know about automatic 11, 11 is in here. Super useful. So installation. 



Oh, I'm on the wrong one. Okay. 

Just going to find the link to the most. Here we go to his there in painting model. You could just Google it as well, but yeah, this is a, it was released by runway ML there originally. Yeah, it helped create a stable diffusion. And you can get the model file in here, files and versions. 

And then you just go. In here it's like in painting. Checkpoint file C K P T. 

All right. So this is running. 

And I think it's almost done. 

All right. First thing it does is give us a grid. So at least it's a yellow dress, or it's just kinda nice. There's some blue as your blue. Swirls here. Addresses and much better. There we go. This is like so much better than what we go from Dolly. It's actually form fitting. 

I think that's really good. That's much better. And then, we could, we might want to just. And do a bit more prompt engineering care or whatever it is. We needed. This is swirly, and then we could say maybe 

Vortex. Whirls. Inspire heavy. Brush strokes. 

Bye. 

Sorry. 

All right. Let's try this one more time. So you can either save images here. Or you can click save, or you can actually send it back to him, painting and start again. But also it's in your file system. So if you go into your folder, by the way, this is where you save the new models. If you download them, like I have the painting one in here. 

It's in models, stabled a fusion in painting. And then if you want to see any of the images you output, if you go in here it's this would be under image, images. And then this is today. And then that's the latest batch of images here. 

If you ever need to get them. That's where you'd find them. So this is running. 

Just going to let that run and then I'll show you two. To other more advanced concepts within painting. One is control net and the other is segment anything. 

And those will be in a different different module. A different course. 

Here we go. This looks much better. 

So that's like much closer to what I wanted. We've got some prompt engineering going on here. That's cool. But yeah, you can keep messing with that. If you want, you can change to the sampling steps, et cetera. The other the other things you can do down here. Control net and painting this we're not going to go into control net right now, but it's just a way to make the image match much closer to the original image. 

It works exactly the same. You just upload an image here and then you want to choose the in paint model. And then run that. And it should perform better. That, that should be, keep it more visually consistent with the rest of the. With the rest of the image, if that's what it, what you're trying to get across. And then a segment, anything it's just a way. I'll just quickly show you this. Same with anything it's like just a quicker way of finding the thing you wanna in paint. So I just clicked once on once in the dress on each side, and then once down here and then just going to preview the segmentation, see if it works. And what it should do is it should just mask out only those parts. Yeah, here we go. Yeah, here we go. This mask is worked. So that is That's just the dress, right? So it's picked out just the dress here. Which is very helpful. So that could make it much more precise. When you're doing this. And you choose the favorite mask and then you'd run it. And it. would just in paint in this area. That's much easier than you doing it yourself, right? All right? Hopefully that's helpful. There's a lot more advanced stuff. You can get him here. You can go much deeper with it. And it is like an intuitive process that takes a lot of experimentation to get really right. So just keep trying and looking at the parameters, if you don't know what they mean.

What does the CFG Scale parameter affect?

How can we find the images we output from the tutorial?

What are the two more advanced concepts mentioned in the tutorial?

How can the image generated by the AI be saved?

One of the coolest demos of AI in recent years is Adobe Photoshop's generative fill, however it doesn't work in many situations and the underlying model is weak.

3 Virality Courses

Courses - It's time to grow👇

Learn the skills to be a Data-Driven Marketer