 Bank of Yu Offices – Monthly Product Meeting

After a successful launch and satisfying early results for user acquisition, your Head of Growth is turning his attention to retention. They want to figure out what the ‘aha’ moment is for your product, similar to Facebook’s famous insight that if they get a user to add 7 friends in 10 days they’re likely to stick around.

If we’re going to focus on retention, we need our ‘north star’ metric
You know, like how Facebook had “7 friends in 10 days”
The ‘aha moment’ where the user has completed enough actions to remain sticky
If we know what correlates with retention we have something to aim for
If we can focus on moving the needle on that metric, growth should follow
Can you do some analysis to find that ‘aha’ moment for us?

How many actions does it take before our users say ‘aha’? If they don’t use your product enough they might not get it, and if you don’t get them to that point in the first few weeks they might never come back. However if you can optimize your product relentlessly towards a ‘north star’ metric that’s correlated with retention, your product will get stickier, and growth will follow. Facebook’s growth team famously optimized to ‘7 friends in 10 days’. They knew that if they got a user to add at least 7 friends, they would get enough value out of the social network to stick around. The goal wasn’t just correlated with success but it was simple: there were thousands of things the team could work on, but with a north star metric in mind they could focus and prioritize only tasks that moved the needle on the metric. 

In order to choose an effective north star metric, you need to do a retention analysis. The ‘aha’ moment is in the intersection of two competing forces. The more actions someone takes the more likely they are to be retained, but also the more people will have been retained without taking that many actions. You need to find a tradeoff between the two. First you find out the share of people who were retained, that performed this many actions. Second you need to find out how many people are retained without performing this action. Bring those two together and you can find the sweet spot, whether it’s 6 friends, 7 friends, or 8 friends, or whatever the equivalent action in your product.

Retention Analysis

Hey, let's learn how to do aha moments, uh, with user retention data. Um, cool. So we're in a Google CoLab notebook.
It's like a Python notebook. Uh, you can think of it as a Google doc with code. Um, so it runs in the cloud and you don't need to set anything up on your computer, which is helpful.
You can share it like any Google doc. Um, so if you run a, if you just shift and enter, or if you could play on any of the, uh, the cells, and then it's going to run that code and output anything that's needed, um, I've already run this part here.
So this is the libraries, and then this is authentication. So we can pull data from the Google spreadsheet. Um, when you run it, uh, it's gonna pop up and ask you, uh, to, uh, kind of authenticate with your Google account.
So you just click on your email and if you're logged in, then, um, then it will connect it. Um, and that's really just to get this data here from the Google spreadsheet.
Um, and that is the data that we're going to use for this analysis. So this is the format you would need it in, uh, we need the user IDs, um, and then whether they were retained or not.
So in this case, uh, we use the date frame of one week, uh, if they're retained a week after they, uh, installed, uh, then we have a one and if there, if they weren't retained, then we have a zero.
Um, the, uh, this is, uh, one of the events that you could do within the app. It's a FinTech app that we can give loans to friends.
So we're going to evaluate the loans, uh, metric, but you could do this analysis for any of these other events that use this do as well.
Cool. So, um, if we run this, it's going to pull in the data, um, and we can see this just the first five rows of that data, just so we can check the, it pulled in correctly.
And if we run this query, uh, the data stored in this DFS data frame variable, um, and then we're just going to group that by how many people were retained.
Um, and then we can get the count of the user IDs. I, so this is essentially what we're going to do, but on a larger scale, um, but you can see here, they have 969 people that retained and then 2,726, that didn't cool.
So, uh, we're using this loans action. And if we run this here, uh, you can see it calculates a table with a zero for people who didn't retain one for people who did retain, um, and then how many loans they gave out.
So what we're looking for is how many loans does someone need to give out to make them very highly correlated with retaining within the app?
So we're trying to find, uh, something similar to Facebook's goal. Uh, they use for their growth team, which was, uh, they figured out if you can get a user to add seven friends in the first 10 days, they're very likely to retain, and we have a trade off to make as so first of all, um, you know, we wanted to find a users who are very likely to retain.
So, um, uh, so people, You know, people who did this action, whether it's say like, you know, five lines, uh, people that did this action were very likely to retain.
Uh, but also, um, that, uh, they, um, that there were a lot of people who did that action. So if loads of people were retained without getting that far without getting to full lens, it wouldn't be a very good measure.
You can see here that actually, um, you know, we only had 75 people who made full lens, so that might not be a good action.
So it, can it go, um, you know, zero loans wouldn't be a good action either because loads of people make zero loans and they're not likely to retain if they do.
Um, so, uh, in order to figure this out, we want to use data visualization. Um, so we're using a map plot, lib and seaborne, uh, to visualization libraries.
Um, and essentially what this is doing is just kind of setting the color codes and then, um, plotting, uh, two bar charts, one on top of each other.
So we can see here, um, that the retention rate, uh, which is, uh, what percentage of people who got to that level, um, retained, uh, is very, very high for people who made like four loans or above.
Um, it's in the middle for, uh, people made two or three lines. So it's like a 40% pretension, right? Um, and then for people who made zero one loans, it's much lower, it's less than 20%.
Um, so that kind of tells us what part of the story that the more loans you make, the more likely you are to retain, which is what we'd expect.
Um, and we can see a couple of reflection points here. It was, it was already telling us something. Uh, the next thing we want to look at though is just the raw numbers.
Uh, and that shows is another part of the story, which is that very few people make it to fall into or above.
Um, uh, so, uh, it wouldn't be an effective target, uh, to go that far, uh, in terms of our goal, because we would just wouldn't have enough data to optimize.
Um, it wouldn't be able to measure that much of an uplift on how many people got to four, uh, plus loans, uh, because, um, you know, very few people do so in the first week.
Cool. So, uh, let's, uh, run this, um, and, uh, what this is doing is working out a few more pieces of information that we're going to use in the next visualization, uh, specifically the share of retained as, so this is how many people who got to that, uh, level and retained, like what share of, of that, uh, um, what share of those people did they make up of all the people who retained meaning, um, you know, did most people who retain, uh, get to that level?
Uh, so that was one piece of what we're trying to look for. The other piece is, uh, how many people retained without that action.
So this is doing a cumulative sum of the retention metric. Um, and you can see here are retained without the action.
Um, so it's saying, um, you know, if we get to two loans, um, then, uh, yes, we had 283 people retain when they go to two lens, I, but 405 people retained without even making it to, to learn.
So they made zero one loan and you can see that 2, 4, 6 plus 1, 5 9 is four or five. Um, the reason we made this negative, I did, uh, you know, multiplied by minus one is just, sorry, could show it nicer in the visualization.
So it makes it on the left-hand side of the axes here. Um, so this visualizes, the problem that we're talking about perfectly.
So as we go down the stack, more people are likely to retain, uh, but, um, as we get down the stack, uh, most people who retained, uh, you know, had, had already been retained at that point.
Um, you know, uh, like if you look at number four, the pink bar here retained without the action is way, way bigger than even the people who made it to this level.
Um, uh, because, uh, you know, most of the people were were before, uh, for loans, they didn't make as many as full loans.
Cool. So, um, now how do we see this in percentages? Because that's gonna make it much easier to decide. Um, so we're just going to run this, but it's works.
Yeah, here we go. So this is, uh, you know, what we've been leading up to this is the money shot.
Um, and, uh, this is the visualization that you'd run around. You could skip a lot of this stuff. Um, you know, if, if you're just running this analysis yourself, um, you can actually, uh, I think here, just go run all.
Um, and then, uh, we'll run everything all the way down to this, if you want. Um, as you don't have to like worry too much about the previous stuff, uh, this is the most important thing.
So this is sharing everything really on, uh, that we are sharing here, but in percentage terms, so over 100% of people, um, you know, uh, the blue is tow people who made it to this level, um, the, uh, dark blue share of Chandra with purple.
Um, that's saying that say 20% of people who made it to one loan, uh, were retained. Um, and then the pink bar is, uh, people who, uh, the share of, um, of, uh, people who retained without taking this action.
So, um, for one loan, the pink bar is pretty low, uh, because, um, uh, you know, not many people who did zero loans, which is above this, uh, were retained, um, as, so that pink bar is pretty low, but then the pink bar girl is a lot just like it did here.
Um, the pink bar grows a lot to then being pretty much everyone by the time we get down to nine months.
Um, so what you're looking at here just to keep it really simple is you want, uh, you know, a huge share of retained.
In this case, we have a big purple buyer here around two loans, uh, relative to the pink bar. So we want like a bigger purple bar.
Then we have a pink bar, um, and we also want a bigger palpable, and we have the blue bar. So, uh, essentially like the bigger, the purple bar and the floor of the pink bar, uh, the, uh, better That is as a target.
And for me, it's pretty clear, like we could choose one loan has a pretty big purple bar and a pretty small pink bar.
Uh, but, uh, this learns, uh, you know, two learns has a much bigger pebble bar and not that much bigger of a peak bar.
So umbrella to temps, this is a better goal. Uh, we wouldn't choose learn three because even though it has a bigger purple Bob than learns one, um, the pink bar is much bigger as well.
By the time you get to three loans, most people have already been retained, uh, without getting to three loans. So this is how you do this analysis.
Um, you can go and run all of this stuff, uh, with another, uh, metric if you want. Um, so you just change it here.
Um, you know, if you want it to be a balance, uh, that you can press shift and search attend to oh yeah.
A lot of, a lot of you balances. Uh, so you could do, you know, like, it just depends on what data you have.
Um, you can, uh, run it with, um, you know, as many as you want, uh, in the, in this case, uh, view balance is probably a bad run to do because, uh, there's lots of these actions.
Um, so like, uh, comments, and maybe that would be a better visualization. All right, here we go. Yeah. So that looks better.
What's more interesting. Um, and you can see, uh, there's people who made like many more comments, but it's a similar effect, uh, to what we had, um, you know, with the other analysis and just to show you, you can click or journal, hopefully that will work, and it just runs everything in order.
Um, and then you just go down to the end. So, um, here you can see as a really big purple bar and a very small pink bar.
Um, so, uh, here, it might be like comment, anyone who makes one comment is very likely to be retained. Cool.
Okay. So, uh, hopefully that's useful. Um, it's, uh, can be a confusing once you try to understand, like, what the difference is between like share of retained and share retained without the action.
Uh, but, uh, just go through it again, if you don't understand and maybe play around with some of the, uh, data to, to, to see what we're talking about.


Why is it not sufficient to just look for the action that had the highest retention rate?

What are you looking for in terms of choosing the number of actions that would make a good north star metric when looking at the final chart?

Run the analysis but for “searches” instead of “loans”. Why wouldn’t this be an effective action to model retention?

What's our 'aha' moment?

Facebook’s growth team famously targeted ‘7 friends in 10 days’, as their north star metric on the path to 1 billion users. If they could get a new user to add enough friends they knew they’d be hooked, so everything they did was in aid of this goal. You can actually calculate the north star metric for your product, and it’s easier than you think.

Vexpower Learning Certificate of Completion

Congratulations Rhys Fisher 💪