A short guide on how to create activation metrics for a product
In a previous articleI talked about the Input > Output > Result framework, and how “output” was the center piece, but not necessarily easy to define, just because you want your inputs to move it, but at the same time, you need to have a causal link to your result. .
User activation metrics belong to this category of metrics. “Activation” is the third stage of the hack metrics framework designed by Dave McClure (the famous AAARRR framework: Awareness, Acquisition, Activation, Retention, Referral, Revenue), and is typically defined as when your user passed the first set of frictions. . , you started using its product, received some value from it, and are now more likely to retain it for the long term.
Some examples of product activation metric:
Loom: Sharing a loom¹
Zappier: Setting a zap¹
Zoom: Completing a zoom meeting within 7d of signup¹
Slack: Sending 2,000+ team messages in the first 30 days²
Dropbox: Uploading 1 file in 1 folder on 1 device within 1 hour²
HubSpot: Using 5 features within 60 days²¹2022 product benchmark from Open View
https://openviewpartners.com/2022-product-benchmarks/
²Stage 2 Capital: the science of scaling:
https://www.stage2.capital/science-of-scaling
Measuring activation is important because it helps you understand how well your product is resonating with new users and whether you are actually getting them to become “active” users. It’s the first step towards user loyalty – this is the stage where you know if your users are likely to stick around for the long haul. If activation is low, it may indicate that there is an issue with the product or the onboarding process, and changes may be needed to improve the user experience and increase activation.
- You want Activation to be a good predictor of Retention, but at the same time, you want it to be simple enough that it should be an easy first step for your users to follow.
- Basically, you’re looking for the smallest action a user can take that shows the product’s value to them, but you want this small action to have a causal link to retention (however you define it).
- As with any ‘top’ metric, the causality piece (“taking action AND leads to long-term retention”) is tricky. You typically start with observational data, and traditional data analysis may not give you the full picture, as it can miss confounding factors that can affect activation/retention.
Using a cohort analysis, you can start to develop some intuition about which user actions might be good candidates for your activation metric.
The idea is:
- Group your users based on where they signed up for your product
- Separate them based on whether they made it to the retention stage or not.
- Look for actions that are overwhelmingly performed by users who made it to the retention stage, but less so by users who didn’t.
Let’s say you run a fitness app. You start to build a monthly cohort and notice that 70% of users who upload at least one workout within the first week of signing up are still engaging with the app a year later, versus 40% who don’t. This can be a first idea for an activation metric.
A prerequisite here is that you have an idea of what stock to study. In the example above, you had to come up with the idea to see who was tracking their workouts. This is where quantity meets quality, and where your ‘user perception’/common sense comes into play. Or your networking skills if you want to enlist the help of other subject matter experts.
Any advice:
- You might want to brainstorm some possible action ideas, not necessarily go through too many, just because, as the adage goes: “if you torture data long enough, it will confess to anything” (Ronald H. Coase). The more actions you select, the more likely you are to find something, but you run a high risk of it being a false positive. So sticking with what makes sense and isn’t too over the top can be a good rule of thumb.
- You might want to take a principled approach to this, and only look for things that you think you could move. If you come up with something too complicated/niche, you may not be able to move it, and this will defeat the purpose of the whole exercise.
With propensity score matching, you can confirm or invalidate your previous insights
Once you’ve identified your potential trigger signals, the next step is to make sure they’re accurate. That’s where propensity score matching can be useful: to understand if the correlation you found above might be causal. Although this is not the only solution out there, and it does require some knowledge about your users (which may not always be the case), it can be relatively easy to implement and can give you more confidence in your result (perhaps even more triangulation). , with more robust approaches like A/B testing).
The idea behind propensity score matching is as follows:
- To find the causal link between taking the action and retention, you would ideally clone the users who took the action and have the clone not take the action, to compare the result.
- Since it’s not possible (yet?), the next best thing is to look inside your data, find users who are very similar (almost identical) to the users who took the action: but who did not take action.
Propensity score matching is a methodology that allows you to find those very similar users and match them. Specifically speaking, it is about:
- Train a model to predict the probability that your users will perform the action you defined (its propensity).
- User matching based on previously found probability (the matching part)
(Note: You have different ways to do both steps, and there are great guidelines available online on how to select a model, how to select the correct variable, which matching algorithm to select, etc. – for more information see “Some practical guidelines for the implementation of propensity score matching”)
Taking our fitness app example again:
- It has found that 70% of users who upload at least one workout within the first week of signing up are still engaged with the app a year later, versus 40% who don’t.
- You train a model to predict the probability that your user will load a workout within a week of signing up, and find that the probability is very high for users who downloaded the app through a referral link from a large website. fitness.
- You rank your users based on probability and start doing a simple 1:1 match (the first users in terms of probability that they took the action are matched with the first users in terms of probability that they didn’t take the action, and etc.) .)
- After the match, you’ll see the difference shrink considerably, but it’s still important to consider as a potential candidate for a trigger metric!
Cohort analysis + propensity score matching can help you isolate the impact of a specific action on user behavior, which is essential for defining accurate trigger metrics.
But this methodology is not a panacea – there are a lot of hypotheses that come with the methodology, and you’ll need to tune it / have some validation to make sure it works for your use case.
In particular, the effectiveness of PSM will largely depend on how well it can predict self-selection. If you are missing key features and the bias for unobserved features is large, then the PSM estimates may be highly skewed and not really useful.
With all of this being said, using this methodology, even imperfectly, can help lead to a more data-driven approach to metric selection, getting you started with “what to focus on”, until you get to the bottom line. stage of executing A/B. evidence and gain a better understanding of what drives long-term success.
I hope you enjoyed reading this piece! Do you have any tips you’d like to share? Let everyone know in the comments section!
And if you want to read more from me, here are some other articles you might like:
