The Heap Guide to Retention · 2019-12-18 · The Heap Guide to Retention heap.io 3 Introduction Use the “leaky funnel” analogy or say your product is a “boat with a hole in

1The Heap Guide to Retention • heap.io

A Smarter Approach to Product Analytics

THE HEAP GUIDE TO RETENTION

The Heap Guide to Retention


Introduction

Section 1: Why Retention Matters

The Business Case for Retention

Down the Funnel and Back

Good vs. Bad Retention Graphs

Three Stages of Retention

Section 2: Improving Retention

Defining Retention

Measuring Baseline

Critical: Avoid Measuring Incorrectly

Choose the Stage to Target

Scenario 1: Hypotheses for Encouraging New Users to Stick Around

3

4

5

6

7

8

9

9

10

11

12

14

15

16

18

18

20

22

23

24

25

26

27

Scenario 2: Hypotheses for Creating Ongoing, Repeatable Value

Scenario 3: Hypotheses for Creating Longer-Term Retention

Section 3: Examples from Heap

Finding Our Retention Metric

Two Retention Metrics

A Hypothesis Story: Suggested Reports

Suggested Reports: What We Learned

Suggested Reports: Impact on Retention

Suggested Reports: Group By

Suggested Reports: Final Learnings

Closing

Table of Contents


Introduction

Use the “leaky funnel” analogy or say your product is a “boat with a hole in it”: if your product can’t retain a solid base of users, your business can’t sustain revenue, and your company can’t grow.

Welcome to the Heap Guide to Retention. Our goal in writing it is to give Product Managers more opportunities to provide undeniable value for their customers. In particular, we believe that Product Managers can increase retention rates most effectively when they take a data-driven, experimental, iterative approach to product development, one that involves forming and testing hypotheses, figuring out what to measure, making small improvements, and learning from every single experiment, successful or not.

In what follows, we describe why retention should remain a critical focus for Product Managers, what hypothesis-driven PMs can do to measure and increase retention. We’ll also provide multiple examples from our own practice that suggests how PMs might go about developing and testing hypotheses that address retention in their own products. At the core is the idea that a PM’s job, when done right, involves using analytics to focus in on problem areas and then creatively designing ways to address them.

As you read, you’ll realize that many of our strategies are about pursuing smaller wins. At Heap, we believe that incremental improvements, when implemented constantly and repeatedly, pay enormous dividends. As long as you’re being creative with your ideas and both rigorous and methodical about testing them (including recognizing when hypotheses are wrong and having the humility and ambition to change directions as needed), you’re doing it right.

Read on to improve your product or feature!

INTRO

4The Heap Guide to Retention • heap.ioThe Heap Guide to Retention • heap.io 4

Why Retention Matters

Retention is the True Measure of Product Value

Building a product-driven business takes energy. There’s marketing to generate interest, and a sales apparatus to bring deals home. On the product side, there’s the initial user experience to develop, and the value your product should demonstrate from the get go.

These activities are all important for getting customers in the door. But once people start using your product, it’s your retention strategy that keeps them there. Plenty of users are willing to try a product once or twice. But if your product or feature doesn’t provide ongoing value over time, or is too difficult to use, or if the pains your product addresses can be more easily solved by a competitor, those users won’t stick around.

In fact, from a product perspective, it’s not unreasonable to say that retention is the best measure of product-market fit. For the bulk of digital products—websites, mobile apps, business platforms, and many more—achieving product-market fit means producing a product that users willingly return to.

It’s for these reasons that Product Managers should be somewhat obsessed with retention: measuring it, understanding it, developing their product or features with an eye towards improving it. Other growth metrics matter too, of course. Acquisition numbers tell you how well you’re able to sell your product to customers who haven’t used it yet. Activation numbers tell you how many people you can get into your product. Retention alone tells you if customers are finding ongoing value in your product.

For hypothesis-driven PMs, there

will rarely be a time when increasing

retention is not a major goal.

01 S E C T I O N O N E


You can see how increasing or decreasing retention by even a single percent has on your yearly revenue. Simply decreasing retention rates to 4% gives you an increase of 7 customers a year. These 7 customers translate to meaningful business value.

As a product manager, you partner with marketing and sales to entice customers to your product. But you also have special insight into what makes them stay.

The Business Case for Retention

Another way to think about retention is to measure business impact. Most of us know that it’s far more expensive to acquire a new customer than to retain a current one—anywhere from 5 to 25 times more expensive. From the business’ perspective, a poor retention strategy puts unnecessary pressure on your acquisition efforts—marketing and sales.

The diagram at left gives a quick explanation. To the question, “would you be happy with a monthly retention rate of 95%?” most product companies tend to answer yes. (When we ask this question in presentations, the majority of the audience raises its hand.)

Yet the numbers at the bottom of the chart display the compounding effect of a 5% churn rate. Simply losing 5% of your users a month means a yearly churn rate of 43%.

This 43% doesn’t only mean a loss of income (via diminished subscription costs, ad sales, renewals, or whatever business model your company supports). It also holds back growth, since you need to grow by 43% per year to simply maintain a constant number of users.

Impact of Monthly Churn Across a Year (# Users)

Jan

100 94 88 83 78 73 69 65 61 57 54

100 95 90 86 81 77 74 70 66 63 60 57

100 96 92 88 85 82 78 75 72 69 66 64

51

Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

100

90

80

70

60

50

Jan Dec

4%

5%

6%

https://hbr.org/2014/10/the-value-of-keeping-the-right-customers


Down the Funnel and Back

One can also think about retention as the gift that keeps on giving. Much of what happens further along the customer journey depends on how well you’re retaining your customers. Renewals, expansions, referrals, recommendations: all only happen when your customers continue to find value in your product.

Let’s think a little bigger. For each individual customer, retention is key for moving them down the path. For your company as a whole, however retention plays an outsized role on acquisition and activation. Why? Because retained customers give you proof points, case studies, and other documentable evidence that your

Key Metrics for Growth

product works. Showing a market that you can provide ongoing value to customers is the easiest way to get prospects excited about what you can do for them.

The effects extend outwards. Demonstrate stickiness and you’ll find partners more interested in working with you. Retain well-known customers and word spreads. Having a stable of satisfied customers to call on gives salespeople a critical tool for their arsenal. And so on.

In fact, we could say that for most product-based companies, retention is the primary objective for sustaining growth.

AcquisitionCustomerssigning up

ActivationFinding value

in your product

RetentionContinue to come

back over time

RevenueRenewals, up-sells,

cross-sells

ReferralCustomer advocates

on your behalf


Good vs. Bad Retention Graphs

At right you can see the difference between an effective and a non-effective retention strategy. The graph at top reflects a user base with a strong initial period of acquisition and activation, then a long-term flattening out, indicating high churn and (assuming the company is putting effort into bringing in new users) poor long-term retention.

If you’re at the early stages of this graph, your growth rates may be encouraging. But they can also be misleading, since they don’t tell you how many of your new customers are going to stay.

If you’re regularly measuring retention and are observing a graph that looks like this, it should be cause for alarm.

On the bottom you can see what a user graph looks like when a company has a good retention strategy in place. In this case the number of users continues to increase over time, ensuring that your company’s ability to retain users matches its ability to acquire them.

9,000

8,000

7,000

6,000

5,000

4,000

3,000

2,000

1,000

-

1/16 2/16 3/16 4/16 5/16 6/16 7/16 8/16 9/16 10/16 11/16 12/161/

14

3/14

5/14

7/14

9/14

11/1

4

1/15

3/15

5/15

7/15

9/15

11/1

5

1/16

3/16

5/16

7/16

9/16

11/1

6

1/17

3/17

5/17

7/17

9/17

11/1

7

120,000

100,000

80,000

60,000

40,000

20,000

-

Bad Retention Example

Good Retention Example


Three Stages of Retention

The graph at right—whose shape is typical for many companies—shows the standard way we measure retention. The initial number—62%—represents the number of acquired users who actually engage with your product. Typically this number drops significantly after a short period of time, here a month, and levels off into a more standard user rate. That number is generally what you’ll use as your baseline as you run experiments designed to increase retention rates.

This number can also be helpful for forecasting revenue. If you know that 16% of acquired users are likely to keep using your product and you have a decent idea of how long they’ll stick around, you can produce a useful forecasting model.

In this graph, you also see how user numbers slowly decrease over the course of the year, indicating slow but steady churn. The final significant drop occurs at the end of the year, when the large majority of users still in the product or using the feature drop off entirely.

As a Product Manager, it’s your job to know the retention rates at each of these stages—initial use, longer sustained period of time, and a larger drop off at the end—and to develop hypotheses around improving them.

Retention Rate

Stage 1 Stage 3Stage 2

Sam

em

onth

Aft

er 2

mon

ths

Aft

er 4

mon

ths

Aft

er 6

mon

ths

Aft

er 8

mon

ths

Aft

er 1

0m

onth

s

Aft

er 1

2m

onth

s

70%

60%

50%

40%

30%

20%

10%

0%

NOTE: Retention doesn’t have to be measured on the level of your entire product. It’s just as important to track retention of a specific feature. Doing this can help you decide whether it makes more sense to prioritize further development, to sunset, to refocus, to simply to do more investigation.


Improving Retention

Defining Retention

In order to develop and test hypotheses about increasing retention, it’s important to first measure your retention rates and assemble them in a graph like the one on the previous page.

Doing this can be tricky. At bottom, your retention metrics should measure how many users performed an event once, and then performed that event again later. The main difficulty is that it can take work to decide what event in your product counts as “usage.”

In Google Analytics, for example, retention is measured by raw visit numbers. If a user visits your site, then visits again later, they are counted as retained. The problem is

that the “visits” metric doesn’t tell you what actions users are taking in your product. For many products, this can be a problem.

For example, many gaming apps add “daily reward” games to the user experience, where users who visit the product on successive days win a prize. (Often the size of the prize increases the more days in a row a user visits the site.) But these kinds of visits rarely correlate with long-term retention.

02 S E C T I O N T W O


Simple visits metrics can similarly cloud your ability to assess the overall satisfaction. If an enterprise company purchases a CRM (for example), that product will get hundreds or thousands of daily visits. Measuring them alone can’t tell you if users are getting the value you want them to be getting from it.

For more focused product leaders, the question is less “how many people come to my site” than “what is the behavior that people take in my site that signifies value, and how many people repeat that behavior over time?”.

What this behavior is often depends on what your product aims to do. For e-commerce sites, the behavior might be “purchase.” For marketing sites, it might be “view the blog.” For subscription services, it might be “renew.” And so on.

The key is that when defining retention, you want to brainstorm with your team, and home in on the sorts of behaviors you want your product to drive long-term.

50%

30%

10%

1 YEAR 2 YEAR 3 YEAR

Percentage of Monthly Return UsersLength of time after customer registration

Evernote’s famous “smile graph.” The longer a customer used Evernote, the more likely they were to return. That’s stickiness.


— Group By (optional): Group by any user property to examine retention across cohorts or segments of users (this can include things like “job title,” “completed X event within a certain date range,” “referred by a specific page,” and many more).

— Date Range: Select the date range you want to look at. Unless you’re focusing on new user retention, you’ll likely want to focus on a longer time frame.

— Granularity: This determines the interval by which we view how often a person completes the return event. It can be set to Day, Week, or Month. The granularity you choose depends on the expected usage of your product (is it the sort of product that people should come to twice a week, or once a month?).

Run the query and you’ll see a line graph; you can also switch to table view to see a breakdown of activity for each cohort.

NOTE: this is only the most basic way to run a retention query. As you develop your analytics practice to run more hypotheses, you can start comparing customer segments and further break down your user base. To learn more, see Heap Docs, or watch the Heap University training course.

Measuring Baseline

To produce the retention graph that captures your company’s user base—the graph that will graph that will give you the baseline measurements against which you’ll want to run experiments—you’ll want to measure the number of users who have taken the action you’ve designated as your retention measurement, and plot those measurements over different time frames.

Since Heap captures all your data and enables retroactive analysis, it’s easy to look back and see how many people took a given action once, then when and how often they took that action again. Here’s how.

— Navigate to Analyze → Retention. Ensure the Retention Analysis option is selected at the top, then select the following:

— Start Event: The start event is the event that you’d like to use as the foundation of the retention report. This would generally be something basic that indicates initial interest in your product, like “signup,” “start trial,” or “purchase.

— Return Event: Generally the repeated action that you want to see over time. This should be the event that best defines a user’s getting value from your product (as described on the previous page).

https://docs.heap.io/docs/retention

https://learn.heap.io/heap-102-heap-essentials


Dashboard

Reports

Analyze

Graph

Funnel

Users

Retention

Influence

Paths

Queries

Define

Capture

Activate

Account

Retention

Group By

Filters

Compare Users

Date Range

First Time

ProductionMain

Save as ReportSearch for reports

Retention Analysis

Start Event

Session

Return Event

Add Filter

Session

Date of Start Event

Run Query

Past 7 Days grouped by Day

Add Comparison

Users retained from Click Login to Click - Save Report comparing 1 segment

Running a Retention Analysis in Heap


Critical: Avoid Measuring Incorrectly

When assessing the effects of a new feature on overall retention, a very common mistake is to simply measure your retention rates before the feature launch and your retention rates after.

The problem with measuring this way is that it doesn’t tell you how directly (or even if) your feature affected retention. (Briefly, it measures correlation, when you really want causation.)

ChurnedUsers

ReturnedUsers

Users Withan Action

A better process is to measure both the number of retained users who used your feature and the number of churned customers who didn’t use it. Comparing these two numbers helps you hone your analyses to better measure causation. (It’s also important to compare the number of retained customers who did use your feature with the number of retained customers who didn’t use it, for the same reason.)

If retained customers were using your new feature but churned customers weren’t, that’s a good indication that your feature did make a difference. (It’s not a conclusive result, but it’s far more indicative than simply measuring retention pre- and post-launch.) At very least, this result suggests that getting more users to adopt your new feature may make them more likely to return to your product.


Choose the Stage to Target

Once you’ve established your baseline retention metrics, you can start focusing in on different stages of the customer journey and develop experiments aimed at improving them. In general, your experiments will target one or more of the following goals.

1. You can get more new users to stick around longer.

2. You can raise the level of ongoing, repeatable value you provide.

3. You can figure out how people can continue to get value from your product, even after using it for a long period of time.

Figuring out which of these stages to focus on can take some work. If you haven’t thought about retention in this way, a good beginning strategy is to first put effort into short-term retention. This effort tends to have a cascading effect, since increasing short-term retention tends to pay dividends on medium- and long-term retention.

On the other hand, if your retention graph tails off strongly at the end, you should absolutely tackle this issue first, for the simple reason that if you cannot retain customers, you cannot sustain a growth business. In the following pages we offer some strategies that can help.

Get morenew users tostick aroundlonger

Get this lineto flattenout/get morepeople whocontinue tofind value

Raise thisline, get morepeople to findrepeatable value

Retention Rate

Stage 1 Stage 3Stage 2

70%

60%

50%

40%

30%

20%

10%

0%

Sam

em

onth

Aft

er 2

mon

ths

Aft

er 4

mon

ths

Aft

er 6

mon

ths

Aft

er 8

mon

ths

Aft

er 1

0m

onth

s

Aft

er 1

2m

onth

s

Finally, if you feel like you’ve sufficiently experimented and iterated with your on-boarding process and feel confident about your product’s ability to activate new users, it’s time to move on to the next part of the curve, and find ways to get users in front of other parts of the product.

The next few pages will provide some hypotheses you can test in each specific area.


S C E N A R I O 1

Hypotheses for Encouraging New Users to Stick Around

If your goal is to encourage more new users to play with your product, there are a number of standard experiments you can run to improve things.

They tend to cluster around a few key areas, though as always you’re encouraged to come up with other creative ideas.

On-Boarding Flow

Revisit your on-boarding flow. Is it so long that users drop off before they get through it? Are there moments of friction you hadn’t anticipated? Are you pointing users to quick wins, so they can see immediate value in your product?

Trial Periods

Depending on your product, you may want to offer (or may already offer) a trial period. There are many experiments you can run here. You can extend your trial period, or shorten it. You can add more features to your trial period, or remove them. You can ask users to provide a credit card number before the trial period, or move that request to the end. And so on!

Direction In the Product

Does your product intuitively direct people to the activities that will gain them value? Could it be organized differently? Could you present information in a different way (through guides, say)?

Cross-Functional Efforts

Cross-functional efforts can go far towards increasing the number of initial users. If launching a new feature, you can write blog posts about it. You can run an email campaign, targeting specific users whom you think would be most interested.

You can hold a launch party. You can do sales enablement to get your salespeople to mention the new feature more often. You can identify a group of users from whom you’d most like feedback and interview them. As with any business proposition, the possibilities truly are limited only by your time and creativity.


S C E N A R I O 2

Hypotheses for Creating Ongoing, Repeatable Value

As with stage one above, there are many directions you can pursue as you test out ways to keep more users engaged in your product.

Tiers of Usage

Spend time identifying what the beginner, intermediate, and advanced features (or types of usage) in your product are. Then figure out how to help users at different stages uncover them. You can do this in your product (with in-app guides, say or by targeting pop-ups to people who have signed up months ago) or through marketing and outreach (you could have your Account Managers call customers and make sure they know about your new feature, or you could launch a social media or paid advertising campaign).

Engagement With New Features

Examine your features and assess how discoverable they are. Are your features easy to use for newer users? Are they enticing to your existing user base? Are there features that will excite ongoing users, giving them new experiences? Are there incentives for users to start adopting different features?

Marketing

A general best practice is to make noise around new features, or around existing features you think are particularly important. You can partner with marketing to draw attention to them. An event? A press release? A customer campaign?

Deprecate

One powerful strategy is to deprecate features that are not used often but which make your product more complex. PMs can hate to let go of features they’ve had a hand in creating, but doing this—or testing the possibility out—can make for a simpler, easier-to-use product, and more quickly get users to value.


S C E N A R I O 3

Hypotheses for Creating Longer-Term Retention

As mentioned above, if you’re witnessing a major drop-off after a period of time, it’s imperative that you deal with it immediately. If you don’t fix this problem, you’re putting your entire business at risk. (If the issues is retention of a feature, your whole business may not be at risk, but it may be worth considering whether the feature is needed or not.)

End-stage drop off is a thornier issue to tackle, since the problems involved may have little to do with the product itself. The issue could be a new competitor to the space. Or a new direction in the way your customers work or do business. (It’s not really the horse’s fault that it got replaced by the train and the car.)

Moreover, the fact that people are leaving after using the product for a longer period of time dramatically extends the length of the experimentation cycle, making it more difficult to gather data on your hypotheses. When fixing this problem, you’re rarely going to get the kind of quick wins you can earn when dealing with the first two stages.

That said, there are a number of areas to investigate and experiment with. If you’re only dealing with a feature, the discussions may not be as intense. But they still merit the same kind of attention.

Examine Product-Market Fit

As before, these discussions will touch some more strategic issues. Take a deep dive into your product and business model to assess whether you’ve really achieved product-market fit. Are you selling the wrong product? Are you selling to the wrong people? Is your pricing model appropriate?

The issue could come from any part of your business. Are you giving customers enough support? Are you following up in the right ways?

Figuring out how to solve these issues will likely involve other key stakeholders in the company. Should you expand your product? Change your pricing model? Sell to a different market? And so on.


Make Your Product Indispensable

Work hard to isolate the features that make your product indispensable. Then double down on your key differentiator(s). What can you do that your competitors cannot? How do you make these features invaluable to customers, so that if you left the market they would feel real pain? These discussions will likely involve more than your product team, and may include any number of C-level executives.

Talk to Customers

If you’re not doing this already, it’s critical that you talk to customers who have churned. Find out why they’ve left and take copious notes. Base your decisions on this information.

Dropbox’s retention graph, from the company’s S-1. Nice and healthy!

2013 2014 2015 2016 2017 2018

Monthly Subscription AmountBy quarterly cohort


Examples From Heap

Finding Our Retention Metric

As we noted above, the first part of developing a retention strategy involves isolating the activity that identifies a user’s finding value in your product. Settling on this activity can take some time, and no small amount of experimentation.

A great strategy for pinpointing this metric (and what we did at Heap) is to convene a cross-functional team for a meeting, then collectively come up with a list of 10-12 activities that people think may work.

Once you’ve agreed on these 10-12 candidates, you can run retention analyses in Heap and see which activities bring the biggest lift.

03 S E C T I O N T H R E E


When Isolating Your Retention Metrics:

Time span matters

Knowing how long it takes users to get to a specific action changes your retention strategy, often dramatically. If the key retention event is something that’s taken in a user’s first day in the product, you’ll be far more aggressive in trying to get users to that event. If it typically takes a week, a drip campaign may be more appropriate.

Measure something useful

As we saw with the Google Analytics example on page (??), it’s important to measure something useful. You could track the number of people who click a log-in button, but simply logging in doesn’t generally designate value. Similarly, usage may simply be dependent on what software a company has purchased. Most large companies have a CRM in place, for example. As a result, many people at that company will perform activities in the CRM, often daily. That in itself tells you little about how happy those users are with that CRM.

Distinguish yourselves from your competitors

At Heap, we first started tracking whether people were running a query with our tool. But we then found that lots of customers ran sample queries just to see how the tool worked, giving us skewed retention metrics. So we decided to get more specific and see if people were using queries to answer legitimate questions.

We settled on “saving a report” as the activity that best indicated this. However, we soon realized that when we looked at the reports people saved, many were saving the exact same kinds of reports they could get in Google Analytics. So we ended up adding “uses the Group By feature” to our metric, so that our retention metric is not “saved a report that uses the Group By feature.” As GA doesn’t provide a Group By feature, the fact that users use it indicates that they’re gaining unique value from Heap.

See below for more information.


Two Retention MetricsAt Heap, we found two metrics that correlated strongly with retention: “Saving Reports” and “Group By.”

Saving Reports

The first, “Saving Reports,” told us that when a Heap user saves a report in their first session, they are 2x more likely to be long-term retained users of Heap.

This is important, we hypothesize, because saving a report is likely to indicate that a user is actually using Heap to run queries and produce an analysis that they found important enough to save.

To find this, we tested all possible activation events and correlated them with long-term retention. Because Heap enables retroactive analysis, we could easily run these queries and extend them into the past.

Importantly, these queries didn’t tell us whether it was the act of saving a report that caused users to be retained, or whether the fact that they saved reports in their first session indicated something else about these customers that made them more likely to keep using the product. The queries did, however, help focus our research.

Retention

Group By

Filters

Compare Segments

Date Range

First Time

Start Event

Start Event

View - Any App Page

Return Event

Add Filter

Execute Query

Dec 31, 2016 - Apr 27, 2018 by Week

Add Segment

Has done sequence Accept Invite → Save Report

Date First Seen after 01/01/2017

Retention of users who did View - Any App Page and then Execute Query, whereDate First Seen after 1/1/2017, by Whether User Has Done Sequence, then undefined


Group By

The second metric, “Group by” told us that when a Heap user uses the “Group By” feature when running a query, their engagement rates tend to be 2x those of users who don’t use Group By.

As noted above, we chose “Group By” because it was a feature that other tools, including Google Analytics, do not offer. So repeatedly using “Group By,” we hypothesized, indicates that a user is using Heap for something no other solution can provide.

As with “Saving Reports,” this query didn’t tell us whether it was the act of using Group By that caused users to be retained (that by using Group By they realized how powerful Heap was, for example), or whether the fact that they used Group By indicated something else about these customers that made them more likely to keep using the product (that they had already found value in Heap, for example, and were experimenting with Group By to see what it could do).

Has Run a Queryw/ GroupBy

Has Run a Query

51.86%

21.85%

(-10.15%)

(-22.84%)

After 10 months

Did Su�ested Reports improve query retention?

Retention

Group By

Filters

Compare Users

Date Range

First Time

Start Event

Start Event

View - Any App Page

Return Event

Add Filter

View - Any App Page

Past Year by Month

Add Group By

vs.

Has Run a Query

Has Run a Query


S U G G E S T E D R E P O R T S

A Hypothesis Story: Suggested Reports

Earlier this year, we hypothesized that our customers would benefit from having more guidance in the product. From customer interviews, we learned that our UI didn’t give customers clear enough information about what questions they could answer and why those questions mattered. We also learned that customers found our query builder difficult to use, since it had so many options.

To solve these problems, we convened a team and held a series of meetings. After running through various options, we settled on a hypothesis we were all willing to go with.

Our hypothesis was that by putting plain-English questions in our product and auto-filling queries to match, we could give customers more direction about the questions they could ask with Heap, and therefore provide more value. The ultimate goal was to drive increased usage.

We called this new feature “Suggested Reports.”



What We Learned

After rolling out Suggested Reports, we gave it some time and started measuring. Or, rather, we looked at the data. (Since Heap autocaptures all customer data, we never needed to “start measuring”; Heap was already measuring anything, without needing“track(‘event’)” tags.)

What we found: Only 10% of users who could discover Suggested Reports ended up using it. This number was far lower than we had hoped, especially since our hypothesis was that Suggested Reports would change the way many customers used Heap.

To learn more, we broke that number down. It turns out that there were two ways our product introduced Suggested Reports to customers. The first was on the “Reports” page, which presented the option directly. The second was on “Analyze” view, which offered Suggested Reports via slide-in panel.

The data showed that 13% of the users who saw the Reports page started creating a Suggested Report, and only 5% of the users who saw the Suggested Reports panel opened it.

These numbers suggested a few options. Maybe the issue was that Suggested Reports needed to be more discoverable. Maybe we needed to do more work to show users why Suggested Reports would be a good option for them. Or perhaps users were aware of what Suggested Reports could do, but simply found it not useful.

Clearly we needed to dive deeper.

View Analytics Page Click - Open SuestedReports Panel

4.70% 32.83%

Click - ChooseSuested Report



Impact on Retention

One way to start answering these questions about Suggested Reports—was the issue that Suggested Reports wasn’t discoverable enough, or was it simply not a useful feature?—was to measure the feature’s impact on retention. Our theory was that if Suggested Reports turned out to have a positive impact on retention, then customers who discovered Suggested Reports were in fact finding the feature valuable. If so, our energy might be better directed towards making Suggested Reports more discoverable.

To learn more, we dug into the Heap data. What did we find? Suggested Reports had an undeniably positive impact on retention.

First, we found that that users who had successfully run a Heap query with Suggested Reports had a 50% higher weekly query retention rate at 10 weeks vs users who hadn’t.

Put differently, if “run query at least once a week” was the metric we used to determine whether a user was finding value in Heap, Suggested Reports was undeniably correlated with increased usage.

While this information doesn’t prove causation (only correlation), as a first step it was quite encouraging.

Retention

Group By

Filters

Compare Users

Date Range

First Time

Retention Analysis

Start Event

Executive Query

Return Event

Add Filter

Executive Query

Users who have done

Run Query

Past 14 Days grouped by

Click Analyze (All)

Week

Add Comparison

Did Su�ested Reports improve query retention?

Has done Click Analyze (All)

Has not done Click Analyze (All)

56.54%

41.47%

3.81%

2.53%

After 2 weeks



Group By

Digging deeper into the data gave us even more information.

A bit of background: we already knew that the activation metric that signaled good retention for Heap was “running a query.” That is, we’ve learned that users who run a query are most likely to be those who find value in the product and continue to use it.

But, as we described above (page 20), to see if a user was getting value from Heap that they couldn’t get anywhere else, we started focusing in on customers who used the “Group By” feature in our product, since “Group By” is a feature unique to Heap. Our hypothesis was that if a customer is using “Group By,” they’re getting undeniable value from our product. And, as it turns out (page 18), users who use “Group By” are far more likely to continue to use Heap.

Given this, we decided to look to see if use of Suggested Reports was positively correlated with using Group By.

The results are at right. The data shows that people who used Suggested Reports were WAY more likely to also use Group By. Almost 50% more likely than people who didn’t use Suggested Reports! This was great news for us.

(Note: as before, this doesn’t necessarily prove causation; this information in itself doesn’t show that using Suggested Reports caused users to also use Group By, or to find more value in Heap. But it does show an association, which is useful, and gives us directions for further investigation.)

Has done Click Analyze (All)

Has not done Click Analyze (All)

31.41%

14.51%

2.91%

2.18%

After 3 weeks

Did Su�ested Reports increase Grouped Query Retention? Retention

Group By

Filters

Compare Users

Date Range

First Time

Retention Analysis

Start Event

View - Any App Page

Return Event

Add Filter

Run Query w/Group By

Users who have done

Run Query

Past 21 Days grouped by

Click Analyze (All)

Week

Add Comparison



Final Learnings

Whenever we launch a new feature, we like to examine the hypotheses we had at the beginning and see how they turned out. We also like to look at the data and see what next steps it suggests.

Hypothesis:

Our hypothesis was the Suggested Reports would make it easier for users to answer questions in Heap, and as a result would increase retention.

What we found:

— The good: It does seem that users who actually used Suggested Reports found value in it. Compared to users who did not use Suggested Reports, users who did use Suggested Reports were more likely to run queries, use Group By, and engage in the activation features that tend to indicate retention in Heap.

— The bad: Of customers who use Heap and were exposed to the Suggested Reports feature, only a very small percentage—10%—actually used the feature.

Next Steps:

Given that Suggested Reports has a positive correlation with retention, it seems worth continuing to build it out. Before doing that, however, we clearly need to make Suggested Reports more discoverable. There are many ways we can do that: we can present it more prominently in the product; we can add guides to the product that direct users to the feature; we can market the feature more aggressively; we can direct our Account Managers to help let customers know what Suggested Reports is and what it can do for them.

On to develop more experiments!


Closing Of all the ways to measure a customer’s interaction with your product, retention is arguably the one that tells you the most about the level and kinds of value customers get from your product. In this short book, we’ve tried to offer some methods for accurately measuring retention, figuring out which stages of retention most deserve your attention, and then implementing hypotheses to test various strategies for increasing the frequency and duration of customer engagement.

At Heap, we believe that the key to improving retention is to develop a hypothesis-driven, experimental, and iterative approach to product development. Our product framework is focused on helping PMs probe for opportunities to develop their product, then to capitalize on those opportunities, knowing that an ongoing series of small wins can add up to or pave the path towards major innovation.

We hope this has been helpful to you. If you have questions or thoughts about any of the material in this e-book, or simply want to know more about how Heap’s approach to product management can be useful for you, we encourage you to get in touch at [email protected].

We thank you very much, and look forward to working with you to innovate the future of product.

At Heap, we believe that the key to improving retention is to develop a hypothesis-driven, experimental, and iterative approach to product development.

04 C L O S I N G

mailto:mailto:support%40heap.io?subject=

www.heap.io

Documents

The Heap Guide to Retention · 2019-12-18 · The Heap Guide to Retention heap.io 3 Introduction Use the “leaky funnel” analogy or say your product is a “boat with a hole in