The product manager’s guide to customer-centered growth

ContentsINTRO

SECTION 1: BEFORE YOU START The two approaches to product growth

Why you need an experimentation process

Set your strategic goals

The four key documents you need

SECTION 2: THE EXPERIMENTATION PROCESSStep 1: Generate data-driven experiment ideas

Step 2: Prioritize your hypotheses

Step 3: Design and implement your experiments

Step 4: Analyze your test results and extract the lessons learned

Step 5: Systematize your results

SECTION 3: SUSTAIN GROWTH OVER TIMERepeat the process

Improve the process

Customer experience is the foundation of growth

03

04

08

13

The product manager’s guide to customer-centered growth 02

IntroMost products don’t fail today because their teams can’t write enough code. They fail because they can’t acquire and retain enough users.

And that’s why so many people are looking for the one tactic, secret, or hack that will skyrocket their growth curve. But we believe that focusing on specific tactics is the wrong approach.

In this eBook, we’ll make the case that, instead, you should focus on implementing a rigorous process of experimentation. And that the key to driving sustainable growth is using customer insights to discover the unique combination of tactics that will work for your product.

The material we’ll discuss is best suited for PMs working on products that have already found product/market fit and have a good handle on retention. You need to have built a “must have” experience that customers love before you transition into the growth phase.

Ideally, you already have a substantial amount of traffic, you’re ready to scale, and you want to make sure you do it right—by putting the user at the center of the process because that’s what pays off in the long run.

The key to driving sustainable growth is using customer insights to discover the tactics that work best for you.

And while every company will adapt this process to fit their own needs, our goal is to give you a step-by-step guide for implementing a highly experimental, customer-centered growth process into your organization. In this eBook, you’ll learn how to:

IMPLEMENT A PROVEN FIVE-STEP EXPERIMENTATION PROCESSUSE CUSTOMER INSIGHTS TO DISCOVER GROWTH OPPORTUNITIESEXTRACT MAXIMUM VALUE OUT OF EACH TEST YOU RUNINCREASE THE ACCURACY OF YOUR TESTS OVER TIME

The product manager’s guide to customer-centered growth 03

Section 1: Before you startTHE TWO APPROACHES TO PRODUCT GROWTH

There are two approaches to growing a product.

You A/B test a handful of tactics you think will work based on your opinion, and hope they improve your growth curve.

You start by looking at your analytics to identify the steps of your product flow where you’re seeing the biggest dropoffs. Once you know where the problems are, you gather qualitative data to get insight into what’s causing them. Then, you turn those insights into hypotheses and design experiments to test them empirically. And finally, you feed the lessons you learn about your product, customers, and channels back into the process to guide which tests you run next.

It’s a no-brainer. Your growth efforts will be much more effective if you establish a continuous, consistent cadence of experimentation to identify the leaks in your funnel and come up with test ideas based on multiple sources of data.

Peep Laja uses a great metaphor: “Would you rather have a doctor operate on you based on an opinion, or careful examination and tests?” You’re like a growth doctor, and your product is your patient.

WHY YOU NEED AN EXPERIMENTATION PROCESS

When you focus on tactics first, you run into three core problems:

Growth is a system, not a bag of tricks.” - Andrew Chen, Head of Supply Growth at Uber

The product manager’s guide to customer-centered growth 04

“1

2

What works for other products won’t necessarily

work for yours.

Your business model, product, and audience are

fundamentally different from anyone else’s.

Distribution channels are changing faster than ever.

Channels that work for you today will likely be saturated

and far less effective 12 months from now. To stay ahead of

the curve, you have to always be experimenting to discover

emerging channels.

There are no silver bullets.

No one-off tactic is going to solve all your growth

problems. The tactics that do produce explosive growth

are usually the result of a combination of lessons

learned from previous experiments.

Tactics are important. But instead of focusing on them first, focus on establishing a systematic and repeatable experimentation process. This will lead you to discover the combination of tactics that result in sustainable growth for your product.

Your growth efforts will be more productive and successful once you move away from testing random tactics and start approaching it as the scientific process it is:

In the following sections, you’ll learn an experimentation process originally developed by Brian Balfour. We combined it with ideas from Peep Laja’s ResearchXL framework, and added our own touches as well.

This process isn’t perfect, but it will give you a solid foundation to build on. Take it, copy it, and adapt it to your own team. But whatever you do, use it.

The product manager’s guide to customer-centered growth 05

There is no one right or perfect growth process. The important part is just to have one, stick to it, and improve it over time.”

- Brian Balfour, VP of Growth, HubSpot

“Set strategic goals

Gather and analyze data

Systematize

Analyze results

Generate hypothesis

Prioritize experiments

Design and implement tests

Once you’ve determined what area to focus on, you need to set a time frame for how long you’ll focus on it. We recommend a period of time between 30 and 90 days. Anything less than that won’t be enough to make substantial progress. And anything more will start to drag on, making it easy for your team to lose motivation.

Next, you have to decide how you’ll measure success. What quantifiable results will you use to measure whether or not you have achieved your goals? For example, if you decided to focus on improving your activation rate, the quantifiable results you measure might look like this:

activation rate

of users come back two times per week after week 1

of users come back two times per week after week 10

SET YOUR STRATEGIC GOALS

Before diving into the process, you need to figure out three things:

What do you want to accomplish?

When will you want to accomplish it by?

How will you measure success?

Start by taking a step back and setting your high-level strategic goals. Ask yourself this question:

What’s the one area of the customer lifecycle that, if you made substantial improvements, would have the biggest impact on your growth curve right now?

Is it acquisition, activation, retention, referral, or monetization? There’s no right or wrong choice, and the answer will be different for every product.

The product manager’s guide to customer-centered growth 06

Objectives and key results

The Objective and Key Result (OKR) framework is a goal-setting method used by many growth teams in Silicon Valley. It was first invented by Intel and popularized at high-growth tech companies like Google, Zynga, and LinkedIn.

If you’d like to learn more about how to set OKRs, we highly recommend Christina Wodtke’s guide.

35%

40%

15%

THE FOUR KEY DOCUMENTS YOU’LL NEED

There are four key documents you’ll use throughout this process to keep it driving forward:

Backlog The purpose of the backlog is to provide a public place for your team to dump and record all the growth ideas they come up with as they go through the experimentation process so they can come back to them later.

Pipeline This is a list of all the experiments your team has previously run, are currently running, and are going to run next. It allows anyone on your team, including your new hires, to see every single experiment you’ve run and the lessons you’ve learned along the way.

Experiment docs Every experiment gets its own document that outlines the objective, hypothesis, experiment design, results, analysis, and lessons learned. Experiment docs force your team members to critically think through their ideas to implement them successfully. To download an example experiment doc template, click here.

Playbooks These are step-by-step guides that anyone can follow to repeat what you’ve learned without having to talk to you.

Now, let’s dive into the details of each step of the experimentation process.

The product manager’s guide to customer-centered growth 07

1

2

3

4

The product manager’s guide to customer-centered growth 08

Section 2: The experimentation processSTEP 1: GENERATE DATA-DRIVEN EXPERIMENT IDEAS

The first step in the growth process is about collecting quantitative behavioral data and qualitative user insights to identify the biggest issues your customers and potential customers are experiencing—and then using those findings to guide your experiments.

Fixing these problems will help you meet your customer’s needs and expectations, craft an experience they’ll love, and improve your growth curve along the way.

But not all data is equally useful. You don’t just need any data; you need actionable data that gives you insight into how to optimize your specific area of focus. For example, if you’re focused on improving activation, break down your product’s onboarding and activation flow into its individual steps. And then collect data for each step.

Narrowing your scope down to the individual steps allows you to gather the right data that will give you insight into which levels you can pull to impact your area of focus—and prevent you from getting overwhelmed by having too much irrelevant data.

Start by gathering quantitative data that shows you where the problem is—where the biggest dropoffs are and where your funnel is leaking.

Once you know where the issues are happening, you can gather qualitative insight to understand why. What’s causing people to have doubts, uncertainties, and hesitations?

Once you know the source of the problem, you can come up with hypotheses to test about what you think would remove the friction.

Where’s your funnel leaking? What’s causing people to have doubts, uncertainties, and hesitations?

On-page surveys & live chat Place surveys on low-performing pages and ask people why they aren’t converting. Chat transcripts give you insight into common frustrations and where people get confused. User testing Observe people while they use your product and speak their thoughts out loud. Hear and see where people get stuck, confused, and frustrated.

Mouse tracking Use click maps, attention maps, and scroll maps to record what people do on your product pages with their mouse or fingers. Product walkthrough Hour-long team exercise. Review your product page by page with your team and identify where you can make the experience more relevant, clear, and value-adding—and where you can reduce friction and distraction.

Web analytics Identify specific pages and steps in your product flow where you’re losing customers and potential customers.

Technical analysis Identify annoying cross-device and cross-browser issues that are hurting your conversion rates. Fix issues if the added value is greater than the value of the time it will take your developers to identify and fix them.

Form analytics Know exactly which form fields people hesitate the most to fill out, which fields they leave blank even though they are required, and which ones cause the most error messages.

User session replays Anonymously record video of everything people do as they interact with your product. Watch how people respond to your product while risking their real money or personal information.

Product comparison Twenty-minute team exercise. Everyone picks one or two products from your competitive and noncompetitive space that they use actively. Walk through their product flow and discuss what they do well, what they could do better, and how you can adapt it to your own product and audience.

Growth network Talk to other growth professionals on the phone on a regular basis. Exchange ideas, share experiments, and discuss what worked, what didn’t, and why you think you got the results you did. Adapt ideas to your own product.

Question brainstorm Hour-long team exercise. Spend the first 20 minutes asking nothing but “why,” “what if,” and “what about,” questions. No discussion or answers. Write your question on a sticky note, announce it to the team, and put it on the whiteboard. This reveals what you don’t know and generates discussion.

The product manager’s guide to customer-centered growth 09

Here are 11 data sources you can use to generate data-driven test ideas. You don’t have to use every single one—choose a combination of three to five quantitative and qualitative sources that make the most sense for what you’re trying to learn. As you do your research, document all the issues you identify and test ideas you come up with in your backlog.

As you gather and analyze the data you collect, you should be dumping all the issues you identify and experiment ideas you come up with into your backlog. The next step in the process is to translate those findings into a prioritized pipeline of hypotheses to test.

These data sources are a combination of ideas from a book called The Innovator’s Solution by Clayton Christensen and the ResearchXL framework, originally developed by Peep Laja.

Once you’ve crafted the hypothesis, the next step is to prioritize each experiment based on three core factors, and, of course, to document them in your experiment doc.

• Probability

What’s the probability that this experiment will be successful? Use a rough estimation of low = 20%, medium = 50%, and high = 80%. You might give your test a high probability of success if it’s based on something you learned from a previous experiment. But if you’re testing a new channel that you have no experience in, you might rank it low.

• Resources

Roughly estimate how much time you’ll need from your design, engineering, and marketing resources to implement the experiment (i.e., 1 hour, 1 day, 1 week, 1 month, etc.).

• Impact

If this test is successful, what’s the total impact you expect to see? What do you predict the results will be? How much will it improve the targeted metric?

Once you’ve documented everything, you can compare experiment docs, weigh them against the other tests in your pipeline, and make an objective decision about which experiments you think will help you make the most progress toward your strategic goals.

STEP 2: PRIORITIZE YOUR HYPOTHESES

Now that you have a backlog full of potential tests to run, the next step is to turn your findings into prioritized hypotheses.

Every test idea needs to have its own experiment doc. And in this step, you’ll use it to document five things: the objective, hypothesis, probability of success, resources needed, and potential impact if the test is successful. The purpose of this exercise is to make you think through why you want to run this experiment and whether or not it’s actually a good idea.

First, describe what you’re testing, and why you’re testing it, in one or two sentences. This is your objective. Describe it so that someone else on your team will understand what you did if they read it a year from now.

Then, you need to write down a hypothesis for every experiment. Surprisingly, most people don’t do this. Here’s an example of what your hypothesis should look like:

You’re basically predicting how much you think the variable you’re targeting will increase, and then you’re laying out the underlying assumptions of your prediction. This allows other people on the team to challenge your assumptions and, hopefully, help you run more accurate tests over time.

The product manager’s guide to customer-centered growth 10

If successful, [variable X] will increase by [impact] because [assumptions].

Keep it as simple as possible. Think about it like this: how can you spend the minimum amount of time and resources to gather the data you need to understand whether or not your hypothesis is valid?

Spend five to ten minutes outlining the steps needed to implement the test, and document it in your experiment doc. The outline doesn’t have to be very detailed, but going through this process will do two things.

It forces you to think through the experiment beforehand and prevents you from making small mistakes and avoidable errors that would force you to rerun the test.

If anyone else on your team ever wants to repeat the test or iterate off it, they can understand exactly what you did without having to talk to you.

Okay, now that you’ve outlined your prioritized experiment, then next step is to run it. There are a few things you need to do to make sure your tests are statistically significant:

Test with the appropriate sample size

Test for an entire business cycle

Don’t end your test based on confidence intervals

We won’t dive too deep into the statistics here because a number of great resources covering the subject have already been created. Here are three articles that will teach you everything you need to know:

• How Not to Run an A/B Test• Crash Course on A/B Testing Statistics• 12 A/B Split Testing Mistakes I See Businesses Make All The Time

Growth experiments build on themselves. The more experiments you run, the more you learn—which allows your experiments to become more accurate over time. That’s why we recommend starting with experiments that offer high business value and aren’t very resource-intensive. Here’s a matrix that really helps simplify the prioritization process:

Finally, once you’ve prioritized the list of hypotheses in your backlog, move the highest priority experiments that you plan to run next into your pipeline.

STEP 3: DESIGN AND IMPLEMENT YOUR EXPERIMENTS

Once you’ve dropped your prioritized hypotheses into the pipeline, the next step is to design and implement your controlled experiments.

The product manager’s guide to customer-centered growth 11

1

2

VALUE VS. COMPLEXITY QUADRANT

1 2

?

HIGH

LOW HIGH

Business value

Complexity / effort

- Values vs. complexity quadrant. Jim Semick, the founder of ProductPlan.

The product manager’s guide to customer-centered growth 12

STEP 4: ANALYZE YOUR TEST RESULTS AND EXTRACT THE LESSONS LEARNED

Once you’ve run your controlled experiment, the next step is to analyze the results and extract the lessons learned. As you determine the success or failure of an experiment, look at three factors and document them in the experiment doc:

Impact. What were the results of the experiment?

Accuracy. How close were the results to your hypothesis?

Why? Why do you think you saw the results you did?

“Why?” is by far the most important question you can ask because the answers you come up with will lead you to new ideas and experiments to run next. Use this question to extract as much insight as possible and, again, document all the lessons you learned in the experiment doc.

Once you capture what you learned, make a list of action items that consists of tests you want to run next. And then transfer those ideas into your backlog. Now that you’ve gained a deeper understanding of your customer, product, and channel, go back to the experiments in your pipeline and adjust the predictions you made based on what you know now.

Feeding what you learn back into your backlog and predictions is the most critical part of this experimentation process—it’s what allows your experiments to become increasingly accurate and successful over time. If you skip this step, you’ll fail to build on your experience and you’ll constantly have to start over from square one. You’ll essentially be throwing away the value you’ve created.

STEP 5: SYSTEMATIZE YOUR RESULTS

This is the final step in the process. Once you discover a successful experiment, try to implement it as a new product feature or initiative. For example, if you only ran your test with 15% of your audience, then launch it to all of your users as a new part of the product experience.

But sometimes your team will discover things that are more manual and can’t be productized—for example when a growth marketer on your team discovers an effective method from promoting a piece of content to a specific segment of your audience. If that’s the case, turn it into a playbook, which is a step-by-step guide that anyone can use to repeat what you did and get the same result. Make your playbooks publicly available to everyone on the team to read, comment, and collaborate on.

This allows you to capture, retain, and share the knowledge and value you create with successful experiments.

is by far the most important question you can ask. The answers you come up with will lead you to new ideas and experiments to run next.

“Why?”

Section 3: Sustain growth over timeREPEAT THE PROCESS

Repeat steps 1-5 as many times as you feasibly can in your 30-90 day period. Take the lessons you learn from analyzing experiment results and continuously feed them back into your backlog, and use them to refine the predictions in your prioritized experiments.

This feedback loop plays a critical role in making your process more accurate with each successive cycle.

IMPROVE THE PROCESS

Once you go through this 30-90 day process a few times, you need to take an even bigger step back to examine the effectiveness of your efforts from a macro level and think about how you can optimize them. You need to look at three things in particular:

Batting average What’s your team’s ratio of successful experiments to failures? This number should be getting better over time.

Accuracy Are your hypotheses getting more accurate as you gain more experience in a channel? The longer you work in a channel, the more accurate your experiments should get.

Throughput How many experiments you do you run in a given period of time? How can you increase the velocity?

You can improve those three things by constantly optimizing your process, your team, and the tools you use. And as a result, your experimentation efforts will become more accurate and more successful over time.

The product manager’s guide to customer-centered growth 13

1

2

3

Feeding the lessons you learn back into your backlog and using them to adjust your predictions is what allows this process to become increasingly accurate and successful over time.

CUSTOMER EXPERIENCE IS THE FOUNDATION OF GROWTH

As you run experiments and optimize the funnel, remember that customer experience and growth are inextricably tied. None of your efforts will matter if you don’t deliver an experience people love.

The key to this process is that your team puts the customer at the center of it—constantly gaining a deeper understanding of your customers, how they use your product, and which channels are effective for reaching them. As Sean Ellis eloquently said:

“I found that the key to sustaining rapid growth is understanding your ‘must have’ experience and then aligning the entire business around that experience. This includes aligning the product roadmap, funnel optimization, and messaging.”- Sean Ellis, founder of Qualaroo and GrowthHackers.com

The most important factor for sustainable growth is whether or not your product meets your customers’ needs, solves their problems, and produces their desired outcome in a way that they enjoy and want more of.

If you don’t deliver a “must have” experience, it won’t matter what process or growth tactics you use. It’s hard to justify investing in top-of-the-funnel acquisition if every new user eventually leaves. Bottom line: the better your customer experience is, the more customers you’ll retain, the more people they’ll share your product with, and the easier it will be to grow.

The product manager’s guide to customer-centered growth 14

If you don’t deliver a “must have” experience, it won’t matter what

process or growth tactics you use.

Create great experiences

UserTesting is the fastest and most advanced user experience research platform on the market. We give marketers, product managers, and UX teams on-demand access to people in their target audience who deliver audio, video,

and written feedback on websites, mobile apps, prototypes, and even physical products and locations.

2672 Bayshore Parkway, Mountain View, CA 94043www.usertesting.com | 1.800.903.9493 | sales@usertesting.com