Validating Ideas with Users

USER-CENTRIC PRODUCT DEVELOPMENT

VALIDATING IDEAS WITH USERS

Sheen YapDreamstake Workshop 6:30pm 9 April 2014 London UK

!..HOW TO CREATE PRODUCTS THAT PEOPLE WILL USE

INTRODUCTION

• Sheen Yap 

• 17-years’ experience across web, mobile, iTV

• UX expert, design leader and lean evangelist

• helping startups build better products

WORKSHOP OBJECTIVES

• Early stage startups - new idea, pivoting or existing

• Go through process of validating ideas

• Walk through and apply at each step

• Understand and willing to perform user testing

• Have fun, make mistakes, learn!

HOW DOES THIS FIT IN?

LEAN STARTUP CYCLEEarly stage application!

CUSTOMER DEVELOPMENT

LEAN RUNNING DEV MODEL

HORROR STORYTIME

PROBLEM-SOLUTION FIT - NOW

• Product idea is about problem-solution fit

• Need to solve Real Problems for Real People

• Avoid eventual failure

• Avoid waste in time, effort and money 

• Avoid pain in dismantling

• Missed opportunity to pivot to a better idea

..HOW TO CREATE PRODUCTS THAT PEOPLE WILL USE HOW TO AVOID CREATING PRODUCTS NO ONE WILL USE

TEST WITH USERS; TEST EARLY.

HOW DO WE DO IT?

1. Create Model

2. List Assumptions

3. Generate Hypothesis

4. Design Experiment

6. Run experiment

7. Analyse Results

8. Decide Next Steps

5. Experiment (interview)

Captured Data

1. Create Model

2. List Assumptions

3. Generate Hypothesis

4. Design Experiment

6. Run experiment

7. Analyse Results

8. Decide Next Steps

5. Experiment (interview)

Captured Data

1. Create Model

2. List Assumptions

3. Generate Hypothesis

4. Design Experiment

6. Run experiment

7. Analyse Results

8. Decide Next Steps

5. Experiment (interview)

Captured Data

CREATE!MODEL

“IDEAS”

• Ideas normally come as a “solution” i.e. A Thing that has A, B and C Features.

• We are seduced by its perceived tangibility

• No way for anyone else to evaluate its true value to the world

• Can’t tell if it fulfils a genuine need

• Restricts the flexibility for a better solution

MODELLING

INITIAL FOCUS

EXAMPLE - FACEBOOK 2004

Online presence lacking

Hard to stay in contact

College Students

Harvard Students

Personal Profile

Friend Connect

EXERCISE

EXERCISE

• Who are your customer segments?

• What are their problems/pain points/issues?

• Who are your customers?

• What are their problems/pain points/issues?

• Who could be your early adopter?

1. Create Model

2. List Assumptions

3. Generate Hypothesis

4. Design Experiment

6. Run experiment

7. Analyse Results

8. Decide Next Steps

5. Experiment (interview)

Captured Data

LIST!ASSUMPTIONS

ASSUMPTIONS

• “Theory behind the statements”

• Unvalidated, each statement is a figment of our imagination: just because we believe it, does not mean it’s true

• Everything on the canvas are assumptions - we need to methodically test each of them

ASSUMPTION STATEMENT

• We believe that: [user segment] want to do [expected behaviour] because [pain point].

• We believe this [user segment] exist and we can find them at the following [channels]

• We believe that [user segment] will pay [amount] to use our service.

• We believe that [our UVP] will solve the [user segment]’s [problem].

EXAMPLE - FB’04

• Online presence lacking:We believe that lots of students find their current online presence unsatisfactory

• Hard to stay in contact: We believe that many students worry about losing contact with others

EXERCISE

• Pick one of your problems

• Rewrite into an assumption: We believe that [user segment] has this [pain point/issue/need].

1. Create Model

2. List Assumptions

3. Generate Hypothesis

4. Design Experiment

6. Run experiment

7. Analyse Results

8. Decide Next Steps

5. Experiment (interview)

Captured Data

GENERATE!HYPOTHESIS

FALSIFIABLE HYPOTHESIS

• Assumptions need to be testable for validation

• Creating a Falsifiable Hypothesis: i.e. can be proved true or false by empirically testing

• Format: If I were to [perform activity] on [sample subject], then I would be able to [observe an outcome]

EXAMPLE AND EXERCISE

• Assumption: We believe that lots of students find their current online presence unsatisfactory

• Hypothesis: If I were to investigate a number of Harvard students, I would find that 90% of them would be unhappy with their online presence

• Create a hypothesis for your assumption:If I were to [perform activity] on [sample subject], then I would be able to [observe an outcome]

1. Create Model

2. List Assumptions

3. Generate Hypothesis

4. Design Experiment

6. Run

experiment (perform

7. Analyse Results

8. Decide Next Steps

5. Experiment (interview)

Captured Data

DESIGN EXPERIMENT

WHY EXPERIMENTS

• Typical statement: “Built something, put it out there, learnt a lot.”

• Question: What did you REALLY learn?

• Stuff that could be semi-relevant, might be useful later. Erm, hopefully.

• Wasted cycle: still no progress on your most critical problems; worse, could be distracted onto a different direction

AN EXPERIMENT

• Has a learning goal

• Explicit stated hypothesis

• Anticipates observable outcome

• Repeatable

• Avoid biases (try our best to!)

EXPERIMENT TEMPLATE

• Goal: to validate your stated assumption

• Hypothesis: state the falsifiable hypothesis

• Procedure: more on that next…

• Expected Results:

• Actual Result:

EXPERIMENT TEMPLATE

Goal: To validate your stated assumption

Hypothesis: Put your hypothesis here

Procedure: will talk about this next…

Expected Results: come back to this later…

Actual Results: input results after running the experiment…

1. Create Model

2. List Assumptions

3. Generate Hypothesis

4. Design Experiment

6. Run experiment

7. Analyse Results

8. Decide Next Steps

5. Experiment (interview)

Captured Data

EXPERIMENT

WHICH PROCEDURE?

Usability Testing

Ethno-graphy

InterviewsPartici-patory Design

Focus Groups

Eye Tracking

Compe-titive

Research

Diary Study

Bench-marking

Forum Mining

Blogpost Test

Landing Page Tests

PPC Tests

A/B Tests

Customer Feedback

Card Sorting

Wizard of Oz

Neuro Imaging

Data Mining

Data Analysis

Ad Hoc Question-

naire

Intercept Surveys

Email Surveys

Product Stubs

INTERVIEW OBJECTIVES

1. Validate ProblemDo customers have the problem you are solving?

2. Capture Existing Alternatives How do customers solve these problems today?

3. Validate Customer Segments Are these customers the ones with the pain?

INTERVIEWING TIPS

• Set context firstStatement or closed question.

• Then ask open questions Use What/How/Why?

• Follow up questions Why? Tell me more?

• Let them talk; expect messy outcome

INTERVIEW QUESTIONS

• Hypothesis: If I were to investigate a number of Harvard students, I would find that 90% of them would be unhappy with their online presence

• Context setting: I’m trying to find out more about people’s online presence at the moment. Can you tell me if you have an online presence?

• Open questions: (If yes) What do you have? How are you finding it? (If no) Why is that?

• Now write your own question(s) for your hypothesis

EXPERIMENT TEMPLATE

Goal: To validate your stated assumption

Hypothesis: Put your hypothesis here

Procedure: Your interview question(s) to applied to ? people

Expected Results: State your expected outcome

Actual Results: input results after running the experiment…

TEMPLATE EXAMPLE

Goal: To validate our belief that lots of students find their current online presence unsatisfactory

Hypothesis: If I were to investigate a number of Harvard students, I would find that 90% of them would be unhappy with their online presence

Procedure: Interview question(s) to applied to 5 people

Expected Results: At least 4 people will register some complaint about their current online presence

Actual Results:

TEMPLATE EXERCISE

Goal: To validate your stated assumption

Hypothesis: Put your hypothesis here

Procedure: Your interview question(s) to applied to ? people

Expected Results: State your expected outcome

Actual Results:

1. Create Model

2. List Assumptions

3. Generate Hypothesis

4. Design Experiment

6. Run experiment

7. Analyse Results

8. Decide Next Steps

5. Experiment (interview)

Captured Data

RUN EXPERIMENT

INTERVIEW PROCESS

!

• Perform interview with the person next to you

• Interviewee pretend to be target audience

• Try noting/recording in verbatim

• Take turns, 5 minutes each

INTERVIEW EXAMPLE

• Interviewing Harvard Student, circa 2004

• I’m trying to find out more about people’s online presence at the moment. Can you tell me if you have an online presence? “Yes I do.”

• What do you have? How are you finding it?“I have the college directory profile but it’s very limiting in terms of the information it shows, doesn’t tell people much about the real me. I also have a myspace page but it’s kinda kept between my friends and not shown to others in college as stuff there isn’t appropriate. It’s fun but I am spending far too much time maintaining it.”

INTERVIEW EXERCISE

!

• Perform interview with the person next to you

• Interviewee to pretend to be target audience

• Try noting/recording in verbatim

• Take turns, 5 minutes each

1. Create Model

2. List Assumptions

3. Generate Hypothesis

4. Design Experiment

6. Run

experiment (perform

7. Analyse Results

8. Decide Next Steps

5. Experiment (interview)

Captured Data

ANALYSE RESULTS

ANALYSIS GUIDELINES

• Complete ALL interviews first

• Based on what you have recorded:

1. Has it met your expected results, not met them or inconclusive? What does it mean to your hypothesis?

2. What new information have you heard?

3. Any new ideas/insights?

ANALYSIS EXAMPLE

• I have the college directory profile but it’s very limiting in terms of the information it shows, doesn’t tell people much about the real me. I also have a myspace page but it’s kinda kept between my friends and not shown to others in college as stuff there isn’t appropriate. It’s fun but I am spending far too much time maintaining it.

Result: complains registered Hypothesis: validated New info:

college directory limiting

New info: myspace is inappropriate for everyone

New info: Both don't present the “real me”

Insight/Idea: Showing Real Identify is important

ANALYSIS EXERCISE

• Based on what you have recorded:

1. Has it met your expected results, not met them or inconclusive? What does it mean to your hypothesis?

2. What new information have you heard?

3. Any new ideas/insights?

1. Create Model

2. List Assumptions

3. Generate Hypothesis

4. Design Experiment

6. Run

experiment (perform

7. Analyse Results

8. Decide Next Steps

5. Experiment (interview)

Captured Data

DECIDE NEXT STEPS

DECIDING NEXT STEPS

Revise Canvas/Assump-

tionsHypothesis Validated?

New Insight

Gained?

New Insight

Gained?Pick Next

Experiment

Modify Experiment

Experiment Completion

Start New Experiment

Yes

No

InconclusiveBrainstorm New Ideas/

Insights

No

Yes

Yes

No

PROCESS RECAP

1. Create Model

2. List Assumptions

3. Generate Hypothesis

4. Design Experiment

6. Run experiment

7. Analyse Results

8. Decide Next Steps

5. Experiment (interview)

Captured Data

THIS IS LEAN UX

TEST ASSUMPTIONS

✔ ✗

✔

✗

✔

✗ ✗

✗ ✔

✗

✔ ✔

✗ ✔

✔ ✗

✗✔ ✔ ✗ ✔

✔✗

✔

PICK THE RIGHT TOOLS

Usability Testing

Ethno-graphy

InterviewsPartici-patory Design

Focus Groups

Eye Tracking

Compe-titive

Research

Diary Study

Bench-marking

Forum Mining

Blogpost Test

Landing Page Tests

PPC Tests

A/B Tests

Customer Feedback

Card Sorting

Wizard of Oz

Neuro Imaging

Data Mining

Data Analysis

Ad Hoc Question-

naire

Intercept Surveys

Email Surveys

Product Stubs

IT IS ABOUT PEOPLE…

…try walking in their shoes!

CHANGE WILL GET HARDER

MAKING THE NECESSARY CALLS

REFERENCES

QUESTIONS?

Sheen Yap twitter: @sheen uk.linkedin.com/in/sheenyap