Measuring What Matters: A UX Approach to Metrics :: UX Days Tokyo [April 2015]

UX Days Tokyo | Measuring What Matters | @katerutter | APRIL 19, 2015

a UX Approach to MetricsMeasuring what Matters

Sunday, April 19, 2015 | Kate Rutter | @katerutter


Letʼs get to know each otherAt each table...* name* role * your favorite superpower

Hello!5 min


@katerutter

tweet-friendly




I am not an expert on your business. You are.

My goal is to help you explorehow to measure how well your product

serves real humans...

so you can use metrics to design & improve the products you build.

What to expect...


What to expect...We are going to be working fast.


What to expect...

There is a purpose to this.I will push you to get something

down rapidly. In 1 day, youʼll learn how to create actionable, UX-centered metrics to

track progress of your product.

We are going to be working fast.


What to expect...You will be working solo and in pairs.Note that these techniques are

very powerful in teams.

Weʼll keep talking to a minimum.(Happy to talk more after the workshop.)


What to expect...You may not like the pace/process/

pressure/perspective, etc. etc.That is okay.

All I ask is that you keep an open mind and try it.


How do you measure UX

now?

10 min


UX + metrics=

<3


•Identify and design for key interactions that deliver measurable value to your users.

•Have a clear set of numbers that indicate progress.

•Know what interactions you need to track to measure user behaviors.

Why?


Why this matters:“Over 10,000 downloads!”

“Avg time on site is 22 minutes!”

“+450 new sign-ins this week!”




“+450 new sign-ins this week!” FAIL




“+450 new sign-ins this week!” is this good?


Product(black box)

Conversion Funnel

Traffic


Product(black box)

Conversion Funnel

Traffic


~ Alistair Croll

Don’t just ask questions. Know how

the answers to the questions will

change your behavior.


A good metric is...

Clear & specificComparativeRate/RatioChanges your behavior


What we will make

Snapshot

What people? What problems? What product?

How will we track progress?

Metric plan

What can a user do with our product?

Key use

What metric matters?

Key metric


Make a product

snapshot


Letʼs get warmed up

A product you work onAn idea for a new product

Pick an idea to use for the day

2 min

Task-a-Doodle

write it on a stickynoteor


Create a Product Snapshot

Snapshot



Key use


Metric plan


Key metric


a mobile app for delegating tasks

Working parents with with kids

need to know when something’s done

Too much to do, not enough time

need to share tasks on the go

Task-a-doodle 4 19 2015

10 minWho are the people?

(the users)What problems do

they have?What is a proposed

solution?

Product Snapshot

BE SPECIFIC


Is it clear?Is it specific?What can be improved?

Product Snapshot

Share as a pair

5 min(each)


What you have now...

Snapshot



Key use


Metric plan


Key metric


theUX STACK


How do I KNOW IT’S WORKING?

I have a product.

so...


Who is it for?

Why does it matter to them?

What can they do that they couldnʼt do before?

What features do they need to do that?

How do those features fit together?

Users

Needs & goals

Uses

Features

��

��

Product, brand & voice

�via @luxrco

UX elements


Users

Needs & goals

Uses

Features

��

��


� Features

Users

Uses

Product

Needs

via @luxrco

UX stack

via @luxrco


Users

Needs & goals

Uses

Features

��

��


� Features

Users

Uses

Product

NeedsCompany Purpose

via @luxrco

sketches, prototypes, pixelsUI

UX stack

via @luxrco


Envision aKey Use


Envision a Key Use

Snapshot



Key use


Metric plan


Key metric


via @luxrco

Features

Users

Uses

Product


via @luxrco


UX stack


An example from Foodspotting

6-up uses


Envision 6 uses

sketch!Task-a-doodle, Mary

4 19 2015



30 sec


Rapid SketchingStar people!

People, Context & Emotion


You can draw a person in 4 simple steps....Star People


Star People in contextUse simple shapes to put a person in a place.


smile frown

up brows

neutral

down brows

no brows

add a mouth...

add eyebrows...

Expressions Matrix


2 min

Speed-sketch 10 star people!


Key Uses 6-up

OK to be obvious

Task-a-doodle, Mary4 19 2015

•1 idea per panel•6 different ideas (not a sequence)•Sketch AND write

What can someone DO with your product?- Solves the problem- Addresses a need

10 min


Key Use

•This is info transfer; not a new idea•Make the sketch clean & clear

Select one and redraw.

5 min

Task-a-doodle,Mary

4 19 2015

Share tasks with others.

What can someone DO with your product?

OK to be obvious



Snapshot



Key use


Metric plan


Key metric


Lunch!

http://www.g4tv.com/attackoftheshow/blog/post/715478/cutest-bento-boxes-for-geeks

1 hr


Identify a Key Metric


Identify a Key Metric

Snapshot



Key use


Metric plan


Key metric



a good metric... a great metric......makes you look at all the other metrics and say “none of those other numbers matter if we donʼt get this right first.”

...measures the usage of your product by a person. The usage should be specific to features that deliver value to your user.

vanity

total number of registered

users

unhelpful

sign-ups

% of users who share a

task 3+ times a day,per week

awesomebetter% of users who sign in 3+ times a

day,per week

good% of new users per

week

Metrics checklist


Brainstorm 5 ideas for metrics

5 min

What could you count to make sure the Key Use is happening?


Pick one to focus on 1 min


3. Is there an object basis?

• “per user” [ ___/user]• “per user per week” (fancy!)• Examples: per user, per session, or by interaction or item (messages per file, emails per thread)

* not required, but super-helpful

• “per week” [ ___/wk ]• “per month” [ ___/mo]2. Is there a time basis?

• “number of...” [ # ]• “average number of...” [avg ]• “percent of...” [ % ]

1. Does the metric begin with a number?

Make it awesome 5 min


Share your workReview the snapshot &

use. Take 3 minutes to

confirm clarityPeers: make

recommendations

Share as a pair

5 min(each)

Key Metric


10 min

Questions & Discussion


% of users who wear a tie within one week of receiving it, per month.

% users who wear a received tie 3+ times/week, per week.

% users who purchase a tie 3 shipments in a row, per week.


Put it on your dashboard 1 min


Pick another one 1 min










Put it on your dashboard


Break!

15 min


One more time 1 min










Share your workReview the snapshot &

use. Take 3 minutes to

confirm clarityPeers: make

recommendations

Share as a pair

5 min(each)

Key Metric


Put it on your dashboard 1 min

Pick ONE to work on next



Snapshot



Key use


Metric plan


Key metric


Make a Plan


Make a Metric Plan

Snapshot



Key use


Metric plan


Key metric


Metrics Plan

•Set a goal•Set a time span•Think about how to collect the data

How will you track progress?

% of users who share a task 3+

times a day,per week

Capture clicks on “share now” - capture # of shares - timestamp each - capture by user ID (dump into raw data file) - use total # of users as base1. Calculate by day, # of users with 3+ shares from 12:00am - 11:59pm2. Use threshold of 3+ to order list3. # of users above threshold / Total # of users x 100 = %Week starts Monday @ 12:00am, ends Sunday @ 11:59pm

30%8 1 20155%

4 19 2015


Metrics Plan


How will you track progress?




30%8 1 20155%

4 19 2015



Three ways to gather data: • Non-technical • Technical “by hand” • Instrumented with software

Instrumentationprocess of setting up the technology to capture the numbers that matter to you. This often means using an analytics package or writing custom code.


Metrics Plan


20 minHow will you track progress?




30%8 1 20155%

4 19 2015


Pass your work1-minute “quiet read”

5 minutes of questions to confirm clarity and

then feedback.Peers: Offer help and

advice.

Share as a pair

15 min(total)

Metrics Plan



Snapshot



Key use


Metric plan


Key metric


Tracking over time


Track over time




Pull it all together.

Tell the story.


via @luxrco

Features

Users

Uses

Product


via @luxrco


UX stack



Snapshot



Key use


Metric plan


Key metric


This is [product name] .

* Minimum Viable Product

10 minMetrics Gallery

The metric & plan:

The value:

The idea...


10 min

Questions & Discussion


Geek out & learn tons more


Kate Rutter@katerutterhttp://intelleto.com

Thanks!

UX Days Tokyo | Measuring What Matters | @katerutter | April 2015

Handouts

a UX Approach to Metrics

Measuring what Matters

UX Days Tokyo | Measuring What Matters | @katerutter | April 2015

Features

Users

Uses

Product



UX Stack

Product Snapshot

people

problem solution

have

addressesuse

Product name date

/ /

With _________________ , _________________ can...

Key Uses 6-up sketchesdate

/ /

With

can...

!

Key Usedate

/ /

6

a great metric......makes you look at all the other metrics and say “none of those other numbers matter if we donʼt get this right first.”

a good metric......measures the usage of the product by a person. The usage should be specific to features that deliver value to the user.

Making metrics actionable

vanity good betterunhelpful

total number of registered

users

% of users who share a

task 3+ times a day,

per week

sign-ups

awesome% of users who sign in 3+ times a

day,per week

% of new users per

week

Does the metric begin with a number?• “percent of...” [ % ]• “average number of...” [avg ]• “number of...” [ # ]

Is there a time basis?Is it comparative? • “per week” [ ___/wk ]• “per month” [ ___/mo]

Is there an object basis?* not required, but super-helpful• “per user” [ ___/user] • “per user per week” (fancy!)

1Metrics Check

2 3

metric

by

Goal

/ /

Notes

Metrics Dashboarddate

/ /

To assess progress, we will measure these metrics:

metric

by

Goal

/ /

Notes

metric

by

Goal

/ /

Notes

by

Goal

/ /

Key Metric plandate

/ /

To assess progress, we will measure this metric:

Measured

Baseline

Daily Weekly Monthly

Hereʼs how weʼll collect the data:

metric

Metric Tracker

Design

Measuring What Matters: A UX Approach to Metrics :: UX Days Tokyo [April 2015]