Safety in Numbers: A framework for benchmarking the user experience

Jessica Cameron & Abi Reynolds

UX Scotland

Thursday, 8 June 2017

1

If you can’t measure it

You can’t manage it

2

Arron Walter, UX Matters

Wisdom

Knowledge

Information

Data

3

I want others to see the value of what we are doing.

I want to see my results make a difference.

I want to see my results acted upon.

UX folk

4

If you don’t know the value of what you are doing, there is no value.

5

Gerry McGovernwww.customercarewords.com

Q. How do we go about

benchmarking or measuring the user experience?

6

7

Open GI – August 2015

7

8

9

UX

Strategy

Profit

SEO

Campaign

Product

Channel shift

Social

More customer focused

A better user experience, easy and simple to use

Customer first

Put the customer at the core

Increase satisfaction

10

11

Start with strategy

12

Vision Landscape Roadmap Measure

1. Where we are now (benchmarking)

2. Where we want to get to (what does success look like in terms of UX)

3. How do we know if we got there (breaking success down into meaningful metrics related to UX)

4. Monitor changes and improvements

13

Measurement plan

Measure

14

Measurement frameworks

15

UX Score

16

17

UX Score

What works for us:• Goes beyond just usability and looks at other

useful attributes of the user experience • Can be compared and tracked over time

What doesn’t work as well for us:• Reduction to a single metric• One size fits all approach

18

Google HEART Framework

19

20

Google HEART Framework

What works for us:• Focus on using categories that suit goals• Process of mapping goals > signals > metrics

What doesn’t work as well for us:• Implicit reliance on large data sets• How can we improve?

21

22

In our heads…

Specific requests

from clients

An objective, repeatable

process

Why aren’t more people booking

flights on mobile?

How can we make sure we’re offering a best-in-class retail

experience?

23

Our method

A structured expert evaluation that assesses how well websites meet (or exceed) the norms and standards within an industry.

A snapshot of the state of UX amongst competitors.

24

25

Goal

Context

Metrics

26

How can UX benchmarks help improve……an airline app?

27

…an eCommerce site?

28

Clarify the goals

29

Goal To increase the percentage of users

completing bookings on the mobile app

30

Understand the context

31

User journey

32

Identify key user journey

a. What is the user journey that is most relevant to meeting your goal?

b. What are the discrete steps in that journey?

33

User journey Book a flight

34

Key steps in the user journey

FIND SELECT CHECKOUT

35

Competitors

36

Select competitors

a. Consider the market (direct competitors)b. Consider the goal (aspirations)c. Research the state of user experience in the

industry (mobile, interfaces)

37

CompetitorsMix of direct competitors, market leaders and

innovative new entrants

38

Choose your metrics

39

Our method

A structured expert evaluation that assesses how well websites meet (or exceed) the norms and standards within an industry.

40

Choosing metrics: step 1

Select UX factors These are the key steps in the user journey plus relevant UX elements and heuristics.

Good news: you have already identified the key steps in the user journey!

41

Key steps in the user journey

FIND SELECT CHECKOUT

42

UX FactorsWhich ones are most appropriate?

43

Choose factors

a. Steps in the user journeyb. Relevant UX elements and heuristics

Which factors are most relevant to the site and to the goals?

44

1. First impressions2. Find products (or information)3. Select products4. Configure products5. Get help and support6. Checkout 7. Post-order communications8. Delivery 9. Returns

Steps in the user journey

45

1. Navigation / menus2. Search3. Clarity4. Readability5. Functionality 6. Visual appeal7. Confidence and trust8. Speed

UX elements

46

Usability heuristics (Nielsen)

1. Visibility of system status2. Match between system and real world3. User control and freedom4. Consistency and standards5. Error prevention6. Recognition rather than recall7. Flexibility and efficiency of use8. Aesthetic and minimalist design9. Help users recognise, diagnose and recover from errors10. Help and documentation

www.nngroup.com/articles/ten-usability-heuristics/

47

1. Aesthetics2. Anticipation3. Autonomy4. Colour5. Consistency6. Defaults7. Discoverability8. Efficiency of the User9. Explorable Interfaces10. Fitts' Law

11. Human-Interface Objects12. Latency Reduction13. Learnability14. Metaphors15. Protect Users' Work16. Readability17. Simplicity18. Track State19. Visible Interfaces

Principles of interaction design (Tognazzini)www.asktog.com/atc/principles-of-interaction-design/

48

Choosing factors

Which UX factors are most relevant to achieving your goals?

What might you want to measure on• A government site, where the goal is to reduce the

number of incoming phone calls? • An airline app, where the goal is to increase mobile

bookings?

49

Choosing factors

Government site:• Find information• Checkout*• Navigation and menus• Search• Readability• Match between system and real world• Aesthetic and minimalist design• Learnability

50

Choosing factors

Airline app:• Select products (flights)• Checkout• Clarity• Visual appeal – aesthetics • Speed• Error prevention• Defaults• Learnability

51

UX FactorsFlight selection, checkout, clarity,

visual appeal, speed

52

MetricsUnderstand the standards

53

Choosing metrics: step 2

Select UX factors These are the key steps in the user journey plus relevant UX elements and heuristics.

Select metricsWe score each UX factor based on the presence or absence of our selected metrics

54

UX Factor + Metrics

Checkout

Ease of passenger details

Range of payment options

Ease of payment Trust / security

55

Our method

A structured expert evaluation that assesses how well websites meet (or exceed) the norms and standards within an industry.

Harmful -1

Absent 0

Present 1

Excellent 2

56

Metrics

57

CLIENT COMP 1 COMP 2 COMP 3 COMP 4 COMP 5

flight selection good poor good good fail good

navigation -1 0 -1 1 1 0

changes/error recovery 1 0 0 1 -1 1

comparing options - price 1 1 1 -1 1 1

comparing options - schedule 2 -1 1 0 -1 0

comparing options - sorting 0 -1 2 1 -1 1

clarity good good fail good poor poor

distinct steps 1 2 -1 -1 1 0

progress indicator 2 0 1 1 0 0

menus 1 1 -1 1 0 -1

clear icons -1 -1 -1 1 1 -1

size/readability 1 0 0 2 0 1

visual appeal good excellent good good poor excellent

design 1 2 1 1 0 1

branding 1 1 1 1 0 2

emotional appeal 0 2 2 1 0 2

consistency 2 0 1 -1 1 1

checkout poor fail* good poor good poor

ease of passenger details 0 1 1 0 1 0

range of payment options 2 2 -1 1 0 0

ease of payment 0 -1 1 1 1 1

trust/security 1 0 1 -1 1 -1

speed good good poor poor good good

number of steps 1 1 0 1 1 1

page load time 0 1 -1 -1 1 1

58

59

Parting questions

What is the appropriate balance between 1/0 ratings and 2/-1 ratings? Objectivity vs. nuance…

What if you want to engage in repeated benchmarking?

What if the norms and standards change over time? What if the goals change?

How can you build this process into a measurement plan?

60

Goal

Context

Metrics

61

@uservision

Abi Reynolds @AbiJreynolds

Jessica Cameron @jessscameron