Eye Tracking the UX of Mobile: What You Need to Know

Eye tracking the UX of mobile: What you need to know

Jen Romano Bergstrom UX Researcher Facebook @romanocog

1

Jon Strohl UX Researcher Fors Marsh Group @jonstrohl

March  23,  2015  ModevUX    |    Arlington  VA  

We will be answering these questions!

2

Ø  Why is mobile important?

Ø  What makes mobile different?

Ø  Do I need to test my design? •  How do I get started? •  What are the benefits?

Ø  Do I need to include eye tracking? •  How do I get started? •  What are the benefits? •  Can you show me how it’s done? •  Are there any examples?

The rise of mobile

3

@romanocog @jonstrohl #modevux

Mobile internet usage continues to increase

4

Mobile retail usage increases

5

@romanocog @jonstrohl #modevux

App usage continues to increase

6

7

What we design for…and reality @romanocog @jonstrohl

#modevux

Krug, S. (2000) Don’t Make Me Think. Pearson Education *NEW 3rd Edition (2014)*

@skrug

8

8

Ø  Controlled  environment  

Ø  All  par>cipants  have  the  same  experience  

Ø  Record  and  communicate  from  control  room  

Ø  Observers  watch  from  control  room  and  provide  addi>onal  probes  (via  moderator)  in  real  >me  

Ø  Incorporate  physiological  measures  (e.g.,  eye  tracking,  EDA)  

Ø  No  travel  costs  

LABORATORY   REMOTE   IN  THE  FIELD  Ø  Par>cipants  tend  to  be  

more  comfortable  in  their  natural  environments  

Ø  Recruit  hard-­‐to-­‐reach  popula>ons  (e.g.,  children,  doctors)  

Ø  Moderator  travels  to  various  loca>ons  

Ø  Bring  equipment  (e.g.,  eye  tracker)  

Ø  Natural  observa>ons  

Ø  Par>cipants  in  their  natural  environments  (e.g.,  home,  work)  

Ø  Use  video  chat  (moderated  sessions)  or  online  programs  (unmoderated)  

Ø  Conduct  many  sessions  quickly  

Ø  Recruit  par>cipants  in  many  loca>ons  (e.g.,  states,  countries)  

> FMG Capabilities

Where to test

9

When to test

Iterative Process Throughout Development

@romanocog @jonstrohl #modevux

10

Research goals and

plan Form

hypotheses Will eye

tracking add value?

Design the study

Planning the test @romanocog @jonstrohl

#modevux

11

Who is involved

SATQ Items

Other materials

Debriefing

Reports

.

. .

.

. .

.

“Make it a spectator sport” @skrug

@romanocog @jonstrohl #modevux

Project Managers

UX Designers

Executives

Engineers

Researchers

User Experience (UX) Defined

12

@romanocog @jonstrohl #modevux

Adapted from Morville, P. Semantic Studios. http://semanticstudios.com/user_experience_design/

A combination of measures are used in UX research.

@romanocog @jonstrohl #modevux

Using multiple measures allows researchers to capture the most detailed and complete picture of the user experience.

14

OBSERVATIONAL First click accuracy

Task accuracy Time on page/task

Selection/click behavior Conversion rate

SELF-REPORT Satisfaction ratings

Difficulty ratings Think-aloud protocol Debriefing interview

UX Measures @romanocog @jonstrohl

#modevux

15 http://blog.thomsonreuters.com/wp-content/uploads/2013/01/smartphones1.jpg

Lots of sizes to consider… @romanocog @jonstrohl

#modevux

16 http://armodilo.com/support/supported-tablets/

Don’t forget about tablets! @romanocog @jonstrohl

#modevux

…our thumbs are only so big

17 http://gizmodo.com/how-to-design-for-thumbs-in-the-era-of-huge-screens-1636955848

…and people use them differently.

18

@romanocog @jonstrohl #modevux

Hoober (2013). UX Matters. http://www.uxmatters.com/mt/archives/2013/02/how-do-users-really-hold-mobile-devices.php

Eye tracking and attention

19

@romanocog @jonstrohl #modevux

@agabojko

Bojko, Aga (2013). Designing the User Experience. Rosenfeld Media

User Experience (UX) Defined

20

Eye tracking helps to

explain how usable,

engaging, and desirable a product is.

@romanocog @jonstrohl #modevux

21

OBSERVATIONAL First click accuracy

Task accuracy Time on page/task

Selection/click behavior Conversion rate

PHYSIOLOGICAL Eye movements

Electrodermal activity (EDA) Pupil dilation

UX Measures @romanocog @jonstrohl

#modevux

SELF-REPORT Satisfaction ratings

Difficulty ratings Think-aloud protocol Debriefing interview

User Experience (UX) Measures •  Self-report metrics tell us why participants focus on

certain site aspects.  

22

•  Eye tracking tells us what, how long, and how often participants focus on design elements.

•  The combination of self-report, observational, and implicit data allows us to better understand perceptions and behavior.  

•  Observational metrics tell us how participants navigate and interact.  

We do not use eye tracking in isolation.

@romanocog @jonstrohl #modevux

23

Eye Tracking Considerations @romanocog @jonstrohl

#modevux

@agabojko

Bojko, Aga (2013). Designing the User Experience. Rosenfeld Media

User Experience Testing + Eye Tracking

24

UX Testing UX Testing with Eye Tracking

Data while completing tasks

Mouse movements, clicks, comments

Scan paths, dwell time, first fixation, re-fixations. Eye-tracking helps determine what led to these behaviors and comments

Findings Usability problems are detected

A more thorough understanding of usability issues and the sources of these issues

Recommendations Quality recommendations based on available data

More insightful and precise recommendations

@romanocog @jonstrohl #modevux

Eye-Tracking Metrics ENGAGEMENT Number of fixations

Total dwell time Percentage of time on

an area

FINDABILITY Time to first fixation Number of fixations prior to first fixation

PROCESSING Fixation

durations

WORKLOAD/EXCITEMENT Pupil dilation

PROCESSING ORDER

Gaze path

COMPREHENSION Repeat fixations

25

26

Data Visualization– Heat & Opacity Maps @romanocog @jonstrohl

#modevux

27

Data Visualization– Gaze Plots @romanocog @jonstrohl

#modevux

28

Modern eye tracking @romanocog @dcaapor

Eye Tracking in the Past

29

Modern eye tracking

Modern Eye Tracking

Mobile Device Stand

30

Glasses

31

Shopping Research

32

Driving Research

33

Design Evaluation Research

34

Moderator & Participant

35 Slide from: Alayoubi, L. & Romano Bergstrom, J. (2014). Paying for Education: User-Centered Research on the Paying for College Website and Tools. Presentation at edUi, Richmond, VA, Sept 2014.

Example Methodology – Formative Evaluation

Eye Tracker

Participants: •  N = 9 •  Mix of gender, ethnicity, income. •  Formative evaluation with a single

web prototype.

User Experience Testing session: •  Participants read a description of

the study. •  The moderator gave instructions

and calibrated the eye tracker. •  Participants completed a set of

tasks provided individually by the moderator.

•  End-of-session satisfaction questionnaire.

•  Debriefing interview.

36

Formative Studies Ø  Are <10 participants appropriate?

•  Very often. Typical for finding large usability problems. •  Problem occurrence is 25% or higher. •  Problem discovery goal is 75% or less.

Ø  Are >20 participants appropriate? •  Not as common, but sometimes used when testing more mature

applications or trying to find smaller, less frequent issues. •  Problem occurrence is 10% or higher. •  Problem discovery goal is 90%.

Ø  If eye tracking, over recruit by ~20% 1.Calculations derived from:

Sauro, J., & Lewis, J. R. (2012). Quantifying the user experience: Practical statistics for user research. Elsevier. Bojko, A. (2013). Eye Tracking the User Experience. Rosenfield Media.

Determining Sample Size1 @romanocog @jonstrohl

#modevux

UX Test with Eye Tracking Demo

37

@romanocog @jonstrohl #modevux

FLEET •  Beta •  Flyfleet.org Features: •  Real-time flight status updates •  Let friends and family track your flight •  Get delay times at check-in and

security •  Find stores and restaurants near you

Fleet was designed and developed by Rob Youmans, Daniel Gartenburg, Melissa Smith, Peter Lee, Jordan Higgins, Octavian, and Vincent Mamo

38

Example Methodology – Summative Evaluation Participants: •  N = 74 | Average Age = 37 •  Mix of gender, ethnicity, income. •  Summative Evaluation with a between

groups design. •  New, Old, Prototype, and Bilingual

media diaries.

User Experience Testing session: •  Participants read a description of the

study. •  The moderator gave instructions and

calibrated the eye tracker. •  Participants completed Steps 1-5 in the

diary at their own pace. •  End-of-session satisfaction questionnaire. •  Debriefing interview.

Eye Tracker

Moderators worked from another room.

Control Room

Slide from: Walton, L., Romano Bergstrom, J., Hawkins, D. & Pierce, C. (2014). User Experience and Eye-Tracking Study: Paper Diary Design Decisions. Paper presentation at the American Association for Public Opinion Research (AAPOR) Conference, Anaheim, CA, May 2014.

39

1.Calculations derived from: Sauro, J., & Lewis, J. R. (2012). Quantifying the user experience: Practical statistics for user research. Elsevier. Bojko, A. (2013). Eye Tracking the User Experience. Rosenfield Media.

Summative Studies Ø  Are <10 participants appropriate?

•  Very rarely, but sometimes when trying to detect very large differences. •  Difference between groups is 60% or more. •  Assuming our confidence goal is 90%. •  Assuming our power is 80%.

Ø  Are >20 participants appropriate? •  Yes, very often. Typical for detecting moderate differences with

confidence. •  Difference between groups is 30% or more. •  Assuming our confidence goal is 90%. •  Assuming our power is 80%.

Ø  If eye tracking, over recruit by ~20%

Determining Sample Size1 @romanocog @jonstrohl

#modevux

Example Methodology – Naturalistic environment

40

Participants: •  N = 8 •  Mix of gender, income. •  Participants use their own

device and Internet connectivity.

User Experience Testing session: •  Sessions occurred in participant’s home. •  The moderator gave instructions. •  Sessions began with an interview about

how participants use the Internet. •  Participants completed a set of tasks

provided individually by the moderator. •  Debriefing interview.

What can we learn from eye tracking?

42

Fixation count heat map of 10 seconds of interaction with the page. 20 participants included.*

Fixation count heat map of 10 seconds of interaction with the page. 10 Novice participants included.*

Fixation count heat map of 10 seconds of interaction with the page. 9 Experienced participants included.*

All Participants Novice Experienced

*Heatmaps include participants who located the link from the Refunds page only. Fixation count refers to the number of times the eye fixates, or directs itself towards, an area of the page.

Novice and experienced users focus on different elements.

43

Making something large does not mean people will notice it.

@romanocog @jonstrohl #modevux

44

New/Old DVR Grid New/Old DVR Grid Prototype Grid

Grids that match users’ mental model are easier to process.

Walton, L., Romano Bergstrom, J., Hawkins, D., & Pierce, C. (2014). User experience testing and eye tracking inform paper diary design. Proceedings from the Human Computer Interaction International Conference, June 2014, Crete, Greece.

@romanocog @jonstrohl #modevux

45

Gaze plot of 60 seconds of engagement from four participants on the Loan Rehabilitation page who scanned the text without reading it thoroughly.

Gaze plot of 60 seconds of engagement from four participants on the Contact Your Servicer page who read most of the text.

People do not read dense text.

@romanocog @jonstrohl #modevux

46

The text within the orange circle indicates the information to enter on the page. The red circled area is where the user enters the ID number. The participant skips the instructions and introductory text and moves on to the actionable areas.

People do not read instructions.

He, J., Siu, C., Chaparro, B. & Strohl, J. (2014). Mobile. In J. Romano Bergstrom & A. Schall (Eds.), Eye Tracking in User Experience Design. San Francisco, CA: Morgan Kaufmann.

@romanocog @jonstrohl #modevux

47

If the mandatory fields were not completed, participants received an error message that was not helpful. This gaze plot begins after the participant clicks “Ok.” The participant looks all over the screen, searching for the missing mandatory fields. The error message did not indicate which fields were missing.

People need to know what to do next

He, J., Siu, C., Chaparro, B. & Strohl, J. (2014). Mobile. In J. Romano Bergstrom & A. Schall (Eds.), Eye Tracking in User Experience Design. San Francisco, CA: Morgan Kaufmann.

@romanocog @jonstrohl #modevux

Activity

48

Awesome! I want to use eye tracking. What’s next?

50

Eye-Tracking Decision Progression

Research goals and plan Form hypotheses Will eye tracking

add value? Design the study

Collect data Clean & prepare data

Visualize qualitative data

Analyze quantitative data

51

Evidence: Younger and middle-age social media users access the sites/apps largely through smartphones, whereas older users access largely through desktop computers. It is important to test across user groups and across devices.  

Older Adults Middle Age Adults Younger Adults

People  fixate  elements  of  displays  differently  across  devices.  Different user groups access via different interfaces

@romanocog @jonstrohl #modevux

52

Android App Homepage Tablet Web Homepage Tablet App Homepage

People  fixate  elements  of  displays  differently  across  devices.  People attend to elements of displays differently across devices

Bristol, K., Romano Bergstrom, J. & Link, M. (2014). Eye Tracking the User Experience of a Smartphone and Web Data Collection Tool. Paper presentation at the AAPOR Conference, Anaheim, CA, May 2014.

@romanocog @jonstrohl #modevux

Thank you!

Jen Romano Bergstrom UX Researcher Facebook @romanocog JenRB@fb.com

Jon Strohl UX Researcher Fors Marsh Group @jonstrohl jstrohl@forsmarshgroup.com

March  23,  2015  ModevUX    |    Arlington  VA  

54

Website  www.forsmarshgroup.com      LinkedIn  hXp://www.linkedin.com/company/fors-­‐marsh-­‐group        Blog    www.forsmarshgroup.com/index.php/blog      TwiXer  @forsmarshgroup