Remote, unmoderated testing is as reliable as lab-based testing - and other controversial beliefs about usability testing (Rolf Molich, David Travis, Jakob Biesterfeldt, Steve Krug)

UXPA Conference Panel, London 2014:

"Remote, unmoderated testing is as reliable as lab-based testing" — and other controversial beliefs about usability testing

Rolf MolichDavid TravisJakob BiesterfeldtSteve Krug

Presentation of panelists

Jakob Biesterfeldt

• Managing Director of UserZoom Germany, provider of UX research tools

• 12+ years of UX research experience

• Focus on professionalization of UX research


David Travis

• Managing Director of Userfocus, a London-based UX consultancy

• Author of ‘E-Commerce Usability’

• PhD in Psychology

• Worked as an internal and external usability consultant since 1989

• Logged over 1500 hours in usability tests and trained over 500 students in usability testing


Steve Krug

• Author of Don’t Make Me Think and Rocket Surgery Made Easy

• Usability consultant for 25 years

• Has taught thousands of people to do their own usability testing

• Thinks people rarely change their minds about anything


Rolf Molich

• Owner, manager of DialogDesign,

a small Danish usability consultancy

• Managed the Comparative Usability Evaluations (CUE-1 to 9)

• 30+ years of usability experience

• Focus on Quality and Professionalism in usability work

Please use a few minutes to mark your position on each of the 10 statements

7

Procedure

To keep the discussion lively and cover as many statements as possible, the procedure will be:

► A statement is presented

► Attendees vote for or against

► The supporting panelist speaks for at most 2 minutes

► The opposing panelist speaks for at most 2 minutes

► 1-2 comments from the audiencePlease queue at the microphone. Limit yourself to 30 secs

► Attendees vote again

1Truth or myth?

Remote, unmoderated testing is as reliable as lab-based testing

Agree

Both agree & disagree

Disagree

2Truth or myth?

Five users are enough to catch 85% of the usability problems in practically any product

Agree


Disagree

2Truth or myth?

Five users are enough to catch 85% of the usability problems in practically any product

Agree


Disagree

”The parabola of optimism” (Jared Spool)

12

Real data: CUE studies

CUE = Comparative Usability Evaluation

► 10-17 professional teams test the same website.

► In all CUE-studies, more than 60% of problems were uniquely reported.

► In CUE-4, more than 300 issues were reported.

Five users are enough to drive a

useful iterative cycle

3Truth or myth?

The moderator should say as little as possible while the test participant is working with the test tasks

Agree


Disagree

3Truth or myth?

The moderator should say as little as possible while the test participant is working with the test tasks

Agree


Disagree

4Truth or myth?

Report only the 3-5 most serious problems to the client. Tell them that you would be happy to give them more problems to work on, but ONLY after they had fixed the most serious ones.

Agree


Disagree

5Truth or myth?

It's simple to identify disastrous usability problems

Agree


Disagree

18

Watch two video clips

From Comparative Usability Evaluation 9 (CUE-9)

► Usability test of U-Haul.com

► Participants have been asked to rent a moving truck

► Participants select rental location, then damage coverage

► Look out for potentially disastrous usability problem

6Truth or myth?

Observers should be told to keep quiet if loud sounds from the observation room can be heard in the test room

Agree


Disagree

7Truth or myth?

Usability testing is an iterative process - one-off tests are useless

Agree


Disagree

8Truth or myth?

Users have valuable insight into why they struggle

Agree


Disagree

9Truth or myth?

Showing picture-in-picture of the participant’s face is an unnecessary distraction for observers

Agree


Disagree

10Truth or myth?

If you've found 90 serious or critical problems in a usability test, it's OK to report them all in the usability test report

Agree


Disagree

Evaluation

Did this presentation change some of your views of usability testing?

Yes

Don’t know

No

11

25

Main Takeaways

► As usability professionals we must set a good example for the world. Our products, presentations, doings must be usable and useful: Correct, relevant, brief, constructive

► Five users are enough to drive a useful iterative cycle. Never claim that you can find all usability problems

► Moderators should say as little as possible while the test participant is working with the test tasks in order not to provide inadvertent clues

► The usability of a test report is influenced by the number of problems reported