Quick Cheap Insightful: Usability testing in the wild

The quick, the cheap, and the insightful

Usability testing in the wild

Dana ChisnellSan Francisco STC November 2008

1

What is a usability test?

2

3

http://www.sigchi.org/chi97/proceedings/overview/tst.htm

4

5

Quick, cheap, insightful

✤ Minimal steps to focus on

✤ Where the value comes from in doing usability tests

✤ Where it may be risky to go minimalist

✤ How to think about the trade-offs

✤ What’s essential

✤ What might be nice to have

6

Minimal steps

Classic, with everything

7

✤ Chapter 5. Develop a test plan

✤ Chapter 6. Choose a testing environment

✤ Chapter 7. Find and select participants

✤ Chapter 8. Prepare test materials

✤ Chapter 9. Conduct the sessions

✤ Chapter 10. Debrief with participants and observers

✤ Chapter 11. Analyze data and observations

✤ Chapter 12. Create findings and recommendations

8

Photo courtesy of Tom Tullis

9

Classic, with everything

Can take a lot of time: weeks or months.

What if you have to do something quick? and cheap?

10

You don’t have to do it by the book.

11

What’s essential?

Develop a test plan

Choose a testing environment

Find and select participants

Prepare test materials

Conduct the sessions

Debrief with participants and observers

Analyze data and observations

Create findings and recommendations

12

What’s essential?

Develop a test plan

Choose a testing environment

Find and select participants

Prepare test materials

Conduct the sessions

Debrief with participants and observers

Analyze data and observations

Create findings and recommendations

13

What’s essential?

Develop a test plan

Find participants

Conduct the sessions

Debrief with observers

14

What’s essential?

Develop a test plan

Find participants

Conduct the sessions

Debrief with observers

15

and decide together

Sit next to someone. Watch them do stuff.

16

Story:

17

California.

18

1 day

19

a ballot and instructions

20

a ballot and instructions

21

random participants

22

no helpers

23

results due: today

24

What to do?

25

Improvise: Ground rules

26

✤ Say “yes.”

✤ Trust your team and your participants

✤ Don’t let the team or stakeholders block

✤ Work to the top of your collective intelligence

27

How we got value

✤ Used the ballots and instructions at hand

✤ Focused on one thing at a time

✤ Had participants generate and collect data

✤ Drafted observers and debriefed

28

More examples

29

Cafe

30

Company lobby

31

Trade show

http://www.flickr.com/photos/dans180/72408664/

32

Listen in on

customer service

33

Transit seat

mates

34

Remote testing

Builds a storyShows organizing strategies

Email trails

Shows deliberate returns to sites

Bookmarks & favorites

Gives a recent snapshot of activity

Browser history

35

Frames the conversations users are having

User forums & wikis

Proxies for buyers, end users

Sales reps

View on persistent, difficult problems

Customer service logs

36

37

Value

Supporting usable design

Each phase includes input from users

Multidisciplinary teams

Enlightened management

Willingness to learn as you go

Defined usability goals and objectives

38

Supporting usable design

Each phase includes input from users

Multidisciplinary teams

Enlightened management

Willingness to learn as you go

Defined usability goals and objectives

39

Where’s the value in testing?

70% watching someone use the design

20% working with the team to prepare to test

8% discussing what happened

40

41

Getting value in the wild

✤ Use what’s at hand

✤ Narrow the scope of the test

✤ Focus on

✤ what can make the most difference to the most users

✤ what can be implemented easily with the resources available

When is testing in the wild not valuable?

42

✤ If you need

✤ summative data

✤ benchmarks

✤ answers to hard problems

43

Risks

Risks of testing in the wild

✤ Participant sample may be too small, biased

✤ Inconsistent approach may net inconsistent data

44

What do you lose?

✤ Quantitative data

✤ Rigor

✤ Relatively unbiased sample, maybe

45

http://drb.lifestreamcenter.net/Lessons/process_maps/

46

Trade offs

What do you need?

✤ To inform a design:

47

Testing in the wild Classic usability testingQualitative data Quantitative dataOpportunity PlanningFits the schedule Don’t know how to fit UT into

the scheduleJust in time After the fact$ $$$$Something Maybe nothing

✤ Qualitative data

✤ Opportunity

✤ Fitting into a schedule

✤ Timeliness

48

Where’s the ROI?

49

So far.You don’t have to do it by the book

Value of usability testing

+ observing users

+ working with the team

Tradeoff: having some data over having no data

50

Coming up.What’s the bare minimum

Steps for testing in the wild

What to add if you have resources

51

Essentials

What you need

✤ Someone who will try the design

✤ Somewhere to test

✤ Something to study

52

Something(Activity)

Some place(Context)

Somebody(Human)

Follow these steps:

53

1. Plan, minimally

2. Get the team on board

3. Design the test, minimally

4. Recruit participants

5. Conduct sessions

6. Debrief and decide

Plan, minimally✤ What

✤ Why

✤ Who

✤ When

✤ Where

54

Plan, minimally: Example

✤ What: near-final design

✤ Why: inform user training and support

✤ Who: inexperienced customers

✤ When: the end of the week

✤ Where: trade show at user group meeting

55

Minimalist plan

56

Goals and objectives

Participant characteristics

Description of method

List of tasks

Find out whether information about admission is easy to find and use

People who have college-bound kids

Sit-by parents attending a high school basketball game, each trying three university sites with the same scavenger hunt task

Find out how and when applications are dueDetermine whether there are fees for applyingLearn when acceptances will be sent

Get the team on board✤ Visualize the desired user

experience

✤ Share the intellectual property of observing users

57

Design the test, minimally✤ Why are you testing?

✤ What questions are you trying to answer?

✤ What constrains the design?

✤ What are you going to do with what you find out?

58

Recruit participants✤ Convenience sample

✤ Requirements, not demographics

59

Sources of participants

60

✤ Staff not on the design project

✤ Friends and family

✤ Personal, professional networks

✤ Online social networks

✤ Community organizations

✤ Online classifieds

✤ Association, society, user group, union contacts

✤ Temp agencies

Conduct sessions✤ Rehearse (P1)

✤ Interview-based tasks (or based on previous field work)

✤ Explore in short, focused sessions

✤ Iterate test design and product design

61

Session outline

62

Greet the participant

Explain the study, your role, and their role

Interview (maybe)

Do tasks from interview

Debrief with participant

Debrief with observers

Moderating, not training

63

✤ Impartial, unbiased observing

✤ No teaching!

✤ Listen and watch

✤ Ask open-ended questions: Why? How? What?

Debrief and decide✤ Write up issues on sticky notes

and sort them into priority lists

✤ Ask for top 10 items -- base on data and observations, not opinion

✤ Take a vote for priorities

64

65

Nice to have

Add these ingredients

66

✤ Screened, scheduled participants

✤ Official paperwork

✤ Recordings

✤ More observers

Participants

67

What Why

appropriate experience

scheduled ahead of time

greater confidence in data

easier to get data

Paperwork

68

What Why

script

consent forms

recording waivers

pre-test questionnaire

post-test questionnaire

ensure consistent instructions and moderating

official acknowledgment for taking part

permission for recordings

experience, knowledge, value

feedback on tasks, UI

Recordings

69

What Why

audio

video or photos

Morae, Camtasia, other

transcripts, verbal protocol analysis

stories, double-check, highlights

digital data, automated collection

More observers

✤ Better-informed design recommendations

70

What Why

developers

management, execs

technology boundaries

business priorities

71

Nutshell

http://www.flickr.com/photos/stoan/145497333/

Take away:

72

✤ Value: observing, planning

✤ Plan: 4 steps

✤ Recruit: behavior

✤ Moderating: listen, observe

73

Quick.

You can compensate for some shortcomings, and just test more

Cheap.

Low risk, little money

Insightful.

Value comes from getting as close as possible to what real people are doing with your design

Where to learn more

74

Dana’s blog: http://usabilitytestinghowto.blogspot.com/

Download templates, examples, and links to other resources from www.wiley.com/go/usabilitytesting

Me

Dana Chisnelldana@usabilityworks.net

www.usabilityworks.net 415.519.1148

75