?

♫

Why keep it to yourself?Getting everyone on the team to do usability testing

Dana ChisnellIA Summit 2010

1

2

Data GutInfluence = +

3

Influence = Data < Gut

4

Influence = Data x Gut

5

Influence = Data^Gut

Throughput is a problem.

6

Influencing design decisions

7

8

Dev

DevI A

IX

D

?

U

♫

D

9

U

?

10

Dev

♫

Dev

IXDD

D

I A

11

Not either/or

12

Do it together

13

U

I

A

D

D

D

DEV

DEV

U

U

14

DEV

DEVIA D

UIX

D

Yay!

15

16

Give up control (you don’t have it, anyway)

Get the team to value data over gut

Research by lots of amateurs produces better results

Otherwise, you’ll burn out

Coach and advise instead

Work on more interesting, harder questions

Influencing design decisions

DEV

DEVIA D

UIX

D

Yay!

Improving the Gut

17

18

Tasks

Moderating

Analysis

19

Tasks: Seeing through the users’ eyes

Help each team member think of a user scenario they care about

Act out the scenario as the user

App: Church social network ✤ Roles:

✤ Senior Pastor

✤ Social Networking Pastor

✤ Volunteer

✤ Lay Leader

✤ Member

✤ First time visitor

✤ Administrative Assistant

20

✤ App: Organizing students to help homebound people vote

✤ User roles:

✤ Student

✤ Voter

✤ Organizer

✤ Trainer

✤ Scheduler

21

22

What happened?

Task scenarios set the scene, give context

You want to create realistic and relatable stories

23

Moderating: Active presence

Role play practice (in dry runs)

Video of the moderator

24

25

Moderating, not training

Impartial, unbiased observing

No teaching!

Listen and watch

Ask open-ended questions: Why? How? What?

Correct and train at the end

Good listenerExcellent memory Develops rapport, empathetic

Quick learner

Who should moderate?

26

Planning, managing data, producing reports

Scientist

Echoing with play-by-playMaximizing flow of information to observers

Sportscaster

Ensuring the safety and comfort of the participant

Flight attendant

27

Joan Dorsey, American Airlines: http://aviationblog.dallasnews.com/archives/aviation-historytrivia/

John Madden, http://www.forbes.com/2008/01/28/television-sports-madden-biz-sports-cx_lh_0128broadcasters_slide_3.html?thisSpeed=15000

Marie Curie, Smithsonian Institution http://photography.si.edu/SearchImage.aspx?t=5&id=3523&q=SIL14-C6-05

Moderators: Run the session

Observers: Note behaviors and quotes

Participant: Do what you normally would do, try to think aloud

28

29

What to look for

Hesitating

Comments

Questions

Body language

Behaviors

Technique

Think aloud - ask:

“Tell me what you’re doing”

“Tell me what you’re thinking”

30

Review at end - ask:

“Walk me through what you did”

“How’d that go?” Use the ballot as a guide for the discussion

“What was confusing or frustrating?”

31

Review

✤ Participant: What was it like?

✤ Observers: What did you see?

✤ Moderators: What questions do you have?

Mid-session reminders

“You are not being tested”

“That’s useful feedback, thank you”

“Please tell me what you’re thinking”

32

Meanwhile, you track time and tasks remaining in the session

Don’t be afraid to move on if something is taking too long

Decide ahead what is low priority or optional

Narration

Don’t be afraid to interact

Say what you are observing - don’t interpret

33

Examples:

“I see you just clicked on the Zit button.”

“Sorry, I didn’t catch that. Would you say that again?”

“That really, really stinks? Could you say what about it stinks?”

Maximizing data

✤ If the participant says “hmmm” or “oops” or “I wonder...”Say: “What questions do you have right now?”

✤ If the participant is silent for 10 - 20 seconds (count!) Say: “What are you thinking?”

✤ If the participant stops because she thinks she’s done or she’s stuck- Summarize what you saw her do - Ask what she will do next

34

Analysis: Structured discussion

Tell stories

KJ - to set priorities

Guess the reason - to exercise inference-making

Observation-inference-opinion-theory

35

KJ Analysis

reach consensus from subjective data

similar to affinity diagramming

invented by Jiro Kawakita

objective, quick

8 simple steps

36

1. Focus questionWhat needs to be fixed in Product X to improve the user experience?(observations, data)

What obstacles do teams face in implementing UE practices? (opinion)

37

2. Organize the groupCall together everyone concerned

For user research, only those who observed

Typically takes an hour

38

3. Put opinions or data on notesFor a usability test, ask for observations

(not inferences, not opinion)

No discussion

39

4. Put notes on a wallRandom

Read others’

Add items

No discussion

40

5. Group similar items In another part of the room

Start with 2 items that seem like they belong together

Place them together, one below the other

Move all stickies

Review and move among groups

Every item in a group

No discussion

41

6. Name each groupUse a different color

Each person gives each group a name

Names must be noun clusters

Split groups

Join groups

Everyone must give every group a name

No discussion

42

7. Vote for the most important groupsDemocratic sharing of opinions

Independent of influence or coercion

Each person writes their top 3

Rank the choices from most to least important

Record votes on the group sticky

No discussion

43

8. Rank the groupsPull notes with votes

Order by the number of votes

Read off the groups

Discuss combining groups

Agreement must be unanimous

Add combined groups’ votes together

Stop at 3-5 clear top priorities

44

45

Observations to directionQuantifying subjective data in 4 easy steps

46

Observations

Sources:

usability testing

user research

sales feedback

support calls and emails

training

47

What you saw

What you heard

Inferences

Judgements

Conclusions

Guesses

Intuition

48

Opinions

Review the inferences

What are the causes?

How likely is this inference to be the cause?

How often did the observation happen?

Are there any patterns in what kinds of users had issues?

49

Direction

What’s the evidence for a design change?

What does the strength of the cause suggest about a solution?

Test theories

50

Resistance from the team

51

I don’t have time

It’s not my job

I don’t know how

What if I screw up

Resistance is futile.

52

Resistance from the team

53

Exchange time in meetings for time with users

It will make you better at your job

I’ll help you learn

You won’t; it doesn’t matter

Stop pouting.

54

Yep, you’re qualified and they aren’t.

Go peddle your fish.

Haven’t you wanted more visibility?

55

A lot of sloppy data is better than a little excellent data

Trust your team to be delegated to

Shift your mindset from your butt to their Gut

Improving the Gut

U

-+ Desktop

D

UsersData

X?

Y?

?

Recommendations

Direction

56

This?

DEV

DEVIA D

UIX

D

Yay!

57

Or this?

Dev I A D DevUIX

D

Why keep it to yourself?

58

Where to learn more

59

Dana’s blog: http://usabilitytestinghowto.blogspot.com/

Download templates, examples, and links to other resources from www.wiley.com/go/usabilitytesting

Dana Chisnelldana@usabilityworks.net

www.usabilityworks.net 415.519.1148

60