73
Getting Started with Usability Testing Instructors: Thyra Rauch, John Schrag, and Carol Smith UXPA 2016

Usability Testing Basics: Remote and In-Person Studies

Embed Size (px)

Citation preview

Page 1: Usability Testing Basics: Remote and In-Person Studies

Getting Started with Usability TestingInstructors: Thyra Rauch, John Schrag, and Carol SmithUXPA 2016

Page 2: Usability Testing Basics: Remote and In-Person Studies

Schedule

05:30 Introductions, Agenda, Materials 05:40 Measurements & Testing 06:05 Activity I 6:25 Planning Test Logistics 6:40 Activity II 7:00 Selecting & Preparing Materials 7:10 Interacting with Test Participants 7:30 Break 8:00 Activity III 8:20 Testing Experiences 8:25 Activity IV and Team Presentations 9:10 Team Presentations 9:25 Final Q&A and Wrap-Up 9:30 Adjourn

2

Page 3: Usability Testing Basics: Remote and In-Person Studies

Introduction

Instructor introductions

Audience experience

3http://creativecommons.org/licenses/by-sa/2.0/http://www.flickr.com/photos/raphaelquinet/513351385/sizes/l/in/photostream/http://www.flickr.com/photos/raphaelquinet/

Page 4: Usability Testing Basics: Remote and In-Person Studies

Why Test?

You are not your user May miss details; too close to design Unforeseen requirements

Validate understanding of tasks and context

More satisfied users Less training needed Fewer services calls

4

Page 5: Usability Testing Basics: Remote and In-Person Studies

Myths of Usability Testing

Need a labTesting occurs at the end of the dev cycle

Must test everythingNeed to test 100 peopleA “good” test means no changesUsability is “system testing”, “user-acceptance testing”, etc.

Need a large test team, each playing a specific role/part

5

Page 6: Usability Testing Basics: Remote and In-Person Studies

Just Do It!

Make it realisticAny locationAny stage of development (earlier is better)

Start small Test an idea (not a feature) Just a few users

Anytime (un-moderated)

6

Page 7: Usability Testing Basics: Remote and In-Person Studies

Getting Started

7

Page 8: Usability Testing Basics: Remote and In-Person Studies

Plan and Prepare for the Test

• Scope the effort–Budget, resources, goals, time–Put together a test team

•Design the test–Goals, participants, tasks, measures, methodology, team roles etc.–Summative or formative?

8

Page 9: Usability Testing Basics: Remote and In-Person Studies

Testing Plan

Goals What testing and how to measure success

Test protocol – remote or in person?

User groups and participants

Recruiting/incentives Scenarios

Share with team Keep concise

9

Page 10: Usability Testing Basics: Remote and In-Person Studies

Create Test Scenarios

Representative of typical tasksDescribe what, not how

“You want to buy a shirt for your father. Find a man’s blue dress shirt for under $40.”

“As a systems administrator for your company, you want to install, configure, and test the xyz application.”

Avoid leading wording (e.g., actions)

10

Page 11: Usability Testing Basics: Remote and In-Person Studies

Measurements

Success Time on task % of tasks completed/not completed

Number of steps to accomplish task

Learning time Number of errors Number of times help consulted

Satisfaction

11Photo By Kenyaboy7Andrew Chipley http://www.flickr.com/photos/kenyaboy7/3749535540/

Page 12: Usability Testing Basics: Remote and In-Person Studies

Facilitator Role

Conducts test and manages time and frustration

Communicates with participant (before, during, after)

Prompts for more informationEnsures legal requirements are met (NDAs, recording permissions)

12

Page 13: Usability Testing Basics: Remote and In-Person Studies

Additional Roles

Observer(s)* Takes careful notes Observations (notes inferences as well)

Exact quotes Context

“Computer” (paper prototyping, wizard of oz)

Mimics actions of computer

13*See guidance for Observers in Resources section

Page 14: Usability Testing Basics: Remote and In-Person Studies

Testing Protocols

Think-aloudActive interventionCo-discoveryRetrospective

14

Page 15: Usability Testing Basics: Remote and In-Person Studies

Activity I

15

Page 16: Usability Testing Basics: Remote and In-Person Studies

Basic Usability Study

Select item to testDiscover stakeholder’s needsHigh level goalsMeasurements to be madeHigh level scenario (sentence or two)

16

Page 17: Usability Testing Basics: Remote and In-Person Studies

Planning Test Logistics

17

Page 18: Usability Testing Basics: Remote and In-Person Studies

Recruit Participants

Use a participant screening profileTwo weeks to complete recruit (average)

Recruiters Marketing or product marketing department

Professional recruitment agency Members of your team or yourself

18

Page 19: Usability Testing Basics: Remote and In-Person Studies

Run a Pilot Study

Discover problems with study or concept being tested

Estimate time needed for the testRefine script and tasks for testVerify if tasks are typical (is it something users actually do?)

Practice before going live with participants

Get new ideas for follow-on questions or things to observe

19

Page 20: Usability Testing Basics: Remote and In-Person Studies

Do I need a lab?

20Rubin, Jeffrey. Handbook of Usability Testing. John Wiley & Sons, Inc.; 1994.

Computer / Concept

FacilitatorParticipant

Observer

Timer Logger

Page 21: Usability Testing Basics: Remote and In-Person Studies

Portable Labs

Temporary setup Capture screen, video and audio

Complicates observation

Anywhere (conference room, remotely)

21Photo by Roebot at http://www.flickr.com/photos/roebot/2964156413/

Page 22: Usability Testing Basics: Remote and In-Person Studies

Remote Testing

Moderated and AutomatedCapture screen, video and audio (not all automated collect this)

Moderated: Connect to both computer and phone

Technology Bandwidth, operating system, browser Test setup and connections ahead of time

22

Page 23: Usability Testing Basics: Remote and In-Person Studies

Activity II

23

Page 24: Usability Testing Basics: Remote and In-Person Studies

Recruiting Profile

Key elements Mimic population using product

Critical factors that make a difference Age Experience

How they will be screened Phone Online Follow-up and scheduling

24

Page 25: Usability Testing Basics: Remote and In-Person Studies

Recruiting Profile

Samples Why decisions made What going after

Create recruiting profile for participants for usability study designed in Activity 1

25

Page 26: Usability Testing Basics: Remote and In-Person Studies

Selecting & Preparing Materials

26

Page 27: Usability Testing Basics: Remote and In-Person Studies

Why Use a Script?

Promotes consistency in conducting the test

Reminds to welcome and thank peopleReminds to reset/configure between participants

Ensures NDAs and other legal requirements are not missed

Prepare for non-native language speakers

Tell me more…

27

Page 28: Usability Testing Basics: Remote and In-Person Studies

Parts of a Script

Welcome to participantsSteps in test (forms, tutorials, tasks, questions)

Thank you to participants (and incentives if any)

Notes to yourself Reset/configuration prompts

28

Page 29: Usability Testing Basics: Remote and In-Person Studies

Test Folders

Organize materials per participant

Easily access individual results

Store as a groupPrepare a folder for each participant

All test materials(script, forms, printed tasks etc.)

Don’t use names- label folders: Pilot, P1, P2, P3

29

Page 30: Usability Testing Basics: Remote and In-Person Studies

Informed Consent

Your responsibility to inform participant

Every time Ethical and legal responsibility Do no harm (physical, emotional)

Explain consent form* Plain language – what, why, stop anytime, etc.

Participant signs (Google form remotely)Rarely lose participants over form

30*Examples in Appendix

Page 31: Usability Testing Basics: Remote and In-Person Studies

Interacting With Test Participants

31

Page 32: Usability Testing Basics: Remote and In-Person Studies

Welcome

Make sure the participant feels welcomed and appreciated.

Stress that you are not testing them, rather you are testing…

Encourage feedback, positive or negative

“I was not involved in the design of this so you can’t hurt my feelings”

32

Page 33: Usability Testing Basics: Remote and In-Person Studies

Facilitating

Follow your scriptFrame questions in unbiased mannerRemain passive in facial expressions and body language as much as possible while participant is working

Don’t lead or give clues (unless frustration is clear and evident)

Probe don’t demandGently prompt during think-aloud protocol if necessary

33

Page 34: Usability Testing Basics: Remote and In-Person Studies

Probes

Ask open-ended questionsEncourage participant to elaborateUse participant’s wordsListen for vocalizations and watch non-verbal gestures

Have participant tell you when they are done

Give them time to think

34

Page 35: Usability Testing Basics: Remote and In-Person Studies

Example Phrases

How does this compare with your expectations?

What specifically are you referring to?What are you looking for?Tell me more…Can you give me an example?

35

Page 36: Usability Testing Basics: Remote and In-Person Studies

Common Issues

Refusal to critique Ask preference and to compare interfaces Watch what they DO, not what they SAY

Taking control - diverting to their issue Let them see you are listening and writing it down

If no resolution, thank and end session

- give them the incentive.

36

Page 37: Usability Testing Basics: Remote and In-Person Studies

Getting Users Unstuck

It’s not a question of “if” but “when” Users might be embarrassed at what they perceive to be their fault

Users may not be able to articulate what they don’t understand

Don’t directly answer initial questions

37

Page 38: Usability Testing Basics: Remote and In-Person Studies

Getting Users Unstuck

Progressive assistance What are you trying to do right now? What do you think the next step is? What would you do in this situation if you were at work?

Hints Do you see anything that might help you? (small hint)

Have you checked the Help? (medium hint)

What do you think the xxx button does? (large hint)

End before frustrated, or take a break

38

Page 39: Usability Testing Basics: Remote and In-Person Studies

Debriefing

Verbal and written questions What did you like best? What was the worst thing? What can be improved? Describe in 3 words Follow up to specific items noted during testing

Ratings Ease of use Willingness to use

Thank participant

39

Page 40: Usability Testing Basics: Remote and In-Person Studies

Activity III

40

Page 41: Usability Testing Basics: Remote and In-Person Studies

Probes – Group Exercise

Are you thinking of using the search feature?

Are you trying to find the price of the book?

41

Page 42: Usability Testing Basics: Remote and In-Person Studies

Alternatives

Are you thinking of using the search feature? (biased)

Can you tell me what you are thinking right now? (unbiased)

Are you trying to find the price of the book? (biased)

What are you trying to do? (unbiased)

42

Page 43: Usability Testing Basics: Remote and In-Person Studies

Rational

Can’t know what participant is thinking

Keep questions brief and clearAvoid

Asking participants to predict the future Putting “words in their mouth” Showing bias or hinting at your opinions

43

Page 44: Usability Testing Basics: Remote and In-Person Studies

Probes – Team Exercises

Break into teamsEach get a set of biased probesGenerate unbiased alternativesPresent/discuss with entire group

44

Page 45: Usability Testing Basics: Remote and In-Person Studies

Testing Experiences

45

Page 46: Usability Testing Basics: Remote and In-Person Studies

Low-fidelity/Paper Prototypes

Includes wireframesGood time to get initial feedback on concept, word choices and placement

Participants May not be able to understand May not be act the same as online

Always test again later in development

46

Page 47: Usability Testing Basics: Remote and In-Person Studies

High Fidelity Prototypes

Ideal time for testing – early enough in development, but participants can “kick the tires”

May be difficult to test with (broken links, undeveloped material, etc.)

47

Page 48: Usability Testing Basics: Remote and In-Person Studies

Functioning Products

Great for redesign projects – prior to start

Get baseline and goals for improvements

Reduce task sequence from 10 clicks to 5 Reduce time to perform task from 10 min to less than 3 min

Increase satisfaction from 4 to 5 on a 5-pt scale

Use existing data (Analytics, Help Desk, etc.) to create tasks and inform study

48

Page 49: Usability Testing Basics: Remote and In-Person Studies

Reporting

What was tested, how and whySuccess and Failure and whyRecommendationsFuture testing – benchmarking Just enough to communicate – short & sweet

49

Page 50: Usability Testing Basics: Remote and In-Person Studies

Activity IV

50

Page 51: Usability Testing Basics: Remote and In-Person Studies

Activity 4

Mock up (15 minutes)Round 1 of testing (5 minutes)Revise mock ups (10 minutes)Round 2 of testing (5 minutes)Prepare team presentation (10 minutes)

51

Page 52: Usability Testing Basics: Remote and In-Person Studies

Hotel Reservation Kiosk

Design and develop a new hotel check-in kiosk that can be installed in airports

Hotel chain wants to make it easy for their customers:

Check into their room at the airport Request a particular room type, if available

Request transportation to the hotel

52

Page 53: Usability Testing Basics: Remote and In-Person Studies

Ideal Scenario

During your search for a local phone book, you notice a hotel chain’s digital Kiosk, and you decide to try it.

You need to: Reserve a Hotel Room for the evening. (A room with 1 bed for 2 people, non-smoking)

Request transportation to the Hotel Reserve transportation from the hotel back to the airport

53

Page 54: Usability Testing Basics: Remote and In-Person Studies

Alternative Scenario

You want to go to the hotel right away to change before going to a fancy restaurant downtown.

You need to: Know if the hotel provides transportation. Reserve a room with a King-Size bed because you're fat and you want a view of the ocean.

Make sure that your “friend” who is coming tomorrow will have a room available next to yours.

54

Page 55: Usability Testing Basics: Remote and In-Person Studies

Team Presentations

55

Page 56: Usability Testing Basics: Remote and In-Person Studies

Final Q&A Questions?Handouts

56

Page 57: Usability Testing Basics: Remote and In-Person Studies

Survey Please complete the session survey:www.uxpa2016.org/survey/69

57

Page 58: Usability Testing Basics: Remote and In-Person Studies

Contact Us

Thyra Rauch IBM [email protected]

+1 (408) 463-2465

Carol Smith [email protected]

+1 (412) 212-8737@carologic

58

John SchragAutodesk [email protected]

@jvschrag

Page 59: Usability Testing Basics: Remote and In-Person Studies

Resources

59

Page 60: Usability Testing Basics: Remote and In-Person Studies

Usability by the numbers…

80% of SW life cycle costs occur during maintenance

80% of maintenance costs are due to unmet or unforeseen user requirements

Only 20% are bugs or reliability problemsTypical software program released with 40 usability design problems

Over 70% of CIO's state that one of their biggest problems is communicating with users to understand their needs

60

Page 61: Usability Testing Basics: Remote and In-Person Studies

Number of Participants

• Contentious issue• Nielsen - 5-7

participants uncover ~80% of major usability problems • IF representative • IF doing

representative tasks. • Testing more =

significant diminishing return.

• Virzi and Carol Barnum • Small number of

users is sufficient.

61http://www.useit.com/alertbox/20000319.htmlJakob Nielsen’s Alertbox. Why You Only Need to Test with 5 Users. March 19, 2000.

Number of Test Participants

Page 62: Usability Testing Basics: Remote and In-Person Studies

Recruitees

Could be Customers General consumers Members of organization or company

Difficulty of recruiting Specificity of screener Availability/interest in study (doctors vs. consumers)

Experience (highly experienced vs. novice)

62

Page 63: Usability Testing Basics: Remote and In-Person Studies

Recruiting Considerations

Accessibility of test location Include people with disabilities “We are all only temporarily able-bodied. Accessibility is good for us all.”

Language interpretersRemote and on-site studies

Technology availability Administrative rights Internet connectivity

63Quote by @mollydotcom at #stirtrek 2011 via @carologic

Page 64: Usability Testing Basics: Remote and In-Person Studies

Data Collection Forms

Customize for each studySpace for writing notes – each scenario spelled out.

Include space for tickmarks regarding easily measurable activities.

E.g. Participant accesses video clip E.g. Participant selects advertisement

64

Page 65: Usability Testing Basics: Remote and In-Person Studies

Tool Considerations

In person or remote? Lab or on-site?Prototype limitations (can it be online?, is it a document or a clickable site?)

Number of observers, number of participants?

Number of facilitators?Logging and video editing needs (time on task, highlight video creation)?

Surveys before or after?Eye tracking?

65

Page 66: Usability Testing Basics: Remote and In-Person Studies

Usability Testing Software

Morae OvoSilverBack (Mac only)Loop11 (Remote only)Tobii (Eye-tracker)SMI (Eye-tracker)SurveyMonkey

66

Page 67: Usability Testing Basics: Remote and In-Person Studies

Screen Sharing Software

GoToMeeting – http://www.gotomeeting.com

Lotus Sametime Unyte – http://www.unyte.com

YuuGuu -- http://www.yuuguu.com WebEx – http://www.webex.comYugma -- https://www.yugma.com/

Trouble Shooting: CoPilot - https://www.copilot.com/

67

Page 68: Usability Testing Basics: Remote and In-Person Studies

Satisfaction Questionnaires

Standard Usability Measurement Inventory (SUMI)

office/desktop software, purchase 50 questions

Website Analysis and MeasureMent Inventory (WHAMMI)

Purchase 20 questions

System Usability Scale (SUS) Free 10 questions

68

Page 69: Usability Testing Basics: Remote and In-Person Studies

Ethics Resources

Treatment of human subjects Web-based course Gives certificate of completion http://cme.cancer.gov/clinicaltrials/learning/humanparticipant-protections.asp

Guidelines for writing informed consent

http://ohsr.od.nih.gov/info/sheet6.htmlUXPA Code of Conduct

69

Page 70: Usability Testing Basics: Remote and In-Person Studies

NDA’s

Do I need a non-disclosure agreement (NDA)?

Maybe Legal department (be friendly) One-way, two-way*

AKA Confidentiality agreement (CA) Confidential disclosure agreement (CDA) Proprietary information agreement (PIA) Secrecy agreement (SA)

70

Page 71: Usability Testing Basics: Remote and In-Person Studies

Activity ______________________________________Who are you observing? (e.g. SME in soft-goods manufacturing, application admin)

What is the situation? ___________________________(e.g. phone interview, their work at their desk in person, remote usability study, etc.)

Date: __________________Time: __________________Location: ________________Note-Taker: ______________

People (P)Who are you observing? Who else is involved in the experience?

Objects (O) What objects do they interact with? What apps? Phone, etc.?

Environment (E) Characteristics of the setting, number/types of interruptions

Messages (M)What is communicated? How is it transmitted?

Services (S)What services are available to them?

Watch for these types of observations and put additional Questions/Ideas on the back. Content in this document described by Vijay Kumar in his book: 101 Design Methods

Physical What do they interact with?

CognitiveHow associate meanings? How learn?

SocialInteractions, decision making, scheduling, work?

CulturalWhat are shared norms, habits, values?

EmotionalWhat emotions are expressed and how?

What are your biggest takeaways?What did you learn? What surprised you?What is the participants need?

Interview & Observation Notetaking Form

Page 72: Usability Testing Basics: Remote and In-Person Studies

Observer RulesEveryone who observes a design session is asked to abide by a set of rules. The purpose of these rules is to minimize stress for the participants and to maximize the amount of information we get from the study.

Stay for the entire study Distractions are unhelpful and participants may get the impression that you’re leaving

because they’ve done something wrong (e.g. walking out in middle of a movie). If you can attend only part of a study, discuss with the facilitator beforehand to determine whether there is a way to accommodate this.

Don’t reveal information about the study to the participant It is often more useful to explore an area of difficulty in detail rather than try to “get

through” all the topics. The facilitator will track time so that we can cover as many of the important areas as possible.

Respect participants and the confidentiality of their data We have promised the participants that their participation is confidential. This means

that we should not include their names in any reports or other communication such as email, and we should refrain from discussing them by name outside the test setting.

Do not make negative comments about people—there is always a risk that a derogatory comment could be overheard or otherwise make its way back to the user.

Adapted from the book Paper Prototyping by Carolyn Snyder, published by Morgan Kaufmann Publishers. Copyright (c) 2003 Elsevier. All rights reserved.

Page 73: Usability Testing Basics: Remote and In-Person Studies

Observer Rules (continued) Remain silent and silence phones You may notice something so surprising that you are tempted to laugh or exclaim out loud.

This is not unusual. Unfortunately, participants might think you are laughing at them. Keep as quiet as possible. You will have opportunities to ask questions at the end. If necessary, pass a note to the facilitator.

Observe only - no questions or support If reviewing an interface, it’s likely participants will have problems, and it is normal to feel a

temptation to help. Please don’t. Instead, try to understand why it was that the user got stuck or went down the wrong path. It’s the facilitator’s role to get users back on track if they get really stuck. And if the facilitator poses a question during the test, he or she is asking the users, not you—please don’t answer unless the facilitator specifically directs a question to you.

Avoid “Design Questions” - Questions that ask the user their opinions about how to design aspects of the application (such as, “Where would you like to see these navigation buttons?”) can take a lot of time to answer and produce only limited results. Instead, focus on trying to understand the problem—we’ll come up with solutions later, outside the test.

In person, be conscious of your body language Although most studies are interesting, not every moment will be fascinating. If something is

happening that isn’t of interest to you but may be to others, sit quietly without fidgeting. Take notes to stay alert.

Adapted from the book Paper Prototyping by Carolyn Snyder, published by Morgan Kaufmann Publishers. Copyright (c) 2003 Elsevier. All rights reserved.