Upload
uxpa-international
View
451
Download
1
Embed Size (px)
Citation preview
Getting Started with Usability TestingInstructors: Thyra Rauch, John Schrag, and Carol SmithUXPA 2016
Schedule
05:30 Introductions, Agenda, Materials 05:40 Measurements & Testing 06:05 Activity I 6:25 Planning Test Logistics 6:40 Activity II 7:00 Selecting & Preparing Materials 7:10 Interacting with Test Participants 7:30 Break 8:00 Activity III 8:20 Testing Experiences 8:25 Activity IV and Team Presentations 9:10 Team Presentations 9:25 Final Q&A and Wrap-Up 9:30 Adjourn
2
Introduction
Instructor introductions
Audience experience
3http://creativecommons.org/licenses/by-sa/2.0/http://www.flickr.com/photos/raphaelquinet/513351385/sizes/l/in/photostream/http://www.flickr.com/photos/raphaelquinet/
Why Test?
You are not your user May miss details; too close to design Unforeseen requirements
Validate understanding of tasks and context
More satisfied users Less training needed Fewer services calls
4
Myths of Usability Testing
Need a labTesting occurs at the end of the dev cycle
Must test everythingNeed to test 100 peopleA “good” test means no changesUsability is “system testing”, “user-acceptance testing”, etc.
Need a large test team, each playing a specific role/part
5
Just Do It!
Make it realisticAny locationAny stage of development (earlier is better)
Start small Test an idea (not a feature) Just a few users
Anytime (un-moderated)
6
Getting Started
7
Plan and Prepare for the Test
• Scope the effort–Budget, resources, goals, time–Put together a test team
•Design the test–Goals, participants, tasks, measures, methodology, team roles etc.–Summative or formative?
8
Testing Plan
Goals What testing and how to measure success
Test protocol – remote or in person?
User groups and participants
Recruiting/incentives Scenarios
Share with team Keep concise
9
Create Test Scenarios
Representative of typical tasksDescribe what, not how
“You want to buy a shirt for your father. Find a man’s blue dress shirt for under $40.”
“As a systems administrator for your company, you want to install, configure, and test the xyz application.”
Avoid leading wording (e.g., actions)
10
Measurements
Success Time on task % of tasks completed/not completed
Number of steps to accomplish task
Learning time Number of errors Number of times help consulted
Satisfaction
11Photo By Kenyaboy7Andrew Chipley http://www.flickr.com/photos/kenyaboy7/3749535540/
Facilitator Role
Conducts test and manages time and frustration
Communicates with participant (before, during, after)
Prompts for more informationEnsures legal requirements are met (NDAs, recording permissions)
12
Additional Roles
Observer(s)* Takes careful notes Observations (notes inferences as well)
Exact quotes Context
“Computer” (paper prototyping, wizard of oz)
Mimics actions of computer
13*See guidance for Observers in Resources section
Testing Protocols
Think-aloudActive interventionCo-discoveryRetrospective
14
Activity I
15
Basic Usability Study
Select item to testDiscover stakeholder’s needsHigh level goalsMeasurements to be madeHigh level scenario (sentence or two)
16
Planning Test Logistics
17
Recruit Participants
Use a participant screening profileTwo weeks to complete recruit (average)
Recruiters Marketing or product marketing department
Professional recruitment agency Members of your team or yourself
18
Run a Pilot Study
Discover problems with study or concept being tested
Estimate time needed for the testRefine script and tasks for testVerify if tasks are typical (is it something users actually do?)
Practice before going live with participants
Get new ideas for follow-on questions or things to observe
19
Do I need a lab?
20Rubin, Jeffrey. Handbook of Usability Testing. John Wiley & Sons, Inc.; 1994.
Computer / Concept
FacilitatorParticipant
Observer
Timer Logger
Portable Labs
Temporary setup Capture screen, video and audio
Complicates observation
Anywhere (conference room, remotely)
21Photo by Roebot at http://www.flickr.com/photos/roebot/2964156413/
Remote Testing
Moderated and AutomatedCapture screen, video and audio (not all automated collect this)
Moderated: Connect to both computer and phone
Technology Bandwidth, operating system, browser Test setup and connections ahead of time
22
Activity II
23
Recruiting Profile
Key elements Mimic population using product
Critical factors that make a difference Age Experience
How they will be screened Phone Online Follow-up and scheduling
24
Recruiting Profile
Samples Why decisions made What going after
Create recruiting profile for participants for usability study designed in Activity 1
25
Selecting & Preparing Materials
26
Why Use a Script?
Promotes consistency in conducting the test
Reminds to welcome and thank peopleReminds to reset/configure between participants
Ensures NDAs and other legal requirements are not missed
Prepare for non-native language speakers
Tell me more…
27
Parts of a Script
Welcome to participantsSteps in test (forms, tutorials, tasks, questions)
Thank you to participants (and incentives if any)
Notes to yourself Reset/configuration prompts
28
Test Folders
Organize materials per participant
Easily access individual results
Store as a groupPrepare a folder for each participant
All test materials(script, forms, printed tasks etc.)
Don’t use names- label folders: Pilot, P1, P2, P3
29
Informed Consent
Your responsibility to inform participant
Every time Ethical and legal responsibility Do no harm (physical, emotional)
Explain consent form* Plain language – what, why, stop anytime, etc.
Participant signs (Google form remotely)Rarely lose participants over form
30*Examples in Appendix
Interacting With Test Participants
31
Welcome
Make sure the participant feels welcomed and appreciated.
Stress that you are not testing them, rather you are testing…
Encourage feedback, positive or negative
“I was not involved in the design of this so you can’t hurt my feelings”
32
Facilitating
Follow your scriptFrame questions in unbiased mannerRemain passive in facial expressions and body language as much as possible while participant is working
Don’t lead or give clues (unless frustration is clear and evident)
Probe don’t demandGently prompt during think-aloud protocol if necessary
33
Probes
Ask open-ended questionsEncourage participant to elaborateUse participant’s wordsListen for vocalizations and watch non-verbal gestures
Have participant tell you when they are done
Give them time to think
34
Example Phrases
How does this compare with your expectations?
What specifically are you referring to?What are you looking for?Tell me more…Can you give me an example?
35
Common Issues
Refusal to critique Ask preference and to compare interfaces Watch what they DO, not what they SAY
Taking control - diverting to their issue Let them see you are listening and writing it down
If no resolution, thank and end session
- give them the incentive.
36
Getting Users Unstuck
It’s not a question of “if” but “when” Users might be embarrassed at what they perceive to be their fault
Users may not be able to articulate what they don’t understand
Don’t directly answer initial questions
37
Getting Users Unstuck
Progressive assistance What are you trying to do right now? What do you think the next step is? What would you do in this situation if you were at work?
Hints Do you see anything that might help you? (small hint)
Have you checked the Help? (medium hint)
What do you think the xxx button does? (large hint)
End before frustrated, or take a break
38
Debriefing
Verbal and written questions What did you like best? What was the worst thing? What can be improved? Describe in 3 words Follow up to specific items noted during testing
Ratings Ease of use Willingness to use
Thank participant
39
Activity III
40
Probes – Group Exercise
Are you thinking of using the search feature?
Are you trying to find the price of the book?
41
Alternatives
Are you thinking of using the search feature? (biased)
Can you tell me what you are thinking right now? (unbiased)
Are you trying to find the price of the book? (biased)
What are you trying to do? (unbiased)
42
Rational
Can’t know what participant is thinking
Keep questions brief and clearAvoid
Asking participants to predict the future Putting “words in their mouth” Showing bias or hinting at your opinions
43
Probes – Team Exercises
Break into teamsEach get a set of biased probesGenerate unbiased alternativesPresent/discuss with entire group
44
Testing Experiences
45
Low-fidelity/Paper Prototypes
Includes wireframesGood time to get initial feedback on concept, word choices and placement
Participants May not be able to understand May not be act the same as online
Always test again later in development
46
High Fidelity Prototypes
Ideal time for testing – early enough in development, but participants can “kick the tires”
May be difficult to test with (broken links, undeveloped material, etc.)
47
Functioning Products
Great for redesign projects – prior to start
Get baseline and goals for improvements
Reduce task sequence from 10 clicks to 5 Reduce time to perform task from 10 min to less than 3 min
Increase satisfaction from 4 to 5 on a 5-pt scale
Use existing data (Analytics, Help Desk, etc.) to create tasks and inform study
48
Reporting
What was tested, how and whySuccess and Failure and whyRecommendationsFuture testing – benchmarking Just enough to communicate – short & sweet
49
Activity IV
50
Activity 4
Mock up (15 minutes)Round 1 of testing (5 minutes)Revise mock ups (10 minutes)Round 2 of testing (5 minutes)Prepare team presentation (10 minutes)
51
Hotel Reservation Kiosk
Design and develop a new hotel check-in kiosk that can be installed in airports
Hotel chain wants to make it easy for their customers:
Check into their room at the airport Request a particular room type, if available
Request transportation to the hotel
52
Ideal Scenario
During your search for a local phone book, you notice a hotel chain’s digital Kiosk, and you decide to try it.
You need to: Reserve a Hotel Room for the evening. (A room with 1 bed for 2 people, non-smoking)
Request transportation to the Hotel Reserve transportation from the hotel back to the airport
53
Alternative Scenario
You want to go to the hotel right away to change before going to a fancy restaurant downtown.
You need to: Know if the hotel provides transportation. Reserve a room with a King-Size bed because you're fat and you want a view of the ocean.
Make sure that your “friend” who is coming tomorrow will have a room available next to yours.
54
Team Presentations
55
Final Q&A Questions?Handouts
56
Survey Please complete the session survey:www.uxpa2016.org/survey/69
57
Contact Us
Thyra Rauch IBM [email protected]
+1 (408) 463-2465
Carol Smith [email protected]
+1 (412) 212-8737@carologic
58
John SchragAutodesk [email protected]
@jvschrag
Resources
59
Usability by the numbers…
80% of SW life cycle costs occur during maintenance
80% of maintenance costs are due to unmet or unforeseen user requirements
Only 20% are bugs or reliability problemsTypical software program released with 40 usability design problems
Over 70% of CIO's state that one of their biggest problems is communicating with users to understand their needs
60
Number of Participants
• Contentious issue• Nielsen - 5-7
participants uncover ~80% of major usability problems • IF representative • IF doing
representative tasks. • Testing more =
significant diminishing return.
• Virzi and Carol Barnum • Small number of
users is sufficient.
61http://www.useit.com/alertbox/20000319.htmlJakob Nielsen’s Alertbox. Why You Only Need to Test with 5 Users. March 19, 2000.
Number of Test Participants
Recruitees
Could be Customers General consumers Members of organization or company
Difficulty of recruiting Specificity of screener Availability/interest in study (doctors vs. consumers)
Experience (highly experienced vs. novice)
62
Recruiting Considerations
Accessibility of test location Include people with disabilities “We are all only temporarily able-bodied. Accessibility is good for us all.”
Language interpretersRemote and on-site studies
Technology availability Administrative rights Internet connectivity
63Quote by @mollydotcom at #stirtrek 2011 via @carologic
Data Collection Forms
Customize for each studySpace for writing notes – each scenario spelled out.
Include space for tickmarks regarding easily measurable activities.
E.g. Participant accesses video clip E.g. Participant selects advertisement
64
Tool Considerations
In person or remote? Lab or on-site?Prototype limitations (can it be online?, is it a document or a clickable site?)
Number of observers, number of participants?
Number of facilitators?Logging and video editing needs (time on task, highlight video creation)?
Surveys before or after?Eye tracking?
65
Usability Testing Software
Morae OvoSilverBack (Mac only)Loop11 (Remote only)Tobii (Eye-tracker)SMI (Eye-tracker)SurveyMonkey
66
Screen Sharing Software
GoToMeeting – http://www.gotomeeting.com
Lotus Sametime Unyte – http://www.unyte.com
YuuGuu -- http://www.yuuguu.com WebEx – http://www.webex.comYugma -- https://www.yugma.com/
Trouble Shooting: CoPilot - https://www.copilot.com/
67
Satisfaction Questionnaires
Standard Usability Measurement Inventory (SUMI)
office/desktop software, purchase 50 questions
Website Analysis and MeasureMent Inventory (WHAMMI)
Purchase 20 questions
System Usability Scale (SUS) Free 10 questions
68
Ethics Resources
Treatment of human subjects Web-based course Gives certificate of completion http://cme.cancer.gov/clinicaltrials/learning/humanparticipant-protections.asp
Guidelines for writing informed consent
http://ohsr.od.nih.gov/info/sheet6.htmlUXPA Code of Conduct
69
NDA’s
Do I need a non-disclosure agreement (NDA)?
Maybe Legal department (be friendly) One-way, two-way*
AKA Confidentiality agreement (CA) Confidential disclosure agreement (CDA) Proprietary information agreement (PIA) Secrecy agreement (SA)
70
Activity ______________________________________Who are you observing? (e.g. SME in soft-goods manufacturing, application admin)
What is the situation? ___________________________(e.g. phone interview, their work at their desk in person, remote usability study, etc.)
Date: __________________Time: __________________Location: ________________Note-Taker: ______________
People (P)Who are you observing? Who else is involved in the experience?
Objects (O) What objects do they interact with? What apps? Phone, etc.?
Environment (E) Characteristics of the setting, number/types of interruptions
Messages (M)What is communicated? How is it transmitted?
Services (S)What services are available to them?
Watch for these types of observations and put additional Questions/Ideas on the back. Content in this document described by Vijay Kumar in his book: 101 Design Methods
Physical What do they interact with?
CognitiveHow associate meanings? How learn?
SocialInteractions, decision making, scheduling, work?
CulturalWhat are shared norms, habits, values?
EmotionalWhat emotions are expressed and how?
What are your biggest takeaways?What did you learn? What surprised you?What is the participants need?
Interview & Observation Notetaking Form
Observer RulesEveryone who observes a design session is asked to abide by a set of rules. The purpose of these rules is to minimize stress for the participants and to maximize the amount of information we get from the study.
Stay for the entire study Distractions are unhelpful and participants may get the impression that you’re leaving
because they’ve done something wrong (e.g. walking out in middle of a movie). If you can attend only part of a study, discuss with the facilitator beforehand to determine whether there is a way to accommodate this.
Don’t reveal information about the study to the participant It is often more useful to explore an area of difficulty in detail rather than try to “get
through” all the topics. The facilitator will track time so that we can cover as many of the important areas as possible.
Respect participants and the confidentiality of their data We have promised the participants that their participation is confidential. This means
that we should not include their names in any reports or other communication such as email, and we should refrain from discussing them by name outside the test setting.
Do not make negative comments about people—there is always a risk that a derogatory comment could be overheard or otherwise make its way back to the user.
Adapted from the book Paper Prototyping by Carolyn Snyder, published by Morgan Kaufmann Publishers. Copyright (c) 2003 Elsevier. All rights reserved.
Observer Rules (continued) Remain silent and silence phones You may notice something so surprising that you are tempted to laugh or exclaim out loud.
This is not unusual. Unfortunately, participants might think you are laughing at them. Keep as quiet as possible. You will have opportunities to ask questions at the end. If necessary, pass a note to the facilitator.
Observe only - no questions or support If reviewing an interface, it’s likely participants will have problems, and it is normal to feel a
temptation to help. Please don’t. Instead, try to understand why it was that the user got stuck or went down the wrong path. It’s the facilitator’s role to get users back on track if they get really stuck. And if the facilitator poses a question during the test, he or she is asking the users, not you—please don’t answer unless the facilitator specifically directs a question to you.
Avoid “Design Questions” - Questions that ask the user their opinions about how to design aspects of the application (such as, “Where would you like to see these navigation buttons?”) can take a lot of time to answer and produce only limited results. Instead, focus on trying to understand the problem—we’ll come up with solutions later, outside the test.
In person, be conscious of your body language Although most studies are interesting, not every moment will be fascinating. If something is
happening that isn’t of interest to you but may be to others, sit quietly without fidgeting. Take notes to stay alert.
Adapted from the book Paper Prototyping by Carolyn Snyder, published by Morgan Kaufmann Publishers. Copyright (c) 2003 Elsevier. All rights reserved.