Upload
helena-meredith-reeves
View
222
Download
0
Embed Size (px)
Citation preview
© Simeon Keates 2008
Usability with ProjectLecture 12 – 20/03/09Susanne Frennert
© Simeon Keates 2008
ISO 9241 definition:
“The effectiveness, efficiency and satisfaction with which specified users achieve specified goals in particular environments”
Effectiveness: can people complete their tasks• Task completion: How much got done? Quality of output: Was what got done any good?
Efficiency: how long do people take Deviations from the critical (i.e. shortest) path: How much did the user wander? Error rate: How often did the user make a mistake? How “bad” were the mistakes?
• Time on task: How long did it take?
• Mental workload: How hard did the user have to think?
Satisfaction: the level of comfort that the user feels when using a product and how acceptable the product is to users as a vehicle for achieving their goals
• Qualitative attitude analysis: Did you like it?
• Quantitative attitude analysis: How much did you like it? (e.g. benchmarking)Page 2
Feedback from Wednesdays exercise
© Simeon Keates 2008Page 3
Feedback from Wednesdays exerciseTypical quantifiable usability measurements:
• The time users take to complete a specific task
• The number of tasks of various kind that can be completed within given time limit
• The number of website features the user can remember during a debriefing after the test
• The proportion of user statement during the test that were positive versus critical towards the website
• The proportion of users who say the would prefer using the improved website over the frozen website
• The number of times the user sidetracked from focusing on the real task
© Simeon Keates 2008Page 4
Feedback from Wednesdays exercise
Example of Hypothesis:
The user will find specific products faster at the revised site compared to the “frozen site” (the revised site offer faster task completion times than the “frozen” site”)
or
Task completion time are different between the “frozen” and the revised site
The revised site is more accessible for users who use screen readers
The user satisfaction will differ between the “frozen” and revised site
The revised site is more visually appealing
© Simeon Keates 2008Page 5
Feedback from Wednesdays exercise
Effectiveness can be measured by recording whether or not a user could complete a task
•Percentage of tasks completed successfully on first attempt
•Per cent of users able to successfully complete the task
•Number of errors made performing a specific tasks (Errors have to be defined! For example errors is defined as an attempt to click on images, icons and buttons that would not result in completing the task assigned)
•Number of requests for assistance accomplishing tasks
© Simeon Keates 2008Page 6
Feedback from Wednesdays exercise
Satisfaction refers to how the user feels about the system. Could be measured by:
•Ratio of positive to negative adjectives used to describe the product
•Per cent of users that the rate the revised site as “more satisfying” than the “frozen” site
•Per cent of customers who feel “in control” of the product
© Simeon Keates 2008Page 7
Feedback from Wednesdays exercise
Efficiency can be measured by:
•Time to execute a particular set of instructions
•Time taken on first attempt
•Time to perform a particular task
•Number of key presses taken to achieve task
© Simeon Keates 2008Page 8Page 8
User Trials Schedule
28 October 13-15 Pilot study 30 October 13-15 Revision of site and research plan
4 November 13-15 User Trials 6 November 13-15 User Trials
11 November User Trials 13 November User Trials
© Simeon Keates 2008
Conducting sessions with “accessibility” users
© Simeon Keates 2008
Conducting sessions with “accessibility” users
We will consider each of the following principal stages:• Pre-session preparation• Selection and preparation of the trial location• Conduct of the sessions• Follow-up to the sessions
Page 10
© Simeon Keates 2008Page 11
Pre-session preparation
Planning the tasks to be performed• Can the users complete the “usability” tasks?• What is the purpose of the user trial? Identify accessibility issues, usability
issues or usable access issues?• Need to add specific “accessibility” questions/tasks• Example: “Make a peanut butter sandwich” becomes “Can you tell which jar
is the peanut butter?”, etc.• Can the users do them?
© Simeon Keates 2008Page 12
Pre-session preparation
Getting ethical approval• Implications of “accessibility” users
• How is approval gained? (Example: user is deaf)
• And recorded? (User cannot sign a consent form)
• How can you be sure that the consent is valid? (Example: user has a cognitive impairment)
© Simeon Keates 2008Page 13
Example: Usability test consent form
Please read and sign this form
In this usability test:
You will be asked to perform certain tasks on a website. We will also conduct interview with you. You will be asked to fill in a questionnaire.
Participation in this usability study is voluntary. All information will remain strictly confidential. The descriptions and findings may be used to help improve the web site. However, at no time will your name or any other identification be used. You can withdraw your consent to the experiment and stop participation at any time.
If you have any questions after today, please contact (name) at (phone).
I have read and understood the information on this form and had all of my questions answered
Subject's Signature Date
© Simeon Keates 2008
[Aside] Good consent practice
Make it clear: That the session can be paused at any point
That the participants can withdraw from the session at any point
That the participants can withdraw their data at any point
Outline the data to be collected, how it will be handled and who will be able to access it
Outline the process by which the data will be fully anonymised
Describe the risks to the participants in clear and unequivocal language
Give this to the participant in writing and ask for them to sign
Page 14
© Simeon Keates 2008
[Aside] Good consent practice
You should also:
Provide examples of consent forms that participants will be expected to sign prior to the session
Ensure that at no point are the participants exposed to potential risks of “harm” (not just physical, but also emotional)
Page 15
© Simeon Keates 2008Page 16
Pre-session preparation
Identifying the needs of the participants:
Navigation • How is the person going to get to the venue?• How is he/she going to navigate around the venue?
Communication• What mode of communication should be used?• Written / spoken / Braille / sign language, etc.?
Assistive technologies• Does the users use AT?• Will they bring their own? Or do you have to supply it?• Will it work with your product?
© Simeon Keates 2008Page 17
Pre-session preparation
Conducting pilot studies
As for “usability”
Also test for accessibility of the venue, instructions, etc.
Also do tests to ensure that the AT works
© Simeon Keates 2008
Pre-session preparation
Preparing and distributing pre-session documentation What to include?• All consent forms that the participants will have to sign• Details of any reimbursement offered• Details of the time and location of the trials• Directions to the location where the trials are to be held• The tasks to be performed and details of how the session will be structured
and run (the experimental protocol)
Send these out in good time (allow extra time for reading and replying) Check the correct format• Braille• Audio files/tapes• etc.
Page 18
© Simeon Keates 2008Page 19
Selecting a trial location
Best for you? Best for data collection/analysis? Best for the user?
© Simeon Keates 2008
Challenges of trial locations
Participants who are: Challenges and needs
Blind Navigating to and around trial site (e.g. to toilet)
Getting used to the layout of the lab
Provision for a guide dog
Low vision Large, clear signposting
Deaf Provision for a sign language interpreter
Motor impaired Access to the building (e.g. ramps)
Page 20
© Simeon Keates 2008
Selecting a trial location – Usability Lab
Advantages: Tightly controlled Already set up (usually) Sufficient space and vantage points
Disadvantages: Unfamiliar for participants Can be intimidating May not be accessible• Getting there• Navigating within
Page 21
© Simeon Keates 2008
Selecting a trial location – 3rd party locations (e.g. day care centres)
Advantages: Usually known by users Familiar Usually already accessible
Disadvantages: Need to set up (possibly daily) Sufficient space and vantage points? Privacy? Possible interruptions
Page 22
© Simeon Keates 2008
Selecting a trial location – Remote testing
Advantages: No travel requirement for users Should have necessary AT and optimised computer access Cheap (time, resources and money)
Disadvantages: Reduced feedback from users Difficulty of communication if user is (e.g.) deaf Difficulty observing coping strategies Often difficulties getting software installed, data mailed back, etc.
Page 23
© Simeon Keates 2008Page 24
Conducting the sessions
Need to consider 3 main components
Pre-trial briefing
Performance of the tasks
Interviews and final debrief
© Simeon Keates 2008
Pre-trial briefing
What mode will be used for the briefing?• Spoken / written / sign language, etc.• Find what works best for the user
Ideally, as much material as possible should be sent to the user ahead of the session
Remind users that they are in control
You may need to reassure them
Page 25
© Simeon Keates 2008
Performance of the tasks
Allow more time for task completion• It may simply take more time
Do not force users into doing tasks that are beyond them• Be prepared for data “incompleteness”
Watch out for tiredness• It is not good for the user or your data collection
Keep reassuring the participants• “No – the computer is not about to blow up!”
Keep obersvers to a miminum• Do not crowd the user
Look out for coping strategies
Page 26
© Simeon Keates 2008
Interviews and final debrief
As for pre-trial briefing
What mode will be used for the briefing?• Spoken / written / sign language, etc.• Find what works best for the user
Ideally, as much material as possible should be sent to the user ahead of the session
Page 27
© Simeon Keates 2008Page 28
Working with “accessibility” users - Summary
Do not make assumptions about the needs of the participants
If in doubt – ask
Make sure that the most relevant users are selected to participate in the trials
Plan ahead and be prepared for unexpected results
© Simeon Keates 2008Page 29
Before the test – Summary(source: Nielsen “Usability Engineering”)
Have everything ready before the user show up Emphasize that it is the system that is being tested, not the user Acknowledge that the website is new and untested, and may have
problems Let the user know that they can stop at any time Explain any recording, keystroke logging, or other monitoring that is
used Tell the user that the test results will be kept completely confidential Make sure you have answered all the user’s questions before
proceeding
© Simeon Keates 2008Page 30
During the test – Summary(source: Nielsen “Usability Engineering”)
Try to give the user an early success experience Hand out the test tasks one at a time Keep a relaxed atmosphere in the test room Avoid disruptions: Close the door and post a sign on it. Never indicate in any way that the user is making mistakes or is too
slow Minimize the number of observers at the test Do not allow the users’ management to observe the test If necessary, have the experimenter stop the test if it becomes too
unpleasant
© Simeon Keates 2008Page 31
After the test – Summary(source: Nielsen “Usability Engineering”)
End by stating that the user has helped you find areas of improvement
Never report results in such way that individual users can be identified
Only show videotapes outside the usability group with the user’s permission
© Simeon Keates 2008
How many users???
Page 32
© Simeon Keates 2008
Choosing how many users
For this discussion we will assume:• (a) you have accurately identified the right types of users• (b) there is one major user demographic• (c) the users are fairly homogeneous (i.e. similar) within that demographic
Page 33
© Simeon Keates 2008
Jakob Nielsen and 5 to 8 users
The most famous answer to this question is from Jakob Nielsen He suggested 5 to 8 users
Logic: Landauer and Nielsen showed that:
Number of usability problems found = N(1-(1-L)n)
where:
N = total number of usability problems in the design
L = proportion of usability problems found by a single user
A typical value for L = 31%
Page 34
© Simeon Keates 2008
Jakob Nielsen and 5 to 8 users
Plotting for L = 31%
Page 35
© Simeon Keates 2008
Jakob Nielsen and 5 to 8 users
Important observations: 0 users find 0 problems! 1 users finds approximately 1/3rd of the problems 15 users find approximately all of the problems
Notes: These numbers are based on L=31%, that may not always be the true
value• Depends on the size and complexity of the product
Nielsen recommends 5 users (not 10 or 15) to save costs• They’ll find approximately 85% of the problem
Page 36
© Simeon Keates 2008
Nielsen and card sorting
Nielsen does not recommend 5 users for everything He recommends 15 to 20 users for card sorting (higher is better since card
sorting uses the statistical method of cluster analysis)
He reasons that there’s only ever a 75% correlation between the results from 5 users and the ultimate results
15 users give 90% correlation 20 users give 93% correlation 30 users give 95% correlation• i.e. an additional 5% for twice the number of people!
60 people reach 98% correlation
Page 37
© Simeon Keates 2008
Nielsen and card sorting
Why the difference from 5-8 to 15-20? User trials are for evaluations• i.e. trying to find out how good something that exists is
Card sorting is a generative method• i.e. trying to establish the parameters for a new product
Greater variability in the data to be collected and analysed• e.g. different perspectives of the fundamental problem• different mental models to be captured and synthesised • etc.
Can (and should) combine card sorting and user trials
Page 38
© Simeon Keates 2008
Other perspectives
Nielsen and Landauer recommend 5 to 8 users
Laurie Faulkner recommends 15 users• “Beyond the 5 user assumption: Benefits of increased sample sizes in
usability testing”
Still more people recommend other numbers So what should it be?
Let’s look at what we are trying to achieve…
Page 39
© Simeon Keates 2008
Why don’t we do large numbers in usability? (source: Frank Spillers “Demystifying usability”
We are looking for behavioural insight (what people do) Statistics tell you how many people failed, but not necessarily why they
did so The objective is usually to apply findings to fix design problems in a
corporate context (not academic analysis) Research shows that even with low numbers, you can gain valid data• Note: this is especially true for “accessibility” testing
Usability testing has been used for 25 years• Experts, authors and academics put their reputations behind the
methodology
Page 40
© Simeon Keates 2008
Behaviour vs. opinion
Market research is largely opinion-driven You need to ask lots of people what they think Needs very large samples – often 00s or 000s
Usability research is behaviour-driven You observe what people do in a restricted context, not what they say If 10/15 users are confused, you can assume many more will be Behaviour is governed by cognitive factors (biological and otherwise)
not opinion which is more emotional (and thus variable)
Page 41
© Simeon Keates 2008
Faulkner’s approach
60 users were involved in usability trials Assumed that 100% of problems were found
Taking random collections of 5 people out of the 60 Some sets found 99% of the problems Some found only 55%
Taking random collections of 10 people out of the 60 Lowest success was 80%
Taking 20 people out of 60 Lowest success was 95%
Note – these results depend on the quality of the 60 users selected
Page 42
© Simeon Keates 2008
Suggested sample sizes
Spillers suggests the following for corporate user research Surveys (phone and web) = ~240 to ~1000+ Focus groups = 15 to 20 (depending on goals of study) Usability trials = 10 to 15 participants Field studies = 15 to 40 participants Card sorting = 15 to 30 (higher is better because of the analyses
methods needed)
Accessibility trials – typically as many as you can get hold of
Page 43
© Simeon Keates 2008
Exercise
Page 44
© Simeon Keates 2008
Exercise – part 1
Prepare the testing protocol for evaluating the accessibility and usability of your web-site
Also, address any additional research aims identified in your research plan from Wednesday
Page 45
© Simeon Keates 2008
Exercise – part 2
You need to consider the following: Pre-session briefing• Prepare your welcome statement• What you are doing and why• Privacy issues and right to withdraw• Any initial questions you wish to ask• Prepare a consent form
Tasks• Identify at least 5 tasks for each user on each site• Ensure you do not introduce systematic errors• Prepare any likely questions you may wish to ask• Remember to add/amend tasks for the “blind” test
Page 46
© Simeon Keates 2008
Exercise – part 3
You need to consider (continued) Post-session de-brief• Ask any remaining questions needed to address your research issues• Thank the user for their time
E-mail your protocol to Simeon, Stina and me
Remember – you will be putting this to the test next week!
Page 47
© Simeon Keates 2008
Exercise – suggestions for tasks
Exploring the site / describe each page• Great for getting users used to what is where
Completing a guided product selection task• Find “this” product
Completing an unguided product selection task• Find “any” product of your choice
Changing your mind• You decide you do not want this
How many types of [x] (example: tea)?
Page 48
© Simeon Keates 2008
Exercise – additional points
Decide whether all users do the same tasks in the same order or not• Be on the lookout for “order” effects
You should randomise the presentation of the sites• ½ do site 1 first• ½ do site 2 first
Page 49