Usability Testing and QAA practical approach
towards creating better user interfaces
Who am I?Shilpa Thanawala
@skthana
And you?Name
Role at your company or agency
Background (eg. designer? developer? etc.)
Experience with usability testing
How you think you’ll use what you learn today
Why this class?
HousekeepingInitial and sign the roster
Evaluations at the end of the day
Cell phones on silent, calls outside
Everyone knows where things are at BAVC?
Load the shared Google Doc (for questions, backchannel interactions, sharing links) http://bit.ly/ux‑oct‑03
Usability TestingSome thoughts to start us off...
Designers know too much about their product to be objective judges;the features they have come to love and prefer may not be understood
or preferred by future customers.Don Norman, nngroup.com
Image credit: templatemonster.com
The test of the machine is the satisfaction it gives you. There isnʹt anyother test. If the machine produces tranquility itʹs right. If it disturbsyou itʹs wrong until either the machine or your mind is changed.
Robert M. Pirsig, Zen and the Art of Motorcycle Maintenance: An Inquiry Into Values
Image credit: Optimum7.com
Supposing is good, but finding out is better.Mark Twain
Image credit: Mediamatic
Usability TestingSome historical highlights...
Early 20th CenturyImprovements in industrial efficiency
In WW I, reduced work motions for soldiers to assemble &take apart weapons in the dark.
http://www.measuringusability.com/blog/usability‑history.php
1947
John Karlin ‑ how short phone cords should be
“It is not so much that Mr. Karlin trainedmidcentury Americans how to use the telephone. It is, rather, that by studying the psychologicalcapabilities and limitations of ordinary people, he
trained the telephone...”Excerpt from New York Times article
1980’s
New methodologies developed and published
Usability becomes a profession
Defined as a function of efficiency, effectiveness, andsatisfaction
first publishedPsychology of Everyday Things
1990’s
Usability matures
Diminishing returns from testing more than 3‑5 participants
Books by Jakob Nielsen
2000 ‑ todaySteve Krug’s Don’t Make Me Think
Remote testing
Statistical analysis of usability data
Mobile devices, touch and gestures
http://www.measuringusability.com/blog/usability‑history.php
Why?
Why spend valuable time and money doingusability tests?
Itʹs the best way to learn how users actually interact withyour website. Watching > interviewing
You donʹt really know your usersʹ needs. Users are not all thesame. And theyʹre not your Mom.
All websites have problems, and the serious ones are easy tofind.
Why spend valuable time and money doingusability tests?
Youʹll learn things you didnʹt know that you didnʹt know.
Watching users gets everyone on the same page.
Watching users will make you a better designer / developer /product manager / content writer / etc.
Usability testing doesnʹt have to be expensive or time‑consuming.
What Testing Canʹt Tell You
Limitations of Usability Testing
Wonʹt fix your siteʹs problems (itʹs not a user training session)
Results are dependent on the questions you ask (tasks)
Canʹt model social interactions very well
Isnʹt always representative of real situations
For more on issues testing the social web see Dana Chisnellʹs talk
When?
When should usability testing bedone?
Before you design or build anything
Before you implement a change
Throughout the project
After implementing a fix
Throughout the life of the website
As soon as possible & repeat often!
Demo: A short usability test
Qualitative vs. Quantitative
Qualitative vs. Quantitative
Quantitative Usability StudiesDesigned and conducted to produce data for statistical analysis
Each measurement represents a count or an amount
Distances, weights, quantities, etc.
Qualitative Usability StudiesFocused on verbal descriptions of usersʹ experiences
Each measurement is a description or category
Words, sentences, feelings, yes/no
Categories are not quantitative, even if they are assignednumbers
userfocus.co.uk
“...qualitative methods are much better suited for answeringquestions about why or how to fix a problem, whereas quantitativemethods do a much better job answering ‘how many’ and ‘how much’
types of questions.”
More on Quantitative methods at
ʺWhen to Use Which User Experience Research Methodsʺ, nngroup.com
measuringusability.com
Usability Metrics
Effectiveness, Efficiency, andSatisfaction
EffectivenessAccuracy, correctness. Are users able to achieve their goals?
EfficiencyEase of use, speed. How much effort (time) does it take tocomplete a task?
SatisfactionUsersʹ perception. What does the user think about theirexperience?
More at , usability.gov wikipedia.org/wiki/Usability
Some Metrics for Common User Tasks
From Measuring the User Experience – Tullis & Albert
Performance‑based Metrics
Task SuccessCommonly used, pass / fail, or degrees of success, shouldhave clear end‑state.
Task TimeParticularly important for repetitive tasks
ErrorsIncorrect actions leading to significant time loss, additionalcosts, or task failure
EfficiencyEffort and time needed to complete the task
LearnabilityEffort and time needed for the user to learn how to use theinterface
Issue Metrics
Usability findings based on usersʹ behavior
Consistent / repeatable, or idiosyncratic?
Remain open‑minded to the unexpected
Other Metrics
Self‑Reporting: Usersʹ perceptions, feelings
Behavioral / Psychological: Eye‑tracking, stress,unprompted verbal expressions
Combinations / Comparisons: Interpretations based onmore than one metric
Card‑Sorting: How to organize information in a way thatmakes sense to users.
A/B Testing: Comparison of two alternate designs
Accessibility: Usability for those with disabilities (, )
CCVA of2010 WCAG
Bias
Every study contains some level of bias.
Participant background, knowledge, comfort level
Task definitions
Methodology (testing process, session length, how muchtalking)
Artifacts of the prototype or product
Environment (lighting, noise, cameras, distractions)
The Facilitator (experience, skill, style)
ExpectationsMeasuring the User Experience – Tullis & Albert
Number of Test Participants
Whatʹs the right number of users to recruit for a study?
3‑5? 4‑6? 10? 20? 100?Those who favor a small number feel 80% of the mostimportant usability problems are detected by the first fewusers.
Cases in which more users may be required:You need to be sure youʹve captured as many problems aspossible
You have more than one user group (persona)
You have a large number of screens, complex design, widevariety of tasks
Youʹd like to account for evaluator bias
Why weʹll focus on qualitative testing today
Anyone can do itNo knowledge of statistics or data analysis needed; based onverbal feedback, observation, and empathy
Cheap, fast, and easyGreat for small budgets, busy teams, minimal resources...and frequent repeats
It gets the job doneMore than sufficient to identify a websiteʹs most serioususability problems
DIY Usability TestsBased largely on Rocket Surgery Made Easy – Steve Krug
Establish your GoalsWhat do you want to learn about your site?Can users find products easily?
Is it clear to users what the site is about?
Is the information organized intuitively?
Is this new feature distracting?
Keep an open mind to allow for unexpected results.
Define your TasksTasks → Activities
First, create one or more scenarios based on essential, real‑world user goals.
Second, define activities that fit each scenario.Activities should be realistic
Activities should be actionable
Phrase them so as not to give away the answer
Print each activity on its own piece of paper. (Avoid labels or numbers.)
ʺTurn User Goals into Task Scenarios for Usability Testingʺ — nngroup.com
Recruiting Test ParticipantsWhat kinds of participants?
Where to find them?
How many?
How to compensate them?
What kinds of participants?How representative of actual users?
Most serious UI problems will be discovered by non‑representative users.
Testing with non‑representative users is better thanobsessing over finding representative users.
Key requirements: not part of the team, fluent in the UIʹslanguage
Where to find participants?Where do your users hang out?
Advertise on the site, message boards, social networks
Craigslist
How many users?For most of us doing DIY testing, 3‑5 participants is fine.
No‑shows or last‑minute cancellations are common. Have 1‑2 backup participants lined up.
Phone‑screen, scheduling, day‑before reminders, directions,parking, NDAʹs... use a checklist
CompensationShows appreciation for participantʹs time and effort
Encourages participants to be engaged and enthusiastic.
Gift cards, free product or service.
Equipment & SetupHint: you donʹt need a lab
Bare minimum (if youʹre the only one): a computer in a quietroom, a pen and notepad, and 2 chairs
For a team, add a remote observation room, a good mic, andscreen‑sharing (Skype or GHangout). And provideirresistable snacks.
Can use screen‑capture software (but most of the time itwonʹt be watched)
Can record the participant (but itʹs not really needed)
Make it a Team EffortTry to get everyone to attend.
Assign a trustworthy team member to manage theobservation room
The Session ScriptStart with a template & edit to suit your needs
Welcome / introduction
Logistics, release forms
Necessary background information
Test: general questions
Test: scenario activities
Wrapup, follow‑up questions
Conclude and provide honorarium
Some Test Session Script Templates
Free templates at: , , infodesign.com Steve Krugʹs site Book siteguide by Rubin, Chisnell & Spool
DebriefOnly for those who attended at least one session
Schedule it on the same day (next day latest), 60‑90 min.Provide food!
Get all observers to state (or stick) UX problems theyobserved
Get general agreement on which are most serious & will betackled first (2‑3 maximum)
Remind everyone when the next round of testing will takeplace
Run your own usability test1. Select a website to test
2. Determine goals
3. Identify at least 3 activities critical to success
4. Formulate scenarios for testing each activity
Lunch1 hour
(back at 2:10pm)
Run your own usability test1. Prepare your script
2. Rehearse and refine
3. Print scenario(s) and activities on 1 sheet (well‑spaced so youcan cut them apart)
Run your own usability test1. Test two participants (~20 min each)
2. Take notes
Run your own usability test1. Debrief within your original group
2. Identify 2‑3 usability issues
ToolsTesting session checklists
, ,
Usability testing software , ,
Online / Remote services , , ,
Usability.gov Krugʹs checklist checklist on book site byRubin, Chisnell & Spool
Silverback (Mac) Morae (Win) Camtasia
Usertesting.com Loop11 usabilityhub.com many more
So how do we fix it?
Fixing usability issuesThe bad news: most issues never get fixed
Make the smallest change possible to ʺstop the bleedingʺ
Large changes might introduce new issues
Itʹs not fixed until you verify by ... testing again!
Demo: Prototyping a UI change usingBalsamiq
A/B Testing
A/B TestingA simple experiment testing two cases.
Before vs. after, one fix vs. another
Qualitative or quantitative
A/B Testing PossibilitiesLayout and design
Relative sizes of UI elements
UI controls
Form elements (how many, field labels, helper‑text)
Images
Copy length or content
Calls to action
...many more
A/B Testing ActivityPrototyping your proposed fixes
A Usability Test using PaperPrototypes
Paper prototype of a kids’ website
Credit: BlueDuckLabs / YouTube.com
View online
A/B Testing Activity1. Select one usability issue from your previous activity
2. Brainstorm at least two ways to fix it
3. Print the page and use either Balsamiq or paper prototypingto mock up your two solutions.
4. Test each case on a separate participant in the other group
5. Debrief in your group. Which worked better, A or B?More detail at: SMʹs Ultimate Guide to A/B Testing
Testing Accessibility
Why is accessibility important?
“The Web is an increasingly important resource in many aspects oflife: education, employment, government, commerce, health care,
recreation, and more. It is essential that the Web be accessible in orderto provide equal access and equal opportunity
to people with disabilities. ”From The W3Cʹs Web Accessibility Initiative
Is my site accessible?Unplug your mouse / turn off your trackpad
Why ‑ blind, low‑vision, compromised motor‑control, nohands
How to test ‑ start at url, use tab / shift‑tab and arrows to seewhether you can access all controls, links, and fields on thepage, in the right order. Popups and dialogs should beusable.
How to fix ‑ proper HTML heirarchy, tabindex attribute,javascript plugins
Is my site accessible?Check color contrast
Why ‑ low‑vision; 1 out of 12 have color deficiency
How to test ‑ online tools ( , ),browser plugins ( )
How to fix ‑ shift color palette to improve contrast, increasefont sizes for low contrast areas, avoid colorful backgroundimages
Check my Colors Contrast RatioGrayscale Tool
Is my site accessible?Turn off images
Why ‑ images useless to low‑vision or blind users, lowbandwidth
How to test ‑ block images in browser, WebAIM tool( )
How to fix ‑ proper alt attributes, donʹt use images when textcan be used
WAVE
Accessibility Resources and ToolsWebAIM
W3Cʹs Web Accessibility Initiative
Quick accessibility tests & fixes
SitePointʹs Easy accessibility checks
Mobile / Touch
Mobile Usability
Context / environment (noise, lighting, glare)Many more contexts beyond desktop
BandwidthMobile devices force us to think about bandwidth again
Touch target sizeAverage fingertip size = 10mm Adjustments in spacing, button size Touch is also being used on large screens
Driving / attention What usability issues contribute to driver distraction?
Mobile Usability Testing Resources
NNGroup recommendations
Article on UXMagazine
Break10 minutes
Accessibility Testing Activity1. Click the URL at the top of the browser window
2. Without using the mouse again, try using only the keyboardto navigate
3. Can you access all the links, menus, and controls?
4. Can you determine where you are, or is it easy to lose yourplace?
Wrap‑upUsability testing is an essential tool for improving yourwebsite or product
Anyone can do a simple usability study
Test as early as possible, as frequently as possible
Tailor the type of testing you conduct (qualitative /quantitative, metrics, number of participants) to give you theinformation youʹre after
Use the available tools and resources to help you run yoursessions
Wrap‑upGet your whole team involved
Small adjustments are preferable (and faster) than a fullredesign
Remember to test your fixes, too
Stay aware of non‑standard users (accessibility) and thechanging technological landscape (new platforms, new uses)
ResourcesWebsites / Blogs: , ,
,
Quantitative Testing: , ,
nngroup.com usability.gov sensible.com(Krug) UIE
measuringusability.com Measuring theUser Experience – Tullis & Albert usability.gov
Thank you!
Shilpa Thanawala | @skthana
Recommended
