View
5.075
Download
5
Embed Size (px)
DESCRIPTION
Usability Testing Workshop heled by DFW-UPA. Spotlights methods and tools, including Wizard of OZ style and lab-based testing. Demos of UserZoom and Morae were also done.
Citation preview
The Internet? That thing is still around? -- Homer Simpson
Usability Boot Camp Usability Boot Camp
Brian SullivanDFW-UPA Workshop
December 5, 2009
2 DFW-UPA Workshop
IntroductionIntroduction
3 DFW-UPA Workshop
What is Usability Testing?What is Usability Testing?What is Usability Testing?What is Usability Testing?
Usability testing involves:Usability testing involves:
1.1. You Watch CustomersYou Watch Customers
2.2. They Perform TasksThey Perform Tasks
3.3. You Note Their ProblemsYou Note Their Problems
4.4. Making RecommendationsMaking Recommendations
5.5. Rinse and repeat (i.e. iterate)Rinse and repeat (i.e. iterate)
Jakob Nielsen’s Definition of Usability
Usability is a quality attribute that assesses how easy user interfaces are to use. The word "usability" also refers to methods for improving ease-of-use during the design process.
4 DFW-UPA Workshop
Cast of Characters: Three Main RolesCast of Characters: Three Main RolesCast of Characters: Three Main RolesCast of Characters: Three Main Roles
• Facilitator:Facilitator: This This person oversees the person oversees the entire test process—entire test process—plan, test, and report.plan, test, and report.
• Participant:Participant: Actual or Actual or potential customers. potential customers. Representative users Representative users (marketing, designers) (marketing, designers) should not be used.should not be used.
• Observer:Observer: Records Records events as they occur. events as they occur. Limits interaction with Limits interaction with the customer. Does the customer. Does contribute to the contribute to the report.report.
In today’s workshop, you will play all three roles!
5 DFW-UPA Workshop
Benefits of Usability Testing: Feedback from Real CustomersBenefits of Usability Testing: Feedback from Real CustomersBenefits of Usability Testing: Feedback from Real CustomersBenefits of Usability Testing: Feedback from Real Customers
• If your website is If your website is difficult to use, difficult to use, customers customers leaveleave..
• If they get lost in If they get lost in your website, your website, customers customers leaveleave..
• If a customer can’t If a customer can’t FINDFIND your product, your product, they can’t they can’t BUYBUY it. it.….….Then, they Then, they leaveleave!!
Real feedback directly from customers that use the product, which provides data—not opinions.
6 DFW-UPA Workshop
Benefits of Usability Testing: Informed Design DecisionsBenefits of Usability Testing: Informed Design DecisionsBenefits of Usability Testing: Informed Design DecisionsBenefits of Usability Testing: Informed Design Decisions
Design A Design B
We encountered these results from a Usability Test:
• Average Time to Book a Limo with Design A = 3:59
• Average Time to Book a Limo with Design B = 8:00
• 12 of 12 customers successfully booked the correct limo with Design A.
• 7 of 12 customers successfully booked the correct limo with Design B.
• All users were TRAINED and currently used Design B.
7 DFW-UPA Workshop
Benefits of Usability Testing: Mitigate Risk (Also, Rewarded)Benefits of Usability Testing: Mitigate Risk (Also, Rewarded)Benefits of Usability Testing: Mitigate Risk (Also, Rewarded)Benefits of Usability Testing: Mitigate Risk (Also, Rewarded)
For years, companies have been performing usability testing. They just call it a product release.
Teams that do usability testing before a release mitigate risk. Plus, they are rewarded. (ex: $300 Million Button)
8 DFW-UPA Workshop
Benefits of Usability Testing: Saves Time and MoneyBenefits of Usability Testing: Saves Time and MoneyBenefits of Usability Testing: Saves Time and MoneyBenefits of Usability Testing: Saves Time and Money
# of users
X # of uses per day
X # of days per year
X Average hourly wage of user
X Increase efficiency (in hours)
Annual Savings in Productivity
• Randy Bias and Deborah Mayhew’s book Randy Bias and Deborah Mayhew’s book provides hundreds of formulas to show the provides hundreds of formulas to show the return on investment for usability testing.return on investment for usability testing.
• Here is one formula:Here is one formula:
Using the previous example, a switch to Design B saves over $2 million USD.
9 DFW-UPA Workshop
Benefits of Usability Testing: Makes Lots of MoneyBenefits of Usability Testing: Makes Lots of MoneyBenefits of Usability Testing: Makes Lots of MoneyBenefits of Usability Testing: Makes Lots of Money
Jared Spool
The $300,000,000 Fix (per Jared Spool):
In the above example, we have a form with two fields, two buttons, and one link. This form appeared on the bottom of a Shopping Cart. What are the problems?
Hint: Current and New Customers both had problems.
10 DFW-UPA Workshop
Benefits of Usability Testing: Makes Lots of MoneyBenefits of Usability Testing: Makes Lots of MoneyBenefits of Usability Testing: Makes Lots of MoneyBenefits of Usability Testing: Makes Lots of Money
Jared Spool
New Customers:
New customers did not want to register. They resisted clicking the Register button. New customers wanted to buy what was in their shopping cart. They did not want a relationship. Most wanted to leave. As it turns out, customers are not required to register—it is optional.
11 DFW-UPA Workshop
Benefits of Usability Testing: Makes Lots of MoneyBenefits of Usability Testing: Makes Lots of MoneyBenefits of Usability Testing: Makes Lots of MoneyBenefits of Usability Testing: Makes Lots of Money
Jared Spool
Existing Customers:
Existing customers couldn’t remember their exact email and/or password. They clicked the “Forgot Password” link. Help Desk logs revealed this link was clicked about 167,000 each day! According to Spool, 75% of these people didn’t make a purchase. They did leave!
12 DFW-UPA Workshop
Benefits of Usability Testing: Makes Lots of MoneyBenefits of Usability Testing: Makes Lots of MoneyBenefits of Usability Testing: Makes Lots of MoneyBenefits of Usability Testing: Makes Lots of Money
Jared Spool
Jared Spool’s $300,000,000 Fix:
Make it obvious. The results: The number of customers purchasing went up by 45%. The extra purchases resulted in an extra $15,000,000 the first month. For the first year, the site saw an additional $300,000,000.
$ $
13 DFW-UPA Workshop
Usability Methods and ToolsUsability Methods and Tools
14 DFW-UPA Workshop
Two Primary Methods of Data CollectionTwo Primary Methods of Data CollectionTwo Primary Methods of Data CollectionTwo Primary Methods of Data Collection
Direct IndirectField Studies SurveysContextual Inquiries QuestionnairesDesign Walkthroughs ProfilingComparative Analysis Web AnalyticsUsability Studies Diary Studies
Usability Methods Table
15 DFW-UPA Workshop
Quantitative and Qualitative TestingQuantitative and Qualitative TestingQuantitative and Qualitative TestingQuantitative and Qualitative Testing
Quantitave Usability Testing Qualitative Usability Testing
Provides opportunity to observe users as they perform tasks.
Provides a focus on quantitative measures, such how many people are clicking on an ad.
Based on expertise and judgment of people in the test (participant, facilitator, observer).
Based primarily on click-stream data and log files of real customers
Facilitator and observer(s) interprets the observed customer behavior and recommends solutions
Data must be interpreted from the click-stream or log files.
Data can be done with small sample sizes. Requires large sample sizes.
16 DFW-UPA Workshop
Quantitative Testing Tool (User Zoom Demo)Quantitative Testing Tool (User Zoom Demo)Quantitative Testing Tool (User Zoom Demo)Quantitative Testing Tool (User Zoom Demo)
• Quantitative Usability Testing has Quantitative Usability Testing has grew in the last decade.grew in the last decade.
• Many tools exist for quantitative Many tools exist for quantitative testing tools are out there.testing tools are out there.
• Keynote Systems has been Keynote Systems has been around for a long time now.around for a long time now.
• You may currently use or know You may currently use or know people that use Google Analytics.people that use Google Analytics.
• Chalkmark is something we will Chalkmark is something we will see later today.see later today.
• Lou Capone will give us a demo Lou Capone will give us a demo of UserZoom (30-minute demo).of UserZoom (30-minute demo).
Web IQ
17 DFW-UPA Workshop
Qualitative Testing Tools and MethodsQualitative Testing Tools and MethodsQualitative Testing Tools and MethodsQualitative Testing Tools and Methods
• Qualitative usability testing for Qualitative usability testing for the web has really matured in the web has really matured in the last 30 years.the last 30 years.
• Quantitative usability testing Quantitative usability testing tools has really matured in the tools has really matured in the last 10 years. last 10 years.
• Morae and OvoStudios both Morae and OvoStudios both offer mature solutions for offer mature solutions for usability professionals.usability professionals.
• IntuitionHQ is very new to the IntuitionHQ is very new to the market—I was just made aware market—I was just made aware of them this week.of them this week.
• We will use Morae today. We will use Morae today.
18 DFW-UPA Workshop
Three Types of Usability Tests (Qualitative Perspective)Three Types of Usability Tests (Qualitative Perspective)Three Types of Usability Tests (Qualitative Perspective)Three Types of Usability Tests (Qualitative Perspective)
Design Walkthrough Formative Usability Test Summative Usability TestConducted during prototype development to validate initial design decisions, determine stregnths and weaknesses, or to compare various designs.
Conducted during the design or development phases. Used to determine if a design objective is being met. Designs change throughout the test.
Conducted at the end of development--code is frozen. Used to validate that a product has accomplished its intended goals.
Example: Which of the two new designs best meets the customer's expectations? Which design was most preferred by the customers?
Example: Can users find and use a "mortgage calculator" to successfully enter the appropriate loan points on an application?
Example: Can users find and use a "mortgage calculator" to successfully enter the appropriate loan points in less than 1 minute?
Today’s workshop will focus on summative usability testing.
19 DFW-UPA Workshop
Basic Usability Testing ProcessBasic Usability Testing ProcessBasic Usability Testing ProcessBasic Usability Testing Process
Step 1: Planning Your Usability TestStep 1: Planning Your Usability Test
• Define what you are testingDefine what you are testing
• Define which customers you are testingDefine which customers you are testing
• Define what tasks you are testingDefine what tasks you are testing
• Recruit your customersRecruit your customers
• Write your usability scenarios and tasksWrite your usability scenarios and tasks
Step 2: Conducting Your Usability TestStep 2: Conducting Your Usability Test
• Conduct a pilot testConduct a pilot test
• Facilitate remaining testsFacilitate remaining tests
Step 3: Analyzing and Reporting Your Usability TestStep 3: Analyzing and Reporting Your Usability Test
• Compile resultsCompile results
• Make recommendationsMake recommendations
20 DFW-UPA Workshop
A Typical Scene Behind The Glass in a Usability TestA Typical Scene Behind The Glass in a Usability TestA Typical Scene Behind The Glass in a Usability TestA Typical Scene Behind The Glass in a Usability Test
21 DFW-UPA Workshop
Step 1: Planning Your Usability TestStep 1: Planning Your Usability Test
22 DFW-UPA Workshop
Basic Usability Testing ProcessBasic Usability Testing ProcessBasic Usability Testing ProcessBasic Usability Testing Process
Step 1: Planning Your Usability TestStep 1: Planning Your Usability Test
• Define what you are testingDefine what you are testing
• Define which customers you are testingDefine which customers you are testing
• Define what tasks you are testingDefine what tasks you are testing
• Recruit your customersRecruit your customers
• Write your usability scenarios and tasksWrite your usability scenarios and tasks
Time = Money:
Most usability tests are less than 90 minutes. Plus, you are taking your customer’s time. Put first things first—what needs testing!!!
(Activities)
(People)
(Context)
23 DFW-UPA Workshop
A Simplified Version of Activities-People-ContextA Simplified Version of Activities-People-ContextA Simplified Version of Activities-People-ContextA Simplified Version of Activities-People-Context
Picture from Stephen Anderson
Somebody (person) does something (activity) some where or some how (context).
This model is Robert Bailey’s Human Performance Model. Let’s use this model to quickly plan out a test!
24 DFW-UPA Workshop
Here is the Royal Caribbean International WebsiteHere is the Royal Caribbean International WebsiteHere is the Royal Caribbean International WebsiteHere is the Royal Caribbean International Website
25 DFW-UPA Workshop
Describe Activities That Might Be Done…..Describe Activities That Might Be Done…..Describe Activities That Might Be Done…..Describe Activities That Might Be Done…..
Picture from Stephen Anderson
• Create a group reservationCreate a group reservation
• Plan an event (ex: wedding)Plan an event (ex: wedding)
• Update profile (ex: Passport)Update profile (ex: Passport)
• Review Sailing RouteReview Sailing Route
• Make a second paymentMake a second payment
• Move to a different shipMove to a different ship
• Upgrade to a new cabinUpgrade to a new cabin
• Buy an excursion dealBuy an excursion deal
26 DFW-UPA Workshop
Describe People That Might Be Using the Site…..Describe People That Might Be Using the Site…..Describe People That Might Be Using the Site…..Describe People That Might Be Using the Site…..
Picture from Stephen Anderson
• Captain of the ShipCaptain of the Ship
• Group Traveler Group Traveler
• Senior CitizenSenior Citizen
• College StudentCollege Student
• Potential customerPotential customer
• Travel AgentTravel Agent
• Special Needs TravelerSpecial Needs Traveler
• Worker (on the Ship)Worker (on the Ship)
27 DFW-UPA Workshop
Describe the Context of Use…..Describe the Context of Use…..Describe the Context of Use…..Describe the Context of Use…..
Picture from Stephen Anderson
• Company websiteCompany website
• IVR systemIVR system
• Kiosk (on the ship)Kiosk (on the ship)
• Printout (ex: Itinerary)Printout (ex: Itinerary)
• Social MediaSocial Media
• Email/text messagingEmail/text messaging
• Voice MailVoice Mail
• TV, Radio, DVRTV, Radio, DVR
• Inside or outsideInside or outside
• Public or privatePublic or private
• Noisy or quietNoisy or quiet
• Legal regulationsLegal regulations
• Social attitudesSocial attitudes
• Company goalsCompany goals
28 DFW-UPA Workshop
It Is Not Too Complicated…..It Is Not Too Complicated…..It Is Not Too Complicated…..It Is Not Too Complicated…..
• People = Usability ParticipantsPeople = Usability Participants
• Context = Test ScenariosContext = Test Scenarios
• Activities = Usability TasksActivities = Usability Tasks
29 DFW-UPA Workshop
Activities: What Concerns Does Your Product Team Have?Activities: What Concerns Does Your Product Team Have?Activities: What Concerns Does Your Product Team Have?Activities: What Concerns Does Your Product Team Have?
Ask the project team to list out Ask the project team to list out their overall concerns with their their overall concerns with their product. Let’s go back to the product. Let’s go back to the Royal Caribbean International.Royal Caribbean International.
Overall Concerns:Overall Concerns:
1.1. Ease of learningEase of learning
2.2. Ease of useEase of use
3.3. Ease of navigationEase of navigation
4.4. Usefulness of HelpUsefulness of Help
5.5. Recovering for errorsRecovering for errors
Specific Concerns:Specific Concerns:
1.1. Booking Group ReservationsBooking Group Reservations
2.2. Splitting Group PaymentsSplitting Group Payments
3.3. Availability of Group CabinsAvailability of Group Cabins
30 DFW-UPA Workshop
People: Who are the Target Customers?People: Who are the Target Customers?People: Who are the Target Customers?People: Who are the Target Customers?
Ask the project team. They tend to Ask the project team. They tend to focus on a focus on a singlesingle user. user.
Frame It This Way:Frame It This Way:
1.1. Who are primary customers?Who are primary customers? (Define their characteristics.) (Define their characteristics.)
2.2. Is there another group?Is there another group? (How are they like 1 (How are they like 1stst group?) group?) (How are thy different?) (How are thy different?)
3.3. Is there another group? Is there another group? (How are they like both groups?) (How are they like both groups?)
(How are they different?) (How are they different?)
Travel Agent Event Planner
31 DFW-UPA Workshop
Context: Where and How are People Going to Use It?Context: Where and How are People Going to Use It?Context: Where and How are People Going to Use It?Context: Where and How are People Going to Use It?
Ask the team. Usually, the team Ask the team. Usually, the team focuses on the delivery device. focuses on the delivery device. They will need help with the other They will need help with the other contextual factors.contextual factors.
Frame It This Way:Frame It This Way:
1.1. What device(s) is it on?What device(s) is it on?
2.2. Is inside or outside?Is inside or outside?
3.3. Is it public or private?Is it public or private?
4.4. Is it work or leisure-related?Is it work or leisure-related?
5.5. Is it noisy or quiet?Is it noisy or quiet?
6.6. Any legal regulations?Any legal regulations?
32 DFW-UPA Workshop
It Is Not Too Complicated…..It Is Not Too Complicated…..Just Ask!!!Just Ask!!!It Is Not Too Complicated…..It Is Not Too Complicated…..Just Ask!!!Just Ask!!!
• People = Usability ParticipantsPeople = Usability Participants
• Context = Test ScenariosContext = Test Scenarios
• Activities = Usability TasksActivities = Usability Tasks
33 DFW-UPA Workshop
With People-Context-Activities Defined, Data Can Be CollectedWith People-Context-Activities Defined, Data Can Be CollectedWith People-Context-Activities Defined, Data Can Be CollectedWith People-Context-Activities Defined, Data Can Be Collected
Performance Data:
• Average Task Completion
• Average Time on Task
• # of clicks to Help
• # of omitted steps
• # of errors made
• # of hints
Subjective Data:
• Prefer Design A or B
• Design Suggestions
• Task Ease Rating
• Satisfaction Rating
• User Comments
• Ranking Features
Performance and subjective data are equally valuable. We will come back to data collection in a few minutes.
34 DFW-UPA Workshop
Basic Usability Testing Process—RecrutingBasic Usability Testing Process—RecrutingBasic Usability Testing Process—RecrutingBasic Usability Testing Process—Recruting
Step 1: Planning Your Usability TestStep 1: Planning Your Usability Test
• Define what you are testingDefine what you are testing
• Define which customers you are testingDefine which customers you are testing
• Define what tasks you are testingDefine what tasks you are testing
• Recruit your customersRecruit your customers
• Write your usability scenarios and tasksWrite your usability scenarios and tasks
(Activities)
(People)
(Context)
35 DFW-UPA Workshop
Number of Participants DebateNumber of Participants DebateNumber of Participants DebateNumber of Participants Debate
36 DFW-UPA Workshop
Number of Participants Debate (Sources) Number of Participants Debate (Sources) Number of Participants Debate (Sources) Number of Participants Debate (Sources)
Jakob Nielsen (circa 1993)
Five users will uncover about 85% of the usability issues of a product. Virzi backs up this claim around the same time.
Jared Spool (circa 2000)
Five users revealed only 35% of the usability issues of a product. This claim is backed up by Schroeder.
Laura Faulkner (2003)
- Five users uncovered 85% of usability issues on average.- But, the percent ranged from 55-100% (confidence interval).- Ten users uncovered 95% with a lower bound of 82%.
37 DFW-UPA Workshop
Number of Participants (Brian Sullivan, 2009)Number of Participants (Brian Sullivan, 2009)Number of Participants (Brian Sullivan, 2009)Number of Participants (Brian Sullivan, 2009)
General Rules:General Rules:
• Test with 6 customers Test with 6 customers (per group)(per group)—if —if time permits, use 8 customers.time permits, use 8 customers.
• More customers = more reliable resultsMore customers = more reliable results
• More customers = fewer new resultsMore customers = fewer new results
• For larger samples, do For larger samples, do multiplemultiple rounds rounds of smaller tests (rather than 1 large UT)of smaller tests (rather than 1 large UT)
Base Your Number On:Base Your Number On:
• Importance of projectImportance of project
• Complexity of the productComplexity of the product
• Experience of testing teamExperience of testing team
• Expected variety of user reactionsExpected variety of user reactions
38 DFW-UPA Workshop
Recruiting ParticipantsRecruiting ParticipantsRecruiting ParticipantsRecruiting Participants
Recruiting:Recruiting:
• In-house sourcesIn-house sources
• Recruiting agencyRecruiting agency
Incentives:Incentives:
• Gift checks ($100 per session)Gift checks ($100 per session)
• Food or gift cardsFood or gift cards
Scheduling:Scheduling:
• Slot timesSlot times
• Breaks between participantsBreaks between participants
• Backups, if neededBackups, if needed
• Floaters, if needed Floaters, if needed
39 DFW-UPA Workshop
Selecting the Right Customers Selecting the Right Customers Selecting the Right Customers Selecting the Right Customers
User profiles:User profiles:
• Potential CustomerPotential Customer
• New CustomerNew Customer
• Returning CustomerReturning Customer
• Primary or tertiary customersPrimary or tertiary customers
Characteristics:Characteristics:
• Job responsibilities (not their title)Job responsibilities (not their title)
• Years experienceYears experience
• Expertise level (novice, intermediate, expert)Expertise level (novice, intermediate, expert)
• Specific skills (example: heart surgeons rather MDs or nurses)Specific skills (example: heart surgeons rather MDs or nurses)
• Geographic locationGeographic location
Source: David Armano
40 DFW-UPA Workshop
Your Recruiting is Only As Good As the ScreenerYour Recruiting is Only As Good As the ScreenerYour Recruiting is Only As Good As the ScreenerYour Recruiting is Only As Good As the Screener
Steve Krug has said this:Steve Krug has said this:
I respectfully disagree. Because…I respectfully disagree. Because…
• Your test is a Your test is a waste of timewaste of time, if you , if you recruit the recruit the wrongwrong people. people.
• Your usability test is Your usability test is dangerousdangerous, if , if you you act upon recommendationsact upon recommendations from a bad recruiting sample.from a bad recruiting sample.
You must You must screenscreen participants. participants.
For recruiting, grade on a curve! Grab a neighbor.
41 DFW-UPA Workshop
Basic Usability Testing Process—Write Out the TestBasic Usability Testing Process—Write Out the TestBasic Usability Testing Process—Write Out the TestBasic Usability Testing Process—Write Out the Test
Step1: Planning Your Usability TestStep1: Planning Your Usability Test
• Define what you are testingDefine what you are testing
• Define which customers you are testingDefine which customers you are testing
• Define what tasks you are testingDefine what tasks you are testing
• Recruit your customersRecruit your customers
• Write your usability scenarios and tasksWrite your usability scenarios and tasks
• People = Usability ParticipantsPeople = Usability Participants
• Context = Test ScenariosContext = Test Scenarios
• Activities = Usability TasksActivities = Usability Tasks
42 DFW-UPA Workshop
Scenarios and TasksScenarios and TasksScenarios and TasksScenarios and Tasks
Some experts distinguish between tasks and scenarios:Some experts distinguish between tasks and scenarios:
Scenarios:Scenarios:
• Provide the participant with motivation and contextProvide the participant with motivation and context
• Sets the stage or tells a storySets the stage or tells a story
• Increases understandingIncreases understanding
• Makes the situation more realisticMakes the situation more realistic
• Includes Includes taskstasks
Tasks:Tasks:
• Quick and to the pointQuick and to the point
• Do not need context to increase understandingDo not need context to increase understanding
43 DFW-UPA Workshop
Example of Scenario with TasksExample of Scenario with TasksExample of Scenario with TasksExample of Scenario with Tasks
Scenario #1:Scenario #1:
You want to book a sailing on Royal Caribbean International for next June You want to book a sailing on Royal Caribbean International for next June with your church group. The group is called “Saint Francis Summer 2010”. with your church group. The group is called “Saint Francis Summer 2010”. The group is selling out fast, so you want to book a cabin, which is close to The group is selling out fast, so you want to book a cabin, which is close to an elevator because your leg hurts from a recent injury.an elevator because your leg hurts from a recent injury.
• Open your browserOpen your browser
• Click the link labeled “Royal Caribbean”Click the link labeled “Royal Caribbean”
• Tell me the available cabins in the “Saint Francis Summer 2010” groupTell me the available cabins in the “Saint Francis Summer 2010” group
• Tell me a cabin number closest to an elevatorTell me a cabin number closest to an elevator
• Book the cabin the best suits your needsBook the cabin the best suits your needs
44 DFW-UPA Workshop
Practice Writing Scenarios and TasksPractice Writing Scenarios and TasksPractice Writing Scenarios and TasksPractice Writing Scenarios and Tasks
• We have two teams today.We have two teams today.--Team Cool--Team Cool--Team Ninja Assassins--Team Ninja Assassins
• Pair up with someone.Pair up with someone.
• Write a scenario with tasks. Write a scenario with tasks.
• We will pass out papers of We will pass out papers of some sample sites. some sample sites.
• You will use your script later You will use your script later today to run some tests.today to run some tests.
45 DFW-UPA Workshop
Step 2: Conducting Your TestStep 2: Conducting Your Test
46 DFW-UPA Workshop
Basic Usability Process—Step 2Basic Usability Process—Step 2Basic Usability Process—Step 2Basic Usability Process—Step 2
Step 1: Planning Your Usability TestStep 1: Planning Your Usability Test
• Define what you are testingDefine what you are testing
• Define which customers you are testingDefine which customers you are testing
• Define what tasks you are testingDefine what tasks you are testing
• Recruit your customersRecruit your customers
• Write your usability scenarios and tasksWrite your usability scenarios and tasks
Step 2: Conducting Your Usability TestStep 2: Conducting Your Usability Test
• Conduct a pilot testConduct a pilot test
• Facilitate remaining testsFacilitate remaining tests
47 DFW-UPA Workshop
Don’t Rely on Luck!!!Don’t Rely on Luck!!!Don’t Rely on Luck!!!Don’t Rely on Luck!!!
48 DFW-UPA Workshop
You Do Not Need a Usability Lab to Conduct a TestYou Do Not Need a Usability Lab to Conduct a TestYou Do Not Need a Usability Lab to Conduct a TestYou Do Not Need a Usability Lab to Conduct a Test
49 DFW-UPA Workshop
Test Side-by-Side in a Conference RoomTest Side-by-Side in a Conference RoomTest Side-by-Side in a Conference RoomTest Side-by-Side in a Conference Room
50 DFW-UPA Workshop
Test Wherever Your Customers Might BeTest Wherever Your Customers Might BeTest Wherever Your Customers Might BeTest Wherever Your Customers Might Be
I have done usability testing at Starbuck’s and a User Conference.
51 DFW-UPA Workshop
Remote Usability Testing Easier Than EverRemote Usability Testing Easier Than EverRemote Usability Testing Easier Than EverRemote Usability Testing Easier Than Ever
52 DFW-UPA Workshop
Morae is Portable, Affordable, and ………………UsableMorae is Portable, Affordable, and ………………UsableMorae is Portable, Affordable, and ………………UsableMorae is Portable, Affordable, and ………………Usable
53 DFW-UPA Workshop
Shhhhh….you don’t need to record, eitherShhhhh….you don’t need to record, eitherShhhhh….you don’t need to record, eitherShhhhh….you don’t need to record, either
54 DFW-UPA Workshop
Basic Usability Process—Step 2Basic Usability Process—Step 2Basic Usability Process—Step 2Basic Usability Process—Step 2
Step 1: Planning Your Usability TestStep 1: Planning Your Usability Test
• Define what you are testingDefine what you are testing
• Define which customers you are testingDefine which customers you are testing
• Define what tasks you are testingDefine what tasks you are testing
• Recruit your customersRecruit your customers
• Write your usability scenarios and tasksWrite your usability scenarios and tasks
Step 2: Conducting Your Usability TestStep 2: Conducting Your Usability Test
• Conduct a pilot testConduct a pilot test
• Facilitate remaining testsFacilitate remaining tests
55 DFW-UPA Workshop
Conduct a Pilot TestConduct a Pilot TestConduct a Pilot TestConduct a Pilot Test
Dry run of a complete usability testDry run of a complete usability test
• Include all test materialInclude all test material
• Focus on the test scenarios, not findingsFocus on the test scenarios, not findings
• At least, one participant should be a pilotAt least, one participant should be a pilot
• Conduct before test begins (one day prior)Conduct before test begins (one day prior)
• Make changes, if neededMake changes, if needed
• May need to test, againMay need to test, again
56 DFW-UPA Workshop
Basic Usability Process—Step 2Basic Usability Process—Step 2Basic Usability Process—Step 2Basic Usability Process—Step 2
Step 1: Planning Your Usability TestStep 1: Planning Your Usability Test
• Define what you are testingDefine what you are testing
• Define which customers you are testingDefine which customers you are testing
• Define what tasks you are testingDefine what tasks you are testing
• Recruit your customersRecruit your customers
• Write your usability scenarios and tasksWrite your usability scenarios and tasks
Step 2: Conducting Your Usability TestStep 2: Conducting Your Usability Test
• Conduct a pilot testConduct a pilot test
• Facilitate remaining testsFacilitate remaining tests
57 DFW-UPA Workshop
Again, Adopt a Workable ScheduleAgain, Adopt a Workable ScheduleAgain, Adopt a Workable ScheduleAgain, Adopt a Workable Schedule
58 DFW-UPA Workshop
Tips for Being a Good ModeratorTips for Being a Good ModeratorTips for Being a Good ModeratorTips for Being a Good Moderator
1.1. Know Your UCD Basics and Design GuidelinesKnow Your UCD Basics and Design Guidelines
2.2. Be a Quick LearnerBe a Quick Learner
3.3. Set an Instant Rapport with ParticipantsSet an Instant Rapport with Participants
4.4. Have an Excellent MemoryHave an Excellent Memory
5.5. Be a Good Listener (Use Active Listening)Be a Good Listener (Use Active Listening)
6.6. Don’t Stress Out About AmbiguityDon’t Stress Out About Ambiguity
7.7. Be Very FlexibleBe Very Flexible
8.8. Have a Long Attention SpanHave a Long Attention Span
9.9. Show Empathy (ie Be a People Person)Show Empathy (ie Be a People Person)
10.10. Thing “Big Picture”Thing “Big Picture”
11.11. Be a Good CommunicatorBe a Good Communicator
12.12. Be a Good Organizer and CoordinatorBe a Good Organizer and Coordinator
59 DFW-UPA Workshop
Tips for Getting the Most from ParticipantsTips for Getting the Most from ParticipantsTips for Getting the Most from ParticipantsTips for Getting the Most from Participants
• Start with an Easy Task To Build ConfidenceStart with an Easy Task To Build Confidence
• Choose the Right Format (remote, on-site UT)Choose the Right Format (remote, on-site UT)
• Sit Beside the Person Not Behind the GlassSit Beside the Person Not Behind the Glass
• Use “Think-Out-Loud” ProtocolUse “Think-Out-Loud” Protocol
• Give Them Time to Think It ThroughGive Them Time to Think It Through
• Offer Appropriate EncouragementOffer Appropriate Encouragement
• Lead Participants, Don’t Answer Question (Being an Enabler)Lead Participants, Don’t Answer Question (Being an Enabler)
• Don’t Act Knowledgeable (Treat Them As the Experts)Don’t Act Knowledgeable (Treat Them As the Experts)
• Don’t Get Too Involved in Data CollectionDon’t Get Too Involved in Data Collection
• Don’t Jump to ConclusionsDon’t Jump to Conclusions
• Don’t Solve Their Problems Immediately (ie act as a Help Desk)Don’t Solve Their Problems Immediately (ie act as a Help Desk)
60 DFW-UPA Workshop
Team Cool Starts With Wizard of Oz StyleTeam Cool Starts With Wizard of Oz StyleTeam Cool Starts With Wizard of Oz StyleTeam Cool Starts With Wizard of Oz Style
Wizard of Oz Style Testing:Wizard of Oz Style Testing:
• ModeratorModerator
• TesterTester
• Wizard (Person moves screen shots)Wizard (Person moves screen shots)
@suredoc (Keith Anderson) is your guide today.
NOTE: Keith is NOT the WWW – Wicked Witch of the West.
61 DFW-UPA Workshop
Rules for the Merry Old Land of Oz…..Rules for the Merry Old Land of Oz…..Rules for the Merry Old Land of Oz…..Rules for the Merry Old Land of Oz…..
Here are The Rules:Here are The Rules:
1.1. Get into Groups of three.Get into Groups of three.
2.2. Each person picks a role.Each person picks a role.
3.3. Wizard does not talk.Wizard does not talk.
4.4. Moderator analyzes TesterModerator analyzes Tester
5.5. Tester talks through the task.Tester talks through the task.
6.6. Complete the Test.Complete the Test.
7.7. Switch Roles.Switch Roles.
8.8. Find two new people.Find two new people.
9.9. Repeat Steps 1-6.Repeat Steps 1-6.
10.10. Compile results via K-J MethodCompile results via K-J Method (well do that later, though). (well do that later, though).
62 DFW-UPA Workshop
Team Ninja Assassins Will Be in Usability LabTeam Ninja Assassins Will Be in Usability LabTeam Ninja Assassins Will Be in Usability LabTeam Ninja Assassins Will Be in Usability Lab
These are the Test Stations:These are the Test Stations:
1.1. Two Waiting ParticipantsTwo Waiting Participants
2.2. One Tester in Test CellOne Tester in Test Cell
3.3. ModeratorModerator
4.4. Test Cell ObserverTest Cell Observer
5.5. Camera OperatorCamera Operator
6.6. TV OperatorTV Operator
7.7. Remote Viewer LoggerRemote Viewer Logger
8.8. Remote Viewer ManagerRemote Viewer Manager
9.9. Results Compiler (2 people)Results Compiler (2 people)
10.10. Two ObserversTwo Observers
11.11. Rotate between each stationRotate between each station
We’ll test four different sites today.
63 DFW-UPA Workshop
After 1.5 Hours, We Switch LocationsAfter 1.5 Hours, We Switch LocationsAfter 1.5 Hours, We Switch LocationsAfter 1.5 Hours, We Switch Locations
The Teams Switch:The Teams Switch:
• Team Cool in LabsTeam Cool in Labs
• Team Ninja Assassins Team Ninja Assassins (Wizard of Oz)(Wizard of Oz)
64 DFW-UPA Workshop
Step 3: Analyzing and ReportingStep 3: Analyzing and Reporting
65 DFW-UPA Workshop
Basic Usability Process—Home Stretch Now!!!Basic Usability Process—Home Stretch Now!!!Basic Usability Process—Home Stretch Now!!!Basic Usability Process—Home Stretch Now!!!
Step 1: Planning Your Usability TestStep 1: Planning Your Usability Test
• Define what you are testingDefine what you are testing
• Define which customers you are testingDefine which customers you are testing
• Define what tasks you are testingDefine what tasks you are testing
• Recruit your customersRecruit your customers
• Write your usability scenarios and tasksWrite your usability scenarios and tasks
Step 2: Conducting Your Usability TestStep 2: Conducting Your Usability Test
• Conduct a pilot testConduct a pilot test
• Facilitate remaining testsFacilitate remaining tests
Step 3: Analyzing and Reporting Your Usability TestStep 3: Analyzing and Reporting Your Usability Test
• Compile resultsCompile results
• Make recommendationsMake recommendations
66 DFW-UPA Workshop
Dilbert’s Boss Solves Your Customer ProblemDilbert’s Boss Solves Your Customer ProblemDilbert’s Boss Solves Your Customer ProblemDilbert’s Boss Solves Your Customer Problem
67 DFW-UPA Workshop
8 Steps to Analyze and Report Your Findings8 Steps to Analyze and Report Your Findings8 Steps to Analyze and Report Your Findings8 Steps to Analyze and Report Your Findings
1.1. Complete Your NotesComplete Your Notes
2.2. Summarize Your FindingsSummarize Your Findings
3.3. Compile and Tabulate DataCompile and Tabulate Data
4.4. Analyze Your FindingsAnalyze Your Findings
5.5. Prioritize Findings by Severity LevelPrioritize Findings by Severity Level
6.6. Create RecommendationsCreate Recommendations
7.7. Develop a Final Document or PresentationDevelop a Final Document or Presentation
8.8. Follow-up Follow-up
68 DFW-UPA Workshop
Step 1—Complete Your NotesStep 1—Complete Your NotesStep 1—Complete Your NotesStep 1—Complete Your Notes
• After each session, fill in any gaps.After each session, fill in any gaps.
• Basically, complete the short-hand used in the UT.Basically, complete the short-hand used in the UT.
• Mark any areas worth noting.Mark any areas worth noting.
• Use a whiteboard! It is really much easier, in the end.Use a whiteboard! It is really much easier, in the end.
69 DFW-UPA Workshop
Step 2—Summarize FindingsStep 2—Summarize FindingsStep 2—Summarize FindingsStep 2—Summarize Findings
• After each session, create a list of the major issues.After each session, create a list of the major issues.
• Organize by Issue and Participant.Organize by Issue and Participant.
• Do Not Start Solution Finding Here!Do Not Start Solution Finding Here!
• Below is an Example of How to Summarize:Below is an Example of How to Summarize:
Number Issue Description Customers1 Unable to locate Group for RCI Sailing from Home Page P1, P2, P3, P42 Credit card payment summary incomplete (no SEC) P2, P63 Rate Category page does not have sort capability P1, P5, P74 Bad error message on Cabin Swap page. P3, P6
70 DFW-UPA Workshop
Step 3—Compile and Tabulate DataStep 3—Compile and Tabulate DataStep 3—Compile and Tabulate DataStep 3—Compile and Tabulate Data
• Spreadsheet work best hereSpreadsheet work best here
• Do not work on solutions, yetDo not work on solutions, yet
• Try to find the real problemTry to find the real problem
71 DFW-UPA Workshop
Step 4—Analyze FindingsStep 4—Analyze FindingsStep 4—Analyze FindingsStep 4—Analyze Findings
• Which tasks were simple?Which tasks were simple?
• Which tasks were problematic?Which tasks were problematic?
• Which participants had issues? (Look at it by group.)Which participants had issues? (Look at it by group.)
• Speculate on solutions.Speculate on solutions.
• Use Balsamiq to sketch out recommendations.Use Balsamiq to sketch out recommendations.
Sketched in Balsamiq in less than 5 minutes.
72 DFW-UPA Workshop
Step 5—Prioritize FindingsStep 5—Prioritize FindingsStep 5—Prioritize FindingsStep 5—Prioritize Findings
• Severity 1 = Showstopper. Unless corrected allSeverity 1 = Showstopper. Unless corrected allusers will essentially have a problem.users will essentially have a problem.
• Severity 2 = High Impact. Significant impact to usability or Severity 2 = High Impact. Significant impact to usability or performance. High customer annoyance and frustration.performance. High customer annoyance and frustration.
• Severity 3 = Medium Impact. Some users experience degraded Severity 3 = Medium Impact. Some users experience degraded performance or frustration. Help Desk calls possible.performance or frustration. Help Desk calls possible.
• Severity 4 = Low Impact. Probably will not have a significant Severity 4 = Low Impact. Probably will not have a significant impact to user performance. Might be cosmetic in nature.impact to user performance. Might be cosmetic in nature.
73 DFW-UPA Workshop
Step 6—Create RecommendationsStep 6—Create RecommendationsStep 6—Create RecommendationsStep 6—Create Recommendations
• Start with Positive FindingsStart with Positive Findings
• Remind the team of people-activities-contextRemind the team of people-activities-context
• Be a problem solverBe a problem solver
• Present Issues by Severity Level (helps to prioritize)Present Issues by Severity Level (helps to prioritize)
• It great to do a MOCKUP hereIt great to do a MOCKUP here
• Tie recommendation back to USER PERSONAS, tooTie recommendation back to USER PERSONAS, too
74 DFW-UPA Workshop
Nobody reads a “BIG, HONKING” report.
75 DFW-UPA Workshop
Step 7—Create Final Document or PresentationStep 7—Create Final Document or PresentationStep 7—Create Final Document or PresentationStep 7—Create Final Document or Presentation
• Consider the audienceConsider the audience
• Some teams just want an Excel spreadsheetSome teams just want an Excel spreadsheet
• Provide enough detail (executive summary v. full report)Provide enough detail (executive summary v. full report)
• Create a distribution list for key stakeholdersCreate a distribution list for key stakeholders
• Purpose (identify issues, prioritize them, make recommendations)Purpose (identify issues, prioritize them, make recommendations)
76 DFW-UPA Workshop
Step 8—Follow-upStep 8—Follow-upStep 8—Follow-upStep 8—Follow-up
• Always Provide Usability Next StepsAlways Provide Usability Next Steps
• Usability Needs to Be Part of the Usability Needs to Be Part of the Development/Design CycleDevelopment/Design Cycle
• Invite People to Other StudiesInvite People to Other Studies
• Create an Action PlanCreate an Action Plan
77 DFW-UPA Workshop
Let’s Compile and Analyze Our ResultsLet’s Compile and Analyze Our ResultsLet’s Compile and Analyze Our ResultsLet’s Compile and Analyze Our Results
78 DFW-UPA Workshop
Conduct K-J Analysis of Wizard of OzConduct K-J Analysis of Wizard of OzConduct K-J Analysis of Wizard of OzConduct K-J Analysis of Wizard of Oz
• List all observations on stickiesList all observations on stickies
• Post them on a wallPost them on a wall
• Cluster into groupsCluster into groups
• Name the groupName the group
• Vote on each groupVote on each group
• If possible, make recommendationsIf possible, make recommendations
79 DFW-UPA Workshop
Conduct On Board Analysis of Lab TestingConduct On Board Analysis of Lab TestingConduct On Board Analysis of Lab TestingConduct On Board Analysis of Lab Testing
• Write issues on board after each participant (positive and Write issues on board after each participant (positive and negative)negative)
• Place checkmark when the issue is found by another userPlace checkmark when the issue is found by another user
• Rank issues at the end of the testRank issues at the end of the test
• If desired, we can think about solutions.If desired, we can think about solutions.
80 DFW-UPA Workshop
Wizard of Oz Testing on ChalkmarkWizard of Oz Testing on ChalkmarkWizard of Oz Testing on ChalkmarkWizard of Oz Testing on Chalkmark
81 DFW-UPA Workshop
Lab Testing of Four Sites (Sprouts)Lab Testing of Four Sites (Sprouts)Lab Testing of Four Sites (Sprouts)Lab Testing of Four Sites (Sprouts)
82 DFW-UPA Workshop
Lab Testing of Four Sites (Consumer Search)Lab Testing of Four Sites (Consumer Search)Lab Testing of Four Sites (Consumer Search)Lab Testing of Four Sites (Consumer Search)
83 DFW-UPA Workshop
Lab Testing of Four Sites (Broccoli.com)Lab Testing of Four Sites (Broccoli.com)Lab Testing of Four Sites (Broccoli.com)Lab Testing of Four Sites (Broccoli.com)
84 DFW-UPA Workshop
Lab Testing of Four Sites (Casa MaLab Testing of Four Sites (Casa Maňňana)ana)Lab Testing of Four Sites (Casa MaLab Testing of Four Sites (Casa Maňňana)ana)
85 DFW-UPA Workshop
Question & AnswerQuestion & Answer