85
ternet? That thing is still around? -- Homer Simpson Usability Boot Camp Usability Boot Camp Brian Sullivan DFW-UPA Workshop December 5, 2009

Usability Testing Workshop

  • View
    5.075

  • Download
    5

Embed Size (px)

DESCRIPTION

Usability Testing Workshop heled by DFW-UPA. Spotlights methods and tools, including Wizard of OZ style and lab-based testing. Demos of UserZoom and Morae were also done.

Citation preview

Page 1: Usability Testing Workshop

The Internet? That thing is still around? -- Homer Simpson

Usability Boot Camp Usability Boot Camp

Brian SullivanDFW-UPA Workshop

December 5, 2009

Page 2: Usability Testing Workshop

2 DFW-UPA Workshop

IntroductionIntroduction

Page 3: Usability Testing Workshop

3 DFW-UPA Workshop

What is Usability Testing?What is Usability Testing?What is Usability Testing?What is Usability Testing?

Usability testing involves:Usability testing involves:

1.1. You Watch CustomersYou Watch Customers

2.2. They Perform TasksThey Perform Tasks

3.3. You Note Their ProblemsYou Note Their Problems

4.4. Making RecommendationsMaking Recommendations

5.5. Rinse and repeat (i.e. iterate)Rinse and repeat (i.e. iterate)

Jakob Nielsen’s Definition of Usability

Usability is a quality attribute that assesses how easy user interfaces are to use. The word "usability" also refers to methods for improving ease-of-use during the design process.

Page 4: Usability Testing Workshop

4 DFW-UPA Workshop

Cast of Characters: Three Main RolesCast of Characters: Three Main RolesCast of Characters: Three Main RolesCast of Characters: Three Main Roles

• Facilitator:Facilitator: This This person oversees the person oversees the entire test process—entire test process—plan, test, and report.plan, test, and report.

• Participant:Participant: Actual or Actual or potential customers. potential customers. Representative users Representative users (marketing, designers) (marketing, designers) should not be used.should not be used.

• Observer:Observer: Records Records events as they occur. events as they occur. Limits interaction with Limits interaction with the customer. Does the customer. Does contribute to the contribute to the report.report.

In today’s workshop, you will play all three roles!

Page 5: Usability Testing Workshop

5 DFW-UPA Workshop

Benefits of Usability Testing: Feedback from Real CustomersBenefits of Usability Testing: Feedback from Real CustomersBenefits of Usability Testing: Feedback from Real CustomersBenefits of Usability Testing: Feedback from Real Customers

• If your website is If your website is difficult to use, difficult to use, customers customers leaveleave..

• If they get lost in If they get lost in your website, your website, customers customers leaveleave..

• If a customer can’t If a customer can’t FINDFIND your product, your product, they can’t they can’t BUYBUY it. it.….….Then, they Then, they leaveleave!!

Real feedback directly from customers that use the product, which provides data—not opinions.

Page 6: Usability Testing Workshop

6 DFW-UPA Workshop

Benefits of Usability Testing: Informed Design DecisionsBenefits of Usability Testing: Informed Design DecisionsBenefits of Usability Testing: Informed Design DecisionsBenefits of Usability Testing: Informed Design Decisions

Design A Design B

We encountered these results from a Usability Test:

• Average Time to Book a Limo with Design A = 3:59

• Average Time to Book a Limo with Design B = 8:00

• 12 of 12 customers successfully booked the correct limo with Design A.

• 7 of 12 customers successfully booked the correct limo with Design B.

• All users were TRAINED and currently used Design B.

Page 7: Usability Testing Workshop

7 DFW-UPA Workshop

Benefits of Usability Testing: Mitigate Risk (Also, Rewarded)Benefits of Usability Testing: Mitigate Risk (Also, Rewarded)Benefits of Usability Testing: Mitigate Risk (Also, Rewarded)Benefits of Usability Testing: Mitigate Risk (Also, Rewarded)

For years, companies have been performing usability testing. They just call it a product release.

Teams that do usability testing before a release mitigate risk. Plus, they are rewarded. (ex: $300 Million Button)

Page 8: Usability Testing Workshop

8 DFW-UPA Workshop

Benefits of Usability Testing: Saves Time and MoneyBenefits of Usability Testing: Saves Time and MoneyBenefits of Usability Testing: Saves Time and MoneyBenefits of Usability Testing: Saves Time and Money

# of users

X # of uses per day

X # of days per year

X Average hourly wage of user

X Increase efficiency (in hours)

Annual Savings in Productivity

• Randy Bias and Deborah Mayhew’s book Randy Bias and Deborah Mayhew’s book provides hundreds of formulas to show the provides hundreds of formulas to show the return on investment for usability testing.return on investment for usability testing.

• Here is one formula:Here is one formula:

Using the previous example, a switch to Design B saves over $2 million USD.

Page 9: Usability Testing Workshop

9 DFW-UPA Workshop

Benefits of Usability Testing: Makes Lots of MoneyBenefits of Usability Testing: Makes Lots of MoneyBenefits of Usability Testing: Makes Lots of MoneyBenefits of Usability Testing: Makes Lots of Money

Jared Spool

The $300,000,000 Fix (per Jared Spool):

In the above example, we have a form with two fields, two buttons, and one link. This form appeared on the bottom of a Shopping Cart. What are the problems?

Hint: Current and New Customers both had problems.

Page 10: Usability Testing Workshop

10 DFW-UPA Workshop

Benefits of Usability Testing: Makes Lots of MoneyBenefits of Usability Testing: Makes Lots of MoneyBenefits of Usability Testing: Makes Lots of MoneyBenefits of Usability Testing: Makes Lots of Money

Jared Spool

New Customers:

New customers did not want to register. They resisted clicking the Register button. New customers wanted to buy what was in their shopping cart. They did not want a relationship. Most wanted to leave. As it turns out, customers are not required to register—it is optional.

Page 11: Usability Testing Workshop

11 DFW-UPA Workshop

Benefits of Usability Testing: Makes Lots of MoneyBenefits of Usability Testing: Makes Lots of MoneyBenefits of Usability Testing: Makes Lots of MoneyBenefits of Usability Testing: Makes Lots of Money

Jared Spool

Existing Customers:

Existing customers couldn’t remember their exact email and/or password. They clicked the “Forgot Password” link. Help Desk logs revealed this link was clicked about 167,000 each day! According to Spool, 75% of these people didn’t make a purchase. They did leave!

Page 12: Usability Testing Workshop

12 DFW-UPA Workshop

Benefits of Usability Testing: Makes Lots of MoneyBenefits of Usability Testing: Makes Lots of MoneyBenefits of Usability Testing: Makes Lots of MoneyBenefits of Usability Testing: Makes Lots of Money

Jared Spool

Jared Spool’s $300,000,000 Fix:

Make it obvious. The results: The number of customers purchasing went up by 45%. The extra purchases resulted in an extra $15,000,000 the first month. For the first year, the site saw an additional $300,000,000.

$ $

Page 13: Usability Testing Workshop

13 DFW-UPA Workshop

Usability Methods and ToolsUsability Methods and Tools

Page 14: Usability Testing Workshop

14 DFW-UPA Workshop

Two Primary Methods of Data CollectionTwo Primary Methods of Data CollectionTwo Primary Methods of Data CollectionTwo Primary Methods of Data Collection

Direct IndirectField Studies SurveysContextual Inquiries QuestionnairesDesign Walkthroughs ProfilingComparative Analysis Web AnalyticsUsability Studies Diary Studies

Usability Methods Table

Page 15: Usability Testing Workshop

15 DFW-UPA Workshop

Quantitative and Qualitative TestingQuantitative and Qualitative TestingQuantitative and Qualitative TestingQuantitative and Qualitative Testing

Quantitave Usability Testing Qualitative Usability Testing

Provides opportunity to observe users as they perform tasks.

Provides a focus on quantitative measures, such how many people are clicking on an ad.

Based on expertise and judgment of people in the test (participant, facilitator, observer).

Based primarily on click-stream data and log files of real customers

Facilitator and observer(s) interprets the observed customer behavior and recommends solutions

Data must be interpreted from the click-stream or log files.

Data can be done with small sample sizes. Requires large sample sizes.

Page 16: Usability Testing Workshop

16 DFW-UPA Workshop

Quantitative Testing Tool (User Zoom Demo)Quantitative Testing Tool (User Zoom Demo)Quantitative Testing Tool (User Zoom Demo)Quantitative Testing Tool (User Zoom Demo)

• Quantitative Usability Testing has Quantitative Usability Testing has grew in the last decade.grew in the last decade.

• Many tools exist for quantitative Many tools exist for quantitative testing tools are out there.testing tools are out there.

• Keynote Systems has been Keynote Systems has been around for a long time now.around for a long time now.

• You may currently use or know You may currently use or know people that use Google Analytics.people that use Google Analytics.

• Chalkmark is something we will Chalkmark is something we will see later today.see later today.

• Lou Capone will give us a demo Lou Capone will give us a demo of UserZoom (30-minute demo).of UserZoom (30-minute demo).

Web IQ

Page 17: Usability Testing Workshop

17 DFW-UPA Workshop

Qualitative Testing Tools and MethodsQualitative Testing Tools and MethodsQualitative Testing Tools and MethodsQualitative Testing Tools and Methods

• Qualitative usability testing for Qualitative usability testing for the web has really matured in the web has really matured in the last 30 years.the last 30 years.

• Quantitative usability testing Quantitative usability testing tools has really matured in the tools has really matured in the last 10 years. last 10 years.

• Morae and OvoStudios both Morae and OvoStudios both offer mature solutions for offer mature solutions for usability professionals.usability professionals.

• IntuitionHQ is very new to the IntuitionHQ is very new to the market—I was just made aware market—I was just made aware of them this week.of them this week.

• We will use Morae today. We will use Morae today.

Page 18: Usability Testing Workshop

18 DFW-UPA Workshop

Three Types of Usability Tests (Qualitative Perspective)Three Types of Usability Tests (Qualitative Perspective)Three Types of Usability Tests (Qualitative Perspective)Three Types of Usability Tests (Qualitative Perspective)

Design Walkthrough Formative Usability Test Summative Usability TestConducted during prototype development to validate initial design decisions, determine stregnths and weaknesses, or to compare various designs.

Conducted during the design or development phases. Used to determine if a design objective is being met. Designs change throughout the test.

Conducted at the end of development--code is frozen. Used to validate that a product has accomplished its intended goals.

Example: Which of the two new designs best meets the customer's expectations? Which design was most preferred by the customers?

Example: Can users find and use a "mortgage calculator" to successfully enter the appropriate loan points on an application?

Example: Can users find and use a "mortgage calculator" to successfully enter the appropriate loan points in less than 1 minute?

Today’s workshop will focus on summative usability testing.

Page 19: Usability Testing Workshop

19 DFW-UPA Workshop

Basic Usability Testing ProcessBasic Usability Testing ProcessBasic Usability Testing ProcessBasic Usability Testing Process

Step 1: Planning Your Usability TestStep 1: Planning Your Usability Test

• Define what you are testingDefine what you are testing

• Define which customers you are testingDefine which customers you are testing

• Define what tasks you are testingDefine what tasks you are testing

• Recruit your customersRecruit your customers

• Write your usability scenarios and tasksWrite your usability scenarios and tasks

Step 2: Conducting Your Usability TestStep 2: Conducting Your Usability Test

• Conduct a pilot testConduct a pilot test

• Facilitate remaining testsFacilitate remaining tests

Step 3: Analyzing and Reporting Your Usability TestStep 3: Analyzing and Reporting Your Usability Test

• Compile resultsCompile results

• Make recommendationsMake recommendations

Page 20: Usability Testing Workshop

20 DFW-UPA Workshop

A Typical Scene Behind The Glass in a Usability TestA Typical Scene Behind The Glass in a Usability TestA Typical Scene Behind The Glass in a Usability TestA Typical Scene Behind The Glass in a Usability Test

Page 21: Usability Testing Workshop

21 DFW-UPA Workshop

Step 1: Planning Your Usability TestStep 1: Planning Your Usability Test

Page 22: Usability Testing Workshop

22 DFW-UPA Workshop

Basic Usability Testing ProcessBasic Usability Testing ProcessBasic Usability Testing ProcessBasic Usability Testing Process

Step 1: Planning Your Usability TestStep 1: Planning Your Usability Test

• Define what you are testingDefine what you are testing

• Define which customers you are testingDefine which customers you are testing

• Define what tasks you are testingDefine what tasks you are testing

• Recruit your customersRecruit your customers

• Write your usability scenarios and tasksWrite your usability scenarios and tasks

Time = Money:

Most usability tests are less than 90 minutes. Plus, you are taking your customer’s time. Put first things first—what needs testing!!!

(Activities)

(People)

(Context)

Page 23: Usability Testing Workshop

23 DFW-UPA Workshop

A Simplified Version of Activities-People-ContextA Simplified Version of Activities-People-ContextA Simplified Version of Activities-People-ContextA Simplified Version of Activities-People-Context

Picture from Stephen Anderson

Somebody (person) does something (activity) some where or some how (context).

This model is Robert Bailey’s Human Performance Model. Let’s use this model to quickly plan out a test!

Page 24: Usability Testing Workshop

24 DFW-UPA Workshop

Here is the Royal Caribbean International WebsiteHere is the Royal Caribbean International WebsiteHere is the Royal Caribbean International WebsiteHere is the Royal Caribbean International Website

Page 25: Usability Testing Workshop

25 DFW-UPA Workshop

Describe Activities That Might Be Done…..Describe Activities That Might Be Done…..Describe Activities That Might Be Done…..Describe Activities That Might Be Done…..

Picture from Stephen Anderson

• Create a group reservationCreate a group reservation

• Plan an event (ex: wedding)Plan an event (ex: wedding)

• Update profile (ex: Passport)Update profile (ex: Passport)

• Review Sailing RouteReview Sailing Route

• Make a second paymentMake a second payment

• Move to a different shipMove to a different ship

• Upgrade to a new cabinUpgrade to a new cabin

• Buy an excursion dealBuy an excursion deal

Page 26: Usability Testing Workshop

26 DFW-UPA Workshop

Describe People That Might Be Using the Site…..Describe People That Might Be Using the Site…..Describe People That Might Be Using the Site…..Describe People That Might Be Using the Site…..

Picture from Stephen Anderson

• Captain of the ShipCaptain of the Ship

• Group Traveler Group Traveler

• Senior CitizenSenior Citizen

• College StudentCollege Student

• Potential customerPotential customer

• Travel AgentTravel Agent

• Special Needs TravelerSpecial Needs Traveler

• Worker (on the Ship)Worker (on the Ship)

Page 27: Usability Testing Workshop

27 DFW-UPA Workshop

Describe the Context of Use…..Describe the Context of Use…..Describe the Context of Use…..Describe the Context of Use…..

Picture from Stephen Anderson

• Company websiteCompany website

• IVR systemIVR system

• Kiosk (on the ship)Kiosk (on the ship)

• Printout (ex: Itinerary)Printout (ex: Itinerary)

• Social MediaSocial Media

• Email/text messagingEmail/text messaging

• Voice MailVoice Mail

• TV, Radio, DVRTV, Radio, DVR

• Inside or outsideInside or outside

• Public or privatePublic or private

• Noisy or quietNoisy or quiet

• Legal regulationsLegal regulations

• Social attitudesSocial attitudes

• Company goalsCompany goals

Page 28: Usability Testing Workshop

28 DFW-UPA Workshop

It Is Not Too Complicated…..It Is Not Too Complicated…..It Is Not Too Complicated…..It Is Not Too Complicated…..

• People = Usability ParticipantsPeople = Usability Participants

• Context = Test ScenariosContext = Test Scenarios

• Activities = Usability TasksActivities = Usability Tasks

Page 29: Usability Testing Workshop

29 DFW-UPA Workshop

Activities: What Concerns Does Your Product Team Have?Activities: What Concerns Does Your Product Team Have?Activities: What Concerns Does Your Product Team Have?Activities: What Concerns Does Your Product Team Have?

Ask the project team to list out Ask the project team to list out their overall concerns with their their overall concerns with their product. Let’s go back to the product. Let’s go back to the Royal Caribbean International.Royal Caribbean International.

Overall Concerns:Overall Concerns:

1.1. Ease of learningEase of learning

2.2. Ease of useEase of use

3.3. Ease of navigationEase of navigation

4.4. Usefulness of HelpUsefulness of Help

5.5. Recovering for errorsRecovering for errors

Specific Concerns:Specific Concerns:

1.1. Booking Group ReservationsBooking Group Reservations

2.2. Splitting Group PaymentsSplitting Group Payments

3.3. Availability of Group CabinsAvailability of Group Cabins

Page 30: Usability Testing Workshop

30 DFW-UPA Workshop

People: Who are the Target Customers?People: Who are the Target Customers?People: Who are the Target Customers?People: Who are the Target Customers?

Ask the project team. They tend to Ask the project team. They tend to focus on a focus on a singlesingle user. user.

Frame It This Way:Frame It This Way:

1.1. Who are primary customers?Who are primary customers? (Define their characteristics.) (Define their characteristics.)

2.2. Is there another group?Is there another group? (How are they like 1 (How are they like 1stst group?) group?) (How are thy different?) (How are thy different?)

3.3. Is there another group? Is there another group? (How are they like both groups?) (How are they like both groups?)

(How are they different?) (How are they different?)

Travel Agent Event Planner

Page 31: Usability Testing Workshop

31 DFW-UPA Workshop

Context: Where and How are People Going to Use It?Context: Where and How are People Going to Use It?Context: Where and How are People Going to Use It?Context: Where and How are People Going to Use It?

Ask the team. Usually, the team Ask the team. Usually, the team focuses on the delivery device. focuses on the delivery device. They will need help with the other They will need help with the other contextual factors.contextual factors.

Frame It This Way:Frame It This Way:

1.1. What device(s) is it on?What device(s) is it on?

2.2. Is inside or outside?Is inside or outside?

3.3. Is it public or private?Is it public or private?

4.4. Is it work or leisure-related?Is it work or leisure-related?

5.5. Is it noisy or quiet?Is it noisy or quiet?

6.6. Any legal regulations?Any legal regulations?

Page 32: Usability Testing Workshop

32 DFW-UPA Workshop

It Is Not Too Complicated…..It Is Not Too Complicated…..Just Ask!!!Just Ask!!!It Is Not Too Complicated…..It Is Not Too Complicated…..Just Ask!!!Just Ask!!!

• People = Usability ParticipantsPeople = Usability Participants

• Context = Test ScenariosContext = Test Scenarios

• Activities = Usability TasksActivities = Usability Tasks

Page 33: Usability Testing Workshop

33 DFW-UPA Workshop

With People-Context-Activities Defined, Data Can Be CollectedWith People-Context-Activities Defined, Data Can Be CollectedWith People-Context-Activities Defined, Data Can Be CollectedWith People-Context-Activities Defined, Data Can Be Collected

Performance Data:

• Average Task Completion

• Average Time on Task

• # of clicks to Help

• # of omitted steps

• # of errors made

• # of hints

Subjective Data:

• Prefer Design A or B

• Design Suggestions

• Task Ease Rating

• Satisfaction Rating

• User Comments

• Ranking Features

Performance and subjective data are equally valuable. We will come back to data collection in a few minutes.

Page 34: Usability Testing Workshop

34 DFW-UPA Workshop

Basic Usability Testing Process—RecrutingBasic Usability Testing Process—RecrutingBasic Usability Testing Process—RecrutingBasic Usability Testing Process—Recruting

Step 1: Planning Your Usability TestStep 1: Planning Your Usability Test

• Define what you are testingDefine what you are testing

• Define which customers you are testingDefine which customers you are testing

• Define what tasks you are testingDefine what tasks you are testing

• Recruit your customersRecruit your customers

• Write your usability scenarios and tasksWrite your usability scenarios and tasks

(Activities)

(People)

(Context)

Page 35: Usability Testing Workshop

35 DFW-UPA Workshop

Number of Participants DebateNumber of Participants DebateNumber of Participants DebateNumber of Participants Debate

Page 36: Usability Testing Workshop

36 DFW-UPA Workshop

Number of Participants Debate (Sources) Number of Participants Debate (Sources) Number of Participants Debate (Sources) Number of Participants Debate (Sources)

Jakob Nielsen (circa 1993)

Five users will uncover about 85% of the usability issues of a product. Virzi backs up this claim around the same time.

Jared Spool (circa 2000)

Five users revealed only 35% of the usability issues of a product. This claim is backed up by Schroeder.

Laura Faulkner (2003)

- Five users uncovered 85% of usability issues on average.- But, the percent ranged from 55-100% (confidence interval).- Ten users uncovered 95% with a lower bound of 82%.

Page 37: Usability Testing Workshop

37 DFW-UPA Workshop

Number of Participants (Brian Sullivan, 2009)Number of Participants (Brian Sullivan, 2009)Number of Participants (Brian Sullivan, 2009)Number of Participants (Brian Sullivan, 2009)

General Rules:General Rules:

• Test with 6 customers Test with 6 customers (per group)(per group)—if —if time permits, use 8 customers.time permits, use 8 customers.

• More customers = more reliable resultsMore customers = more reliable results

• More customers = fewer new resultsMore customers = fewer new results

• For larger samples, do For larger samples, do multiplemultiple rounds rounds of smaller tests (rather than 1 large UT)of smaller tests (rather than 1 large UT)

Base Your Number On:Base Your Number On:

• Importance of projectImportance of project

• Complexity of the productComplexity of the product

• Experience of testing teamExperience of testing team

• Expected variety of user reactionsExpected variety of user reactions

Page 38: Usability Testing Workshop

38 DFW-UPA Workshop

Recruiting ParticipantsRecruiting ParticipantsRecruiting ParticipantsRecruiting Participants

Recruiting:Recruiting:

• In-house sourcesIn-house sources

• Recruiting agencyRecruiting agency

Incentives:Incentives:

• Gift checks ($100 per session)Gift checks ($100 per session)

• Food or gift cardsFood or gift cards

Scheduling:Scheduling:

• Slot timesSlot times

• Breaks between participantsBreaks between participants

• Backups, if neededBackups, if needed

• Floaters, if needed Floaters, if needed

Page 39: Usability Testing Workshop

39 DFW-UPA Workshop

Selecting the Right Customers Selecting the Right Customers Selecting the Right Customers Selecting the Right Customers

User profiles:User profiles:

• Potential CustomerPotential Customer

• New CustomerNew Customer

• Returning CustomerReturning Customer

• Primary or tertiary customersPrimary or tertiary customers

Characteristics:Characteristics:

• Job responsibilities (not their title)Job responsibilities (not their title)

• Years experienceYears experience

• Expertise level (novice, intermediate, expert)Expertise level (novice, intermediate, expert)

• Specific skills (example: heart surgeons rather MDs or nurses)Specific skills (example: heart surgeons rather MDs or nurses)

• Geographic locationGeographic location

Source: David Armano

Page 40: Usability Testing Workshop

40 DFW-UPA Workshop

Your Recruiting is Only As Good As the ScreenerYour Recruiting is Only As Good As the ScreenerYour Recruiting is Only As Good As the ScreenerYour Recruiting is Only As Good As the Screener

Steve Krug has said this:Steve Krug has said this:

I respectfully disagree. Because…I respectfully disagree. Because…

• Your test is a Your test is a waste of timewaste of time, if you , if you recruit the recruit the wrongwrong people. people.

• Your usability test is Your usability test is dangerousdangerous, if , if you you act upon recommendationsact upon recommendations from a bad recruiting sample.from a bad recruiting sample.

You must You must screenscreen participants. participants.

For recruiting, grade on a curve! Grab a neighbor.

Page 41: Usability Testing Workshop

41 DFW-UPA Workshop

Basic Usability Testing Process—Write Out the TestBasic Usability Testing Process—Write Out the TestBasic Usability Testing Process—Write Out the TestBasic Usability Testing Process—Write Out the Test

Step1: Planning Your Usability TestStep1: Planning Your Usability Test

• Define what you are testingDefine what you are testing

• Define which customers you are testingDefine which customers you are testing

• Define what tasks you are testingDefine what tasks you are testing

• Recruit your customersRecruit your customers

• Write your usability scenarios and tasksWrite your usability scenarios and tasks

• People = Usability ParticipantsPeople = Usability Participants

• Context = Test ScenariosContext = Test Scenarios

• Activities = Usability TasksActivities = Usability Tasks

Page 42: Usability Testing Workshop

42 DFW-UPA Workshop

Scenarios and TasksScenarios and TasksScenarios and TasksScenarios and Tasks

Some experts distinguish between tasks and scenarios:Some experts distinguish between tasks and scenarios:

Scenarios:Scenarios:

• Provide the participant with motivation and contextProvide the participant with motivation and context

• Sets the stage or tells a storySets the stage or tells a story

• Increases understandingIncreases understanding

• Makes the situation more realisticMakes the situation more realistic

• Includes Includes taskstasks

Tasks:Tasks:

• Quick and to the pointQuick and to the point

• Do not need context to increase understandingDo not need context to increase understanding

Page 43: Usability Testing Workshop

43 DFW-UPA Workshop

Example of Scenario with TasksExample of Scenario with TasksExample of Scenario with TasksExample of Scenario with Tasks

Scenario #1:Scenario #1:

You want to book a sailing on Royal Caribbean International for next June You want to book a sailing on Royal Caribbean International for next June with your church group. The group is called “Saint Francis Summer 2010”. with your church group. The group is called “Saint Francis Summer 2010”. The group is selling out fast, so you want to book a cabin, which is close to The group is selling out fast, so you want to book a cabin, which is close to an elevator because your leg hurts from a recent injury.an elevator because your leg hurts from a recent injury.

• Open your browserOpen your browser

• Click the link labeled “Royal Caribbean”Click the link labeled “Royal Caribbean”

• Tell me the available cabins in the “Saint Francis Summer 2010” groupTell me the available cabins in the “Saint Francis Summer 2010” group

• Tell me a cabin number closest to an elevatorTell me a cabin number closest to an elevator

• Book the cabin the best suits your needsBook the cabin the best suits your needs

Page 44: Usability Testing Workshop

44 DFW-UPA Workshop

Practice Writing Scenarios and TasksPractice Writing Scenarios and TasksPractice Writing Scenarios and TasksPractice Writing Scenarios and Tasks

• We have two teams today.We have two teams today.--Team Cool--Team Cool--Team Ninja Assassins--Team Ninja Assassins

• Pair up with someone.Pair up with someone.

• Write a scenario with tasks. Write a scenario with tasks.

• We will pass out papers of We will pass out papers of some sample sites. some sample sites.

• You will use your script later You will use your script later today to run some tests.today to run some tests.

Page 45: Usability Testing Workshop

45 DFW-UPA Workshop

Step 2: Conducting Your TestStep 2: Conducting Your Test

Page 46: Usability Testing Workshop

46 DFW-UPA Workshop

Basic Usability Process—Step 2Basic Usability Process—Step 2Basic Usability Process—Step 2Basic Usability Process—Step 2

Step 1: Planning Your Usability TestStep 1: Planning Your Usability Test

• Define what you are testingDefine what you are testing

• Define which customers you are testingDefine which customers you are testing

• Define what tasks you are testingDefine what tasks you are testing

• Recruit your customersRecruit your customers

• Write your usability scenarios and tasksWrite your usability scenarios and tasks

Step 2: Conducting Your Usability TestStep 2: Conducting Your Usability Test

• Conduct a pilot testConduct a pilot test

• Facilitate remaining testsFacilitate remaining tests

Page 47: Usability Testing Workshop

47 DFW-UPA Workshop

Don’t Rely on Luck!!!Don’t Rely on Luck!!!Don’t Rely on Luck!!!Don’t Rely on Luck!!!

Page 48: Usability Testing Workshop

48 DFW-UPA Workshop

You Do Not Need a Usability Lab to Conduct a TestYou Do Not Need a Usability Lab to Conduct a TestYou Do Not Need a Usability Lab to Conduct a TestYou Do Not Need a Usability Lab to Conduct a Test

Page 49: Usability Testing Workshop

49 DFW-UPA Workshop

Test Side-by-Side in a Conference RoomTest Side-by-Side in a Conference RoomTest Side-by-Side in a Conference RoomTest Side-by-Side in a Conference Room

Page 50: Usability Testing Workshop

50 DFW-UPA Workshop

Test Wherever Your Customers Might BeTest Wherever Your Customers Might BeTest Wherever Your Customers Might BeTest Wherever Your Customers Might Be

I have done usability testing at Starbuck’s and a User Conference.

Page 51: Usability Testing Workshop

51 DFW-UPA Workshop

Remote Usability Testing Easier Than EverRemote Usability Testing Easier Than EverRemote Usability Testing Easier Than EverRemote Usability Testing Easier Than Ever

Page 52: Usability Testing Workshop

52 DFW-UPA Workshop

Morae is Portable, Affordable, and ………………UsableMorae is Portable, Affordable, and ………………UsableMorae is Portable, Affordable, and ………………UsableMorae is Portable, Affordable, and ………………Usable

Page 53: Usability Testing Workshop

53 DFW-UPA Workshop

Shhhhh….you don’t need to record, eitherShhhhh….you don’t need to record, eitherShhhhh….you don’t need to record, eitherShhhhh….you don’t need to record, either

Page 54: Usability Testing Workshop

54 DFW-UPA Workshop

Basic Usability Process—Step 2Basic Usability Process—Step 2Basic Usability Process—Step 2Basic Usability Process—Step 2

Step 1: Planning Your Usability TestStep 1: Planning Your Usability Test

• Define what you are testingDefine what you are testing

• Define which customers you are testingDefine which customers you are testing

• Define what tasks you are testingDefine what tasks you are testing

• Recruit your customersRecruit your customers

• Write your usability scenarios and tasksWrite your usability scenarios and tasks

Step 2: Conducting Your Usability TestStep 2: Conducting Your Usability Test

• Conduct a pilot testConduct a pilot test

• Facilitate remaining testsFacilitate remaining tests

Page 55: Usability Testing Workshop

55 DFW-UPA Workshop

Conduct a Pilot TestConduct a Pilot TestConduct a Pilot TestConduct a Pilot Test

Dry run of a complete usability testDry run of a complete usability test

• Include all test materialInclude all test material

• Focus on the test scenarios, not findingsFocus on the test scenarios, not findings

• At least, one participant should be a pilotAt least, one participant should be a pilot

• Conduct before test begins (one day prior)Conduct before test begins (one day prior)

• Make changes, if neededMake changes, if needed

• May need to test, againMay need to test, again

Page 56: Usability Testing Workshop

56 DFW-UPA Workshop

Basic Usability Process—Step 2Basic Usability Process—Step 2Basic Usability Process—Step 2Basic Usability Process—Step 2

Step 1: Planning Your Usability TestStep 1: Planning Your Usability Test

• Define what you are testingDefine what you are testing

• Define which customers you are testingDefine which customers you are testing

• Define what tasks you are testingDefine what tasks you are testing

• Recruit your customersRecruit your customers

• Write your usability scenarios and tasksWrite your usability scenarios and tasks

Step 2: Conducting Your Usability TestStep 2: Conducting Your Usability Test

• Conduct a pilot testConduct a pilot test

• Facilitate remaining testsFacilitate remaining tests

Page 57: Usability Testing Workshop

57 DFW-UPA Workshop

Again, Adopt a Workable ScheduleAgain, Adopt a Workable ScheduleAgain, Adopt a Workable ScheduleAgain, Adopt a Workable Schedule

Page 58: Usability Testing Workshop

58 DFW-UPA Workshop

Tips for Being a Good ModeratorTips for Being a Good ModeratorTips for Being a Good ModeratorTips for Being a Good Moderator

1.1. Know Your UCD Basics and Design GuidelinesKnow Your UCD Basics and Design Guidelines

2.2. Be a Quick LearnerBe a Quick Learner

3.3. Set an Instant Rapport with ParticipantsSet an Instant Rapport with Participants

4.4. Have an Excellent MemoryHave an Excellent Memory

5.5. Be a Good Listener (Use Active Listening)Be a Good Listener (Use Active Listening)

6.6. Don’t Stress Out About AmbiguityDon’t Stress Out About Ambiguity

7.7. Be Very FlexibleBe Very Flexible

8.8. Have a Long Attention SpanHave a Long Attention Span

9.9. Show Empathy (ie Be a People Person)Show Empathy (ie Be a People Person)

10.10. Thing “Big Picture”Thing “Big Picture”

11.11. Be a Good CommunicatorBe a Good Communicator

12.12. Be a Good Organizer and CoordinatorBe a Good Organizer and Coordinator

Page 59: Usability Testing Workshop

59 DFW-UPA Workshop

Tips for Getting the Most from ParticipantsTips for Getting the Most from ParticipantsTips for Getting the Most from ParticipantsTips for Getting the Most from Participants

• Start with an Easy Task To Build ConfidenceStart with an Easy Task To Build Confidence

• Choose the Right Format (remote, on-site UT)Choose the Right Format (remote, on-site UT)

• Sit Beside the Person Not Behind the GlassSit Beside the Person Not Behind the Glass

• Use “Think-Out-Loud” ProtocolUse “Think-Out-Loud” Protocol

• Give Them Time to Think It ThroughGive Them Time to Think It Through

• Offer Appropriate EncouragementOffer Appropriate Encouragement

• Lead Participants, Don’t Answer Question (Being an Enabler)Lead Participants, Don’t Answer Question (Being an Enabler)

• Don’t Act Knowledgeable (Treat Them As the Experts)Don’t Act Knowledgeable (Treat Them As the Experts)

• Don’t Get Too Involved in Data CollectionDon’t Get Too Involved in Data Collection

• Don’t Jump to ConclusionsDon’t Jump to Conclusions

• Don’t Solve Their Problems Immediately (ie act as a Help Desk)Don’t Solve Their Problems Immediately (ie act as a Help Desk)

Page 60: Usability Testing Workshop

60 DFW-UPA Workshop

Team Cool Starts With Wizard of Oz StyleTeam Cool Starts With Wizard of Oz StyleTeam Cool Starts With Wizard of Oz StyleTeam Cool Starts With Wizard of Oz Style

Wizard of Oz Style Testing:Wizard of Oz Style Testing:

• ModeratorModerator

• TesterTester

• Wizard (Person moves screen shots)Wizard (Person moves screen shots)

@suredoc (Keith Anderson) is your guide today.

NOTE: Keith is NOT the WWW – Wicked Witch of the West.

Page 61: Usability Testing Workshop

61 DFW-UPA Workshop

Rules for the Merry Old Land of Oz…..Rules for the Merry Old Land of Oz…..Rules for the Merry Old Land of Oz…..Rules for the Merry Old Land of Oz…..

Here are The Rules:Here are The Rules:

1.1. Get into Groups of three.Get into Groups of three.

2.2. Each person picks a role.Each person picks a role.

3.3. Wizard does not talk.Wizard does not talk.

4.4. Moderator analyzes TesterModerator analyzes Tester

5.5. Tester talks through the task.Tester talks through the task.

6.6. Complete the Test.Complete the Test.

7.7. Switch Roles.Switch Roles.

8.8. Find two new people.Find two new people.

9.9. Repeat Steps 1-6.Repeat Steps 1-6.

10.10. Compile results via K-J MethodCompile results via K-J Method (well do that later, though). (well do that later, though).

Page 62: Usability Testing Workshop

62 DFW-UPA Workshop

Team Ninja Assassins Will Be in Usability LabTeam Ninja Assassins Will Be in Usability LabTeam Ninja Assassins Will Be in Usability LabTeam Ninja Assassins Will Be in Usability Lab

These are the Test Stations:These are the Test Stations:

1.1. Two Waiting ParticipantsTwo Waiting Participants

2.2. One Tester in Test CellOne Tester in Test Cell

3.3. ModeratorModerator

4.4. Test Cell ObserverTest Cell Observer

5.5. Camera OperatorCamera Operator

6.6. TV OperatorTV Operator

7.7. Remote Viewer LoggerRemote Viewer Logger

8.8. Remote Viewer ManagerRemote Viewer Manager

9.9. Results Compiler (2 people)Results Compiler (2 people)

10.10. Two ObserversTwo Observers

11.11. Rotate between each stationRotate between each station

We’ll test four different sites today.

Page 63: Usability Testing Workshop

63 DFW-UPA Workshop

After 1.5 Hours, We Switch LocationsAfter 1.5 Hours, We Switch LocationsAfter 1.5 Hours, We Switch LocationsAfter 1.5 Hours, We Switch Locations

The Teams Switch:The Teams Switch:

• Team Cool in LabsTeam Cool in Labs

• Team Ninja Assassins Team Ninja Assassins (Wizard of Oz)(Wizard of Oz)

Page 64: Usability Testing Workshop

64 DFW-UPA Workshop

Step 3: Analyzing and ReportingStep 3: Analyzing and Reporting

Page 65: Usability Testing Workshop

65 DFW-UPA Workshop

Basic Usability Process—Home Stretch Now!!!Basic Usability Process—Home Stretch Now!!!Basic Usability Process—Home Stretch Now!!!Basic Usability Process—Home Stretch Now!!!

Step 1: Planning Your Usability TestStep 1: Planning Your Usability Test

• Define what you are testingDefine what you are testing

• Define which customers you are testingDefine which customers you are testing

• Define what tasks you are testingDefine what tasks you are testing

• Recruit your customersRecruit your customers

• Write your usability scenarios and tasksWrite your usability scenarios and tasks

Step 2: Conducting Your Usability TestStep 2: Conducting Your Usability Test

• Conduct a pilot testConduct a pilot test

• Facilitate remaining testsFacilitate remaining tests

Step 3: Analyzing and Reporting Your Usability TestStep 3: Analyzing and Reporting Your Usability Test

• Compile resultsCompile results

• Make recommendationsMake recommendations

Page 66: Usability Testing Workshop

66 DFW-UPA Workshop

Dilbert’s Boss Solves Your Customer ProblemDilbert’s Boss Solves Your Customer ProblemDilbert’s Boss Solves Your Customer ProblemDilbert’s Boss Solves Your Customer Problem

Page 67: Usability Testing Workshop

67 DFW-UPA Workshop

8 Steps to Analyze and Report Your Findings8 Steps to Analyze and Report Your Findings8 Steps to Analyze and Report Your Findings8 Steps to Analyze and Report Your Findings

1.1. Complete Your NotesComplete Your Notes

2.2. Summarize Your FindingsSummarize Your Findings

3.3. Compile and Tabulate DataCompile and Tabulate Data

4.4. Analyze Your FindingsAnalyze Your Findings

5.5. Prioritize Findings by Severity LevelPrioritize Findings by Severity Level

6.6. Create RecommendationsCreate Recommendations

7.7. Develop a Final Document or PresentationDevelop a Final Document or Presentation

8.8. Follow-up Follow-up

Page 68: Usability Testing Workshop

68 DFW-UPA Workshop

Step 1—Complete Your NotesStep 1—Complete Your NotesStep 1—Complete Your NotesStep 1—Complete Your Notes

• After each session, fill in any gaps.After each session, fill in any gaps.

• Basically, complete the short-hand used in the UT.Basically, complete the short-hand used in the UT.

• Mark any areas worth noting.Mark any areas worth noting.

• Use a whiteboard! It is really much easier, in the end.Use a whiteboard! It is really much easier, in the end.

Page 69: Usability Testing Workshop

69 DFW-UPA Workshop

Step 2—Summarize FindingsStep 2—Summarize FindingsStep 2—Summarize FindingsStep 2—Summarize Findings

• After each session, create a list of the major issues.After each session, create a list of the major issues.

• Organize by Issue and Participant.Organize by Issue and Participant.

• Do Not Start Solution Finding Here!Do Not Start Solution Finding Here!

• Below is an Example of How to Summarize:Below is an Example of How to Summarize:

Number Issue Description Customers1 Unable to locate Group for RCI Sailing from Home Page P1, P2, P3, P42 Credit card payment summary incomplete (no SEC) P2, P63 Rate Category page does not have sort capability P1, P5, P74 Bad error message on Cabin Swap page. P3, P6

Page 70: Usability Testing Workshop

70 DFW-UPA Workshop

Step 3—Compile and Tabulate DataStep 3—Compile and Tabulate DataStep 3—Compile and Tabulate DataStep 3—Compile and Tabulate Data

• Spreadsheet work best hereSpreadsheet work best here

• Do not work on solutions, yetDo not work on solutions, yet

• Try to find the real problemTry to find the real problem

Page 71: Usability Testing Workshop

71 DFW-UPA Workshop

Step 4—Analyze FindingsStep 4—Analyze FindingsStep 4—Analyze FindingsStep 4—Analyze Findings

• Which tasks were simple?Which tasks were simple?

• Which tasks were problematic?Which tasks were problematic?

• Which participants had issues? (Look at it by group.)Which participants had issues? (Look at it by group.)

• Speculate on solutions.Speculate on solutions.

• Use Balsamiq to sketch out recommendations.Use Balsamiq to sketch out recommendations.

Sketched in Balsamiq in less than 5 minutes.

Page 72: Usability Testing Workshop

72 DFW-UPA Workshop

Step 5—Prioritize FindingsStep 5—Prioritize FindingsStep 5—Prioritize FindingsStep 5—Prioritize Findings

• Severity 1 = Showstopper. Unless corrected allSeverity 1 = Showstopper. Unless corrected allusers will essentially have a problem.users will essentially have a problem.

• Severity 2 = High Impact. Significant impact to usability or Severity 2 = High Impact. Significant impact to usability or performance. High customer annoyance and frustration.performance. High customer annoyance and frustration.

• Severity 3 = Medium Impact. Some users experience degraded Severity 3 = Medium Impact. Some users experience degraded performance or frustration. Help Desk calls possible.performance or frustration. Help Desk calls possible.

• Severity 4 = Low Impact. Probably will not have a significant Severity 4 = Low Impact. Probably will not have a significant impact to user performance. Might be cosmetic in nature.impact to user performance. Might be cosmetic in nature.

Page 73: Usability Testing Workshop

73 DFW-UPA Workshop

Step 6—Create RecommendationsStep 6—Create RecommendationsStep 6—Create RecommendationsStep 6—Create Recommendations

• Start with Positive FindingsStart with Positive Findings

• Remind the team of people-activities-contextRemind the team of people-activities-context

• Be a problem solverBe a problem solver

• Present Issues by Severity Level (helps to prioritize)Present Issues by Severity Level (helps to prioritize)

• It great to do a MOCKUP hereIt great to do a MOCKUP here

• Tie recommendation back to USER PERSONAS, tooTie recommendation back to USER PERSONAS, too

Page 74: Usability Testing Workshop

74 DFW-UPA Workshop

Nobody reads a “BIG, HONKING” report.

Page 75: Usability Testing Workshop

75 DFW-UPA Workshop

Step 7—Create Final Document or PresentationStep 7—Create Final Document or PresentationStep 7—Create Final Document or PresentationStep 7—Create Final Document or Presentation

• Consider the audienceConsider the audience

• Some teams just want an Excel spreadsheetSome teams just want an Excel spreadsheet

• Provide enough detail (executive summary v. full report)Provide enough detail (executive summary v. full report)

• Create a distribution list for key stakeholdersCreate a distribution list for key stakeholders

• Purpose (identify issues, prioritize them, make recommendations)Purpose (identify issues, prioritize them, make recommendations)

Page 76: Usability Testing Workshop

76 DFW-UPA Workshop

Step 8—Follow-upStep 8—Follow-upStep 8—Follow-upStep 8—Follow-up

• Always Provide Usability Next StepsAlways Provide Usability Next Steps

• Usability Needs to Be Part of the Usability Needs to Be Part of the Development/Design CycleDevelopment/Design Cycle

• Invite People to Other StudiesInvite People to Other Studies

• Create an Action PlanCreate an Action Plan

Page 77: Usability Testing Workshop

77 DFW-UPA Workshop

Let’s Compile and Analyze Our ResultsLet’s Compile and Analyze Our ResultsLet’s Compile and Analyze Our ResultsLet’s Compile and Analyze Our Results

Page 78: Usability Testing Workshop

78 DFW-UPA Workshop

Conduct K-J Analysis of Wizard of OzConduct K-J Analysis of Wizard of OzConduct K-J Analysis of Wizard of OzConduct K-J Analysis of Wizard of Oz

• List all observations on stickiesList all observations on stickies

• Post them on a wallPost them on a wall

• Cluster into groupsCluster into groups

• Name the groupName the group

• Vote on each groupVote on each group

• If possible, make recommendationsIf possible, make recommendations

Page 79: Usability Testing Workshop

79 DFW-UPA Workshop

Conduct On Board Analysis of Lab TestingConduct On Board Analysis of Lab TestingConduct On Board Analysis of Lab TestingConduct On Board Analysis of Lab Testing

• Write issues on board after each participant (positive and Write issues on board after each participant (positive and negative)negative)

• Place checkmark when the issue is found by another userPlace checkmark when the issue is found by another user

• Rank issues at the end of the testRank issues at the end of the test

• If desired, we can think about solutions.If desired, we can think about solutions.

Page 80: Usability Testing Workshop

80 DFW-UPA Workshop

Wizard of Oz Testing on ChalkmarkWizard of Oz Testing on ChalkmarkWizard of Oz Testing on ChalkmarkWizard of Oz Testing on Chalkmark

Page 81: Usability Testing Workshop

81 DFW-UPA Workshop

Lab Testing of Four Sites (Sprouts)Lab Testing of Four Sites (Sprouts)Lab Testing of Four Sites (Sprouts)Lab Testing of Four Sites (Sprouts)

Page 82: Usability Testing Workshop

82 DFW-UPA Workshop

Lab Testing of Four Sites (Consumer Search)Lab Testing of Four Sites (Consumer Search)Lab Testing of Four Sites (Consumer Search)Lab Testing of Four Sites (Consumer Search)

Page 83: Usability Testing Workshop

83 DFW-UPA Workshop

Lab Testing of Four Sites (Broccoli.com)Lab Testing of Four Sites (Broccoli.com)Lab Testing of Four Sites (Broccoli.com)Lab Testing of Four Sites (Broccoli.com)

Page 84: Usability Testing Workshop

84 DFW-UPA Workshop

Lab Testing of Four Sites (Casa MaLab Testing of Four Sites (Casa Maňňana)ana)Lab Testing of Four Sites (Casa MaLab Testing of Four Sites (Casa Maňňana)ana)

Page 85: Usability Testing Workshop

85 DFW-UPA Workshop

Question & AnswerQuestion & Answer