Lecture #7COLLECTION, ANALYSIS AND PRESENTATION OF DATA FROM USABILITY TESTS

Y39TUR Spring 2011

Testování uživatelského rozhraní

TUR 2011

(2)

Today Sběr a analýza dat při uživatelských testech v laboratoři + Sběr a analýza dat při uživatelských testech v laboratoři +

Zpracování výsledků kvalitativních testůZpracování výsledků kvalitativních testů– priprava testupriprava testu

• co je potreba pripravit, jak si muzu pomoct ruznymi nastrojico je potreba pripravit, jak si muzu pomoct ruznymi nastroji– sber dat pri testusber dat pri testu

• jak muzu logovat (xls, Morae), na co se pri logovani zameritjak muzu logovat (xls, Morae), na co se pri logovani zamerit• jaka dalsi data se daji logovat, jaky maji uplatneni/vyuziti v analyzejaka dalsi data se daji logovat, jaky maji uplatneni/vyuziti v analyze

– analyza dat po testuanalyza dat po testu• co je dulezite hledat v datech z kvalitativnich testuco je dulezite hledat v datech z kvalitativnich testu• jak si pomoct nastrojijak si pomoct nastroji

MaterialyMaterialy– hlavne z lekce 8, neco malo z lekce 5hlavne z lekce 8, neco malo z lekce 5

Casove rozvrzeniCasove rozvrzeni– Obecny uvod – 10 minutObecny uvod – 10 minut– Priprava testu – 10 minutPriprava testu – 10 minut– Sber dat pri testu – 20 minutSber dat pri testu – 20 minut– Analyza dat po testu – 20 minutAnalyza dat po testu – 20 minut

DATA COLLECTION

TUR 2010

(4)

Standard Method of Usability Test

User gets an assignment The observer observes Problem

– It is hard to remember all the user activities

S. Greenberg

TUR 2010

(5)

Recording the Observation

Video Recording– Can see what the user does– Typical are multi-camera setup

• One camera records the screen

• One camera records the participant face and body

• Problem of synchronization

– “Big Brother problem”

Audio Recording– Can hear what the user does– Good for think-aloud protocol– Audio recording is very important

TUR 2010

(6)

Think-aloud Protocol

The participants vocalizes their thoughts– What they are trying to do– Why they are taking those actions– How they interpret the reactions of the system– What are their suggestions

Hmm, what does this do? I’ll try it… Ooops, now what

happened?

S. Greenberg

TUR 2010

(7)

Think-aloud Protocol

Pros: Provides insights to the user’s thinking Most common method in the Usability Engineering Cons:

– Can alter the mental process of the user– Unnatural– It’s difficult to speak when focusing on the task

TUR 2010

(8)

Example of Hints (Think-aloud)

“Please, speak on” “Please tell me what you think.” “Please tell me what you are trying to do.” “Are you searching for something in particular?” “What do you think that it’s going to happen now?” “What did you mean by that?”

Adapted from Jake Wobbrock

TUR 2010

(9)

Application Support – Morae I

Audio/Video Recording- Morae Recorder– Allows up to 2 video sources– Video recording can take some processing resources of the PC

TUR 2010

(10)

Video Annotations - Datalogging

Processing of AV recording takes 4-8 times the length of the recording– E.g.: 6 participants, 1h each video = minimum 24h! ~ 2-4 days

Typical solution:– While capturing: Annotate video– While processing: Focus on important parts only

Problems of video annotations– It is hard to keep up with the tempo of what's going on – You may miss important interaction during annotation – Annotations contain data from different categories

• e.g. user opinions, behavioral observations and demographic notes

Improvement - Markers

Basic Marker DefinitionBasic Marker Definition

Code Definition Description

X Usability problem

D Duplicate usability problem (described earlier)

V Video highlight — an "Ah-ha!" moment

C Comment (general comment by participant)

P Positive opinion expressed by participant

N Negative opinion expressed by participant

B Bug

F Facial reaction (e.g. surprise)

H Help or documentation accessed

A Assist from moderator

G Gives up or wrongly thinks finished

I Design idea (design insight by logger)

M Misc. (general observation by logger)

TUR 2011

(11)

TUR 2010

(12)

Example of Video Annotations

An actual usability test Website

– Web portal of a university

Task #8 (10 tasks total)– “Find information on life-long education program at the

Faculty of Architecture.”

TUR 2010

(13)

Example of Video Annotations

Time Note Task ID Marker

15:06 Session Starts M[some lines omitted]

15:16 Tries to find “Info for the Students” #8 M

15:17 Can’t find in the left-hand menu X

Goes systematically through all links in this menu X

Wishes to use full text search, does not how C

Found link “Alumni” M

15:20 Life-long Education link NOT FOUND X

15:21 Found link “For applicants” M

FOUND link “Life-long education” M[some lines omitted]

15:49 Session Ends M

Video Annotations Tips

Marker categories are difficult to remember– Start with subset of the markers– Add more markers when the technique became familiar

Multiple observers:– Can increase number of found usability problems– Can put more information into annotations

TUR 2010

(14)

TUR 2010

(15)

Other Collected Data

Low level interaction data– Mouse clicks, mouse movements– Key presses– Can be used for location of interesting parts in video recording

Application data– Generally any information that may be interesting or valuable during

evaluation– Typically requires some hooks, API or functionality inside the

applications– Examples

• Web - url, rendered page, Basket content• Mobile – GPS position, content of search field

Eye-tracking

TUR 2010

(15)

TUR 2010

(16)

After-test Activities

It is important to find out what the participants think Do they find the test easy? Difficult? Conditions of the test: Good, bad? Other comments?

TUR 2010

(17)

Interview

Good for finding out specific problems– “Set up” the question to match the context– Can focus on the problems as they show up during the

interview– Good for research studies (open-ended questioning)– Leads to specific suggestions

Problems– Statements are subjective– Time-consuming– The interviewer can easily affect the results

TUR 2010

(18)

How to carry out an interview

Plan the list of basic questions– Several good questions can start an interview

• (But avoid leading questions.)

– These questions can focus the interview.– Can be based on the results received from the

observation.

TUR 2010

(19)

Retrospective testing interviews

Post-observation interview– Observe (make the test)– Capture a video recording– User watches the recording and comments on in

• Explains an unclear behavior during the test

• Great for interpretation of the post-test interview

• Avoids misinterpretation

• Can identify particular improvements

Do you know

why you never tried that

option?I didn’t see it. Why don’t you make it look

like a button?

S. Greenberg

TUR 2010

(20)

Critical Situations during an Interview

People speak about problems that have emerged– People vividly speak about marginal problems

• Important only for them

– Problem has emerged that wasnot captured during the test

I can never get my figures in the right

place. Its really annoying. I spent

hours on it and I had to…

Tell me about the last big problem you

had with Word

S. Greenberg

TUR 2010

(21)

Data Collection from Interview

Same as for usability test– Audio/video recording– Annotations– Optionally other data (low level, application)

TUR 2010

(22)

Post Test Questionnaire

Allows easier data collection compared to interview

Typically uses Likert scale (1-5) Allows also Yes-No questions Allows also open ended questions (limited)

Example of Post Test Questionnaire

An actual usability test Website

– Web portal of a university

TUR 2011

(23)

TUR 2010

(24)

Application support – Morae III

Autopilot mode - Morae Recorder– Instructions of the tasks are provided online– Pre/Post test questionnaires are filled online– Subjective difficulty rating after each task is filled online– Data are stored into recordings for each participant

Autopilot disadvantages– Autopilot interface can hide interface of the tested application– You may loose data when recording is corrupted

DATA ANALYSIS

Factors that impact data analysisFactors that impact data analysis

Factors suggesting formal data presentation: Factors suggesting formal data presentation: – Summative studiesSummative studies– Lower level of support in company UCD processesLower level of support in company UCD processes– External audienceExternal audience– NovicesNovices

Factors suggesting informal data presentation:Factors suggesting informal data presentation:– Formative studiesFormative studies– Higher level of support in company UCD processesHigher level of support in company UCD processes– Internal audienceInternal audience– ExpertsExperts

TUR 2011

(26)

Source: UPA 2006 Idea Markets - Analyzing usability study results: Is it magic behind the curtain?Activator: Emma J. Rose, University of Washington

Strategies for Usability Data AnalysisStrategies for Usability Data Analysis

Formal vs. informal approaches to data analysisFormal vs. informal approaches to data analysis– Statistical analysisStatistical analysis

• Used less frequently, requires knowledge about statisticsUsed less frequently, requires knowledge about statistics

– Calculating metricsCalculating metrics• Calculation of success and failure rates and questionnaire data such as Calculation of success and failure rates and questionnaire data such as

Likert scalesLikert scales

– Analyzing notes for patternsAnalyzing notes for patterns• Looking for trends and patterns, across tasks and users.Looking for trends and patterns, across tasks and users.

– Physical observationsPhysical observations• Observing the facial and bodily expressions especially in regards to Observing the facial and bodily expressions especially in regards to

frustration or confusionfrustration or confusion

– Analysis “on-the-go”Analysis “on-the-go”• Changes to the design are made immediately based on informal notes Changes to the design are made immediately based on informal notes

during a study during a study

TUR 2011

(27)

Source: UPA 2006 Idea Markets - Analyzing usability study results: Is it magic behind the curtain?Activator: Emma J. Rose, University of Washington

TUR 2010

(28)

Statistical analysisStatistical analysis

Time needed to carry out each task. Frequency of errors made by the users. Responses to questionnaire items Etc.

– See lecture #12

TUR 2010

(29)

Calculating metrics (examples)Calculating metrics (examples)

Time to carry out the task

Number of tasks carried out

Number of errors

Number of used (or unused) commands and functions

Frequency of help access

Frequency of useful help access

Frequency of positive (negative) comments of the participant

Ratio of participants preferring the tested system

etc …

TUR 2010

(30)

Application support – Morae

Graph Visualizations– Mouse Clicks Graph– Count of Markers Graph– SUS Survey Graph– Time on Task Graph– Web Page Changes Graph

TUR 2010

(31)

Analyzing notes for patterns IAnalyzing notes for patterns I

Summarize the findings from the collected data– List of all important events

• Positive or negative aspects

– It’s a good idea to link back to the original data– Identify why there were problems

Things you can look for in your data:– Is the UI behaving predictably?

• Have the people behaved in the way you expected?

– Were all necessary functions available?– Wasn’t there too many unnecessary functions?

TUR 2010

(32)

Analyzing notes for patterns IIAnalyzing notes for patterns II

Read through the notes Look for:

– Repetitions– Things that could be caused by the same underlying

problem Can be done in the whole group of testers Cluster the observations

– By underlying problem• E.g.: Group all problems related to poor structure of the information

– By feature• E.g.: Group all problems related to printing

TUR 2010

(33)

Analyzing notes for patterns IIIAnalyzing notes for patterns III

Describe the clusters– What was the problem

• E.g.: “5 out of 8 participants could not locate the menu item X”

– The impact of the problem• E.g.: “Function Y could not therefore be accessed.”

– Place where the problem occurred

TUR 2010

(34)

Analyzing notes for patterns IVAnalyzing notes for patterns IV

(continuation of the university website example)

Searching for course Priority: 2 The participants do not know where to look for

courses. They go to "study programmes." They do not understand the distinction between

"career courses" and "retraining courses" They do not know what the Masaryk Institute of

Advanced Studes is.

TUR 2010

(35)

Application support – Morae I

Timeline visualization– Shows annotations in the timeline– Each marker category has its own color

TUR 2010

(35)

Taskstart

Final anchorCurrentposition

Play task Eventmarkers

Editation andlogging tools

Timelinezoom

End oftask

TUR 2010

(36)

Application support – Morae II

Filtering of tasksFiltering of tasks Filtering of participantsFiltering of participants Annotation listAnnotation list Annotations frequency graphAnnotations frequency graph

TUR 2010

(36)

TUR 2010

(37)

Application support – Morae III

Filtering of tasksFiltering of tasks Filtering of participantsFiltering of participants Annotation listAnnotation list Annotations frequency graphAnnotations frequency graph

TUR 2010

(37)

Physical observationsPhysical observations

TUR 2011

(38)

Physical observationsPhysical observations

TUR 2011

(39)

Physical observationsPhysical observations

TUR 2011

(40)

USAGE OF LOW LEVEL DATAA CASE STUDY R&D at CTU

Maly, Mikovec, Slavik 2007

TUR 2010

(42)

Use case: UI for inventory application on PDAUse case: UI for inventory application on PDA

Task scenarioTask scenario– User comes to the roomUser comes to the room– User looks aroundUser looks around– User checks hardware User checks hardware

inventory in the roominventory in the room– User writes down additional User writes down additional

notes about the room to the notes about the room to the PDAPDA

– User restsUser rests– User leaves the roomUser leaves the room

Office

TUR 2010

(43)

Categories of activities for mobile applicationCategories of activities for mobile application

Activities that represent user’s Activities that represent user’s interactioninteraction with a program with a program (picking objects on the screen, selection from menus, (picking objects on the screen, selection from menus, entering data etc.)entering data etc.)

BehaviorBehavior of the user more in detail (like her/his gestures, of the user more in detail (like her/his gestures, sitting, standing up, communicating with other persons etc.)sitting, standing up, communicating with other persons etc.)

MotionMotion of the user in environment where s/he should of the user in environment where s/he should perform interaction (e.g. in interior of a building)perform interaction (e.g. in interior of a building)

TUR 2010

(44)

Visualization of detailed behaviorVisualization of detailed behavior

TUR 2010

(45)

Visualization of detailed behaviorVisualization of detailed behavior

TUR 2010

(46)

Visualization of detailed behaviorVisualization of detailed behavior

TUR 2010

(47)

Visualization of detailed behaviorVisualization of detailed behavior

TUR 2010

(48)

Visualization of user motionVisualization of user motion

TUR 2010

(49)

Visualization of user motionVisualization of user motion

High cognitive

load

HighROI

MiddleROI

Low ROI

Eye tracking visualization – Heat MapEye tracking visualization – Heat Map

TUR 2011

(50)

http://blogs.lib.ucdavis.edu/hsl/2010/01/14/peeking-at-jakob-nielsens-eyetracking-web-usability/

Eye tracking visualization – Heat MapEye tracking visualization – Heat Map

AdvantagesAdvantages– Good for still imagesGood for still images– Can show interesting informationCan show interesting information

DisadvantagesDisadvantages– Some behavior is know (reading from left to right)Some behavior is know (reading from left to right)– Some eye trackers are obtrusiveSome eye trackers are obtrusive– Calibration is necessaryCalibration is necessary– There must be pauses during the test for some eye trackersThere must be pauses during the test for some eye trackers

TUR 2011

(51)

PRESENTATION OF THE TEST

TUR 2010

(53)

Where the Test Results are Used?

Design

Evaluation Implementation

TUR 2010

(54)

Presenting the Results

Informal Report– Initial feedback for the

designers– Within 2 days after the end

of the test

– Up to cca. 4 pages– Short descriptions of the

problems– Organized in bullet points

– A preview of the Formal Report

Formal Report– Detailed description with

analysis– Within weeks after the end of

the test

– 10 – 100 pages

TUR 2010

(55)

Contents of the Formal Report Purpose and aim of the test Demographics of the participants

– Anonymized! Overview of the tasks Pre-test questionnaire and responses Post-test questionnaire and responses Qualitative and quantitative results of the tasks

– Using screenshots, photographs, videos, transcripts of the audio recordings, …

All forms for the test Suggestion of further tests Links to A/V recordings (highlights)

TUR 2010

(56)

Contents of the Formal Report

See also– http://www.usability.gov/templates/index.html

TUR 2010

(57)

Priority assessment for each usability problem

Indicate how severe some problems were Questions:

– Does the problem affect big amount of users?– Is the problem difficult for users to overcome?– Is the problem persistent?

Severity levels:– Critical – Prevent from finishing common task – Fix urgently– Serious – User slowdown – Fix as soon as possible– Medium – Frustration and irritation – Fix during update– Low – Cosmetic issue or spelling error – Fix when have time

Source: http://www.userfocus.co.uk/articles/prioritise.html

TUR 2010

(58)

Anonymizing the Results

Make sure that no information that could lead to the identification of the participants appears in the report.– Unless the participant gives an explicit consent

E.g. use visualizationof the test in VR

TUR 2010

(59)

Suggesting the Solutions

Be careful when formulating suggestions– It’s the designers’ responsibility to identify underlying

causes of the problem– Recommendations should serve as guidelines where the

solutions could be found

TUR 2010

(60)

Audio/video highlights

Good to illustrate problem (for designer, programmer), they need to accept there’s a problem before they can fix it.

Show top 5 issues, show them in 5 videos, 5 minutes each (for problem understand or behavior comparison).

Videos can be misleading when context is missing (e.g. video recording starts too late)

TUR 2010

(60)

TUR 2010

(61)

Data: Example Video Recording

Typical setup– Camera #1: The participant– Camera #2: The screen (or work space)

Entire session is recorded

Example recording:– Actual usability test of CarDialer system by IBM

• Speech-controlled hands-free telephone for cars• Simulated car• Camera pointed at the participant and the moderator

– <video>

CarDialer Example

TUR 2011

(63)

Conclusion

Data collection– Audio/video, video annotations with markers– Optionally Low level data and Application data (difficult to visualize)

Data analysis– Statistical analysis for quantitative data– Search for behavioral patterns in annotations and video

Data presentations– Informal and formal report– Video highlights