Evaluation for 2008

Michael GaiggMarch, 18th 2009

Remote User Testing

Remote User Testing 2

Agenda

• Show how – User-centered Design helps teams to save:

• Time• Money• Aggravation from clients

– Usability testing early in the design process identifies:• Areas of confusion and frustration • Where the user can’t complete a task • While it is still cheap and easy to mitigate these problems

03/18/2009

Remote User Testing 303/18/2009

Agenda

• User-centered Design• User Testing• Output (Qualitative & Quantitative)• Software & Usage• Time Estimates & Cost• Advantages & Disadvantages

Remote User Testing 4

Introduction to User-centered Design

• UCD is a design philosophy and a process in which the needs, wants, and limitations of the end user of an interface or document are given extensive attention at each stage of the design process.

• Its measures are:– Effectiveness (accuracy and completeness to achieve goals)– Efficiency (resources expended in relation to the accuracy

and completeness)– Satisfaction (comfort and acceptability of the work system to

its users)

03/18/2009

Remote User Testing 5

User Testing is a UCD Method

03/18/2009

Method Cost Output Sample Size When to Use

Competitive Studies Medium Stat. & Non-Stat. 5 Requirements

Focus Groups High Non-Statistical 6-9 Requirements

Field Studies High Non-Statistical 2-3 Requirements

Heuristic Evaluation Low Statistical 2-3 Design

Paper Prototyping Medium Stat. & Non-Stat. 5 Design

Card Sorting High Statistical 15-20 Design

Participatory Design Low Non-Statistical n/a Design

User Testing Medium Stat. & Non-Stat. 5 Design & Evaluation

Surveys Low Statistical 20+ Requirements & Evaluation

Interviews High Non-Statistical 3-5 Requirements & Evaluation

Server Traffic Log A. Low Statistical n/a Evaluation

Search Log Analysis Low Statistical n/a Evaluation

Remote User Testing 6

Overview

• User testing involves: – A design– A user (participant) – An observer (facilitator).

• Observe user behavior & receive user feedback.• Test tasks (not functions).

03/18/2009

Observer

UserDesign

Remote User Testing 7

Qualitative Output

• Understand mental model of the user (navigation).• Validate requirements and design assumptions

(focus on important functionality early).• Discover non-functional design mistakes and areas

of improvement (unforeseen user requirements).• Evaluate effectiveness of design elements.

03/18/2009

Remote User Testing 8

Quantitative Output

• Success rates / Error rates• Task time• Satisfaction ratings

03/18/2009

Task Success Task Time

Remote User Testing 9

User Testing Software

• TechSmith Morae• Manager: $ 1,295.-• Bundle: $ 1,495.-

• Morae Manager uses the collected data (observation markers and notes, video, keyboard and mouse inputs) to analyze and calculate task times, error quotes and other common measurements.

03/18/2009

Remote User Testing 10

Remote User Testing Software

• TechSmith UserVue• $ 149.- / Month or• $ 1,495. / Year

• Web-based app enabling a Facilitator to remotely observe (and record) a Participant using a phone line for communication.

• Multiple Observers can passively join the Session and share their observations with the Facilitator.

03/18/2009

Remote User Testing 11

The Session

• Setting up a session.• Connecting (download of

software bundle required).• Calling participant.• Starting the session.• Viewing (and recording) the participant.• Removing software bundle from computers of

involved parties.

03/18/2009

Remote User Testing 12

Time Estimates & Cost

• Preparation (18 hours)– User screening: 8 hours– Task creation: 8 hours– Environment: 2 hours

• User testing (10 hours)– 2 hours per user– 5 users per round

• Post-test (32 hours)– Test report: 16 hours– Implementation: 12 hours– Communication: 4 hours

03/18/2009

Our time:1 Round = 60 hours2 Rounds = 110 hrs.3 Rounds = 160 hrs.

+ User comps+ Time additional observers

Remote User Testing 13

Advantages of remote UT

• Reduced (or no) travel time and expenses.• Higher exposure through easy screen sharing

(managers can sneak in easily).• Actual user environment, familiar and comfortable.• Possibly fewer drop-outs.

03/18/2009

Remote User Testing 14

Disadvantages of remote UT

• Facilitator not physically present (degree of separation can be challenging).

• Can't see facial expressions or non-verbal cues.• Difficult to build rapport and trust.• Difficult to control environment.• Possibly technical difficulties (firewall, etc.).• Setup and use of software or usability lab might be

challenging and requires a liaison.

03/18/2009

Remote User Testing 15

Summary

• Increased usability– Higher degree of client satisfaction– Continued business = Higher revenues– Lesser client aggravation (“This is completely unusable”)

• Assist PM during requirements and design– Focus and prioritize on important functionality– Identify unforeseen user requirements early!

• Reduced costs– Training and service– Viewer Help-Desk calls

03/18/2009