User Testing and Modeling

User Testing and Modeling

ICS 205

Chris Wesson

Christina Wuerth

November 14, 2003

User Testing

What is User Testing– Measuring performance of typical users doing

typical tasks

Goal:– Obtain objective performance data showing how

usable a system is in terms of usability goals (ease of use, learnability)

User Testing

Applied form of experimentation Tests whether product is usable by intended

user group(s) Measures/Records:

– Time to complete typical tasks– Number and type of errors– Routes users take through tasks (Web)

Typically 6-12 users

User Testing

Is part of Usability Testing:– Obsevations– Questionaires– Interviews– User Testing

User Testing and Modeling

Doing User Testing

Experiments

Predictive Models

Keys to Doing User Testing

Control testing conditions Careful planning Same conditions for each participant Measurements correspond to test Explicit assumptions

DECIDE framework

Q: How Long Should User Testing Process Take?

A: As little time as possible while still yielding useful information.

On average most companies take more than

8 weeks for user testing

Testing of small piece of well developed procedures can be done in less than 1 week if needed

Things to Consider in Planning a User Test

What aspects of the product need to be improved? Do the participants in the study represent actual

users of the product? What tasks should the participants perform? What information will you observe during the tests? How will you analyze the data you collect? What will you do with this information once it is

analyzed?

User Test Planning Steps

1. Define goals and concerns2. Decide who will participate3. Recruit participants4. Select and organize tasks to be tested5. Create task scenarios6. Decide how to measure usability7. Prepare other materials for the tests (questionnaires, etc.)8. Prepare the testing environment9. Prepare the test team10. Conduct pilot tests making changes where needed

The Testing Team

The Roles:Usability Specialists

The usability specialists know what can be accomplished in a user tests and how a test should be planned. They should also understand the interface and potential problems users may have with it.

Designers/DevelopersThe designers and developers understand the product and

what the user should be able to do with it.

Technical CommunicatorsTechnical communicators know what aspects of the

documentation should be tested and also see potential problems in how the product communicates to the users.

More User Test Team Roles

TrainersThe trainers understand the problems that users will have when learning to use the product.

Marketing SpecialistsMarketing specialists know who the users will be.

Helpdesk/Customer Service RepsHelpdesk and customer service reps know what problems users have had with earlier versions of the software.

Preparing for the Test

1. Schedule activities2. Assign roles and responsibilities3. Train team4. Write out a test plan5. Practice by running pilot tests

Throughout the entire process remember to:- track progress - document all decisions

DECIDE:User Testing Guideline

D etermine goals E xplore questions C hoose paradigm and techniques I dentify practical issues D eal with ethical issues E valuate, analyze, present data

Determine Goals, Explore Questions

User testing is best used for testing prototypes and working systems

Goals can be broad (how usable?)

Specific questions required to focus study (can task X be done in time Y?)

Choose Paradigm and Techniques

User testing is part of Usability testing paradigm

Data can be recorded using:– Video– Interaction log– User satisfaction questionnaires– Interviews

Identify Practical Issues:Design Typical Tasks

Create set of “completion” tasks (finding a website)

Choosing which tasks to test is critical Tasks generally last 5-20 minutes Tasks are often straightforward Complex tasks (create design, solve

problem) are okay Start simple to build user confidence

Identify Practical Issues:Select Typical Users

Must know users’ characteristics

Most important: previous experience

Use short questionnaire to identify users

Equal number of males and females

Identify Practical Issues:Prepare Testing Conditions

Control testing environment

Minimize outside influences/noises that could distort results

Identify Practical Issues:Plan How to Run Tests

Create schedule and scripts for running tests Start with familiarization task Contingency plan for spending too much time

on a given task Avoid long tasks and long testing procedure Session < 1 hour

Deal with Ethical Issues

Informed consent form

Point out any:– One-way mirrors– Video cameras– Interaction logging

Evaluate, Analyze,and Present Data

Performance measures recorded from video, interaction logs

Only use simple statistics:– Maximum Minimum– Mean Standard Deviation

Allows evaluators to compare performance between systems and across tasks

Establishing Goals and Concerns

Step 1 of the planning phase is to define goals and concerns.

Q: What is a goal?

A: A quantitative usability aim that can be stated as a declarative sentence

example: A user should be able to locate the print menu item in less than 5 seconds with less than 3 errors during first attempt.

Q: What is a concern?

A: A concerns are questions raised about the usability of the product – these are often raised while planning for the tests.

example: While a user find the new error messages confusing?

Establish Which Goals and Concerns will be tested.

Not all goals and concerns can be tested simultaneously.

Example of 2 conflicting concerns:1. Will the user use feature X?2. Will the user find X easy to use?

These two questions can’t be answered at the same time instead you must prioritize.

Using Concerns to Plan the User Test

Use General concerns to decide who your participants should be

Example: Will new users find this product easy to learn?Make sure you test with some beginning users.

Use specific concerns to decide which tasks should be performed.

Example: Will users be able to find the print command quickly? In less than 5

seconds? Have the users perform a task of printing.

Sources of Goals and Concerns

1. Task Analysis and Quantitative usability goals

Example:

General: Menus should be easy to navigate

Quantitative Goal: Users should find menu choice in less than 2 minutes with no more than 2 wrong choices when first need it. Task should be done with no errors in less than 1 minute after first attempt.


2. Timely Issues – Certain concerns arrive at different stages of development and should be resolved at that time. Example:

1) Two different design philosophies that you can’t decide between…plan a user test at start of the project to test prototypes of each

2) There has been previous testing earlier in the project, now you want to test a new feature before continuing development.


3. Heuristic Analysis and Expert Reviews a) Problems that have been predicted by heuristic analysis and

expert reviews should be candidates for user tests.

b) Problems predicted by designers, planners, developers, human factor specialists, technical communicators should be tested.

c) Concerns found by reflecting on the product itself should be tested.


4. Previous Tests Example:

iterative testing with rapid prototypes

run several user tests with the same concerns

Example:

large scale user tests that raise future concerns

retest with same concerns to make sure changes improved the problem

Who Should be Participants?

Participants must be like the people who will actually use the product.

Develop a user profile then choose participants that fit the profile.

Developing a User Profile

Base profile on:

General market research

Analysis of customers of competitors’ products

Focus group sessions

Observing and interviewing prospective users

Developing User Profile

1. Think about relevant characteristicsa) those that all users shareb) those that might make a difference among users

example: shared characteristic:

users will be undergraduate university studentsdifferences:

users will have lots of computer experienceusers will have very little computer experienceusers will have registered for classes beforeusers will have never registered for a class before


2. Decide which characteristics matter most in establishing usability

experience and motivation contribute more towards and individual’s understanding than education, income, age, etc.

Good Factors to Consider1. Work experience2. General computer experience3. Specific computer experience4. Experience with this product5. Experience with similar products


3. Think broadly about who the users will beSample questions to reflect on:

1. Not just those currently working, but what about new hires? What is the rate of turnover that the software will need to handle?

2. Don’t limit yourself to the current market…what about growth, who might want to use this product in the future?

3. What other areas of the corporation may what to adopt this software?

4. Reflect on differences within a category of users. Older users versus younger users? Users who have done the job longer versus users that more adaptable to computer environments?


1. Product Name: Class Registration Software

2. General Characteristics of User Population

University Students

3. Characteristics of Users Relevant to the test

Year in School

Computer Experience

Previous Experience Registering for Classes


4. Which Characteristics listed in 3 should all users in the test have in common & how will you define them?All users should be students currently enrolled in a universityThere will undergraduate and graduate students


5. Which characteristics listed in 3 will vary in the test and how will you define them?

Computer Experience will vary.

Novice = < 1 year experience

Medium = > 1 year experience used < 5 days a week regularly

Expert = > 1 year experience used > 5 days a week regularly

Select Subgroups for a User Test

Subgroups are people who share specific characteristics important to the user profile

Divide groups by one characteristic at a time.

Example:University Students

Subgroup 1 Subgroup 2Undergraduates Graduates

Subgroup 1a Subgroup 2aUndergraduates GraduatesLittle computer experience Little computer experience

Subgroup 1b Subgroup 2bUndergraduates GraduatesLots of computer experience Lots of computer experience

Define Characteristics for Each Subgroup

Define what you mean by characteristicsExample

novice = less than 1 year experience with this product

medium = 1 year to 3 years experience with this product

expert = more than 3 years experience with this product

Qualify Characteristics of the Subgroups

1. The middle group is often omitted from the subgroups in user studies since most problems found here will be found in novice and expert studies

2. Select a range of participants in each subgroup.Example: Subgroup of novices with < 1 year experience

Find users with 1 month, 6 months, and 11 months experience

3. Establish range of each subgroupset min and max for subgroup participantsExample: Subgroup of experts with > 3 year experience

min would be 3 years but perhaps you want max experience to be 5 years…you don’t want someone whose a hacker even though they may have lots of experience

How Many Participants?

Depends on:1. How many subgroups you need2. Time and money constraints3. How important it is to compute statistically

significant results

* Most of the time all you will care about is inferential statistics

How Many Participants?

6 – 12 participants is the typical choice

With a minimum of 3 individuals in each subgroup.

* Problems will show up across subgroups

Making the Most of a limited Number of Participants

1. Decide which characteristics are the most important so that your subgroups will be most useful

2. Collect relevant information from participants via interviews or questionnaires to help account for differences that show up in results.Example: one novice user performs faster than all the others, but you find out that he has used a similar product before

3. Select people in subgroups representing the full range of the group

Experiments

User testing based on scientific experimentation

Aim: test hypothesis that predicts a relationship between variables

Variables

Independent: manipulated by researcher Often multiple independent variables:

– System version, User experience

Dependent: affected by independent variable Common dependent variable:

– Time to complete task, Number of errors

Assigning Participants to Conditions

Three Experimental Designs:

Different participants for all conditions Same participants for all conditions Matched pairs of participants

Matched Participants

Participants matched in pairs

Based on specific characteristics (expertise, gender, etc.)

Used when participants cannot perform in all conditions

Design Comparison

Design Advantages Disadvantages

Different Participants

No order effects. Many participants needed. Individual differences can be a problem.

Same Participants

Eliminates individual differences.

Must counterbalance ordering effects.

Matched Participants

No ordering effects. Reduces effects of individual differences.

No guarantee subjects are match across all variables.

Data Collection

Data should measure users’ performance

Typical measures:– Response times– Number of errors– Time to complete task

Data Analysis

Questions to ask:– Do data sets from two conditions look similar or

different?– Any extreme atypical values?– If so, what does that mean?

Graph data to display differences

Predictive Models

Provide measures of user performance without testing users

Useful when it is difficult to do user testing Estimate efficiency of different systems on

different tasks Most common models: GOMS family:

– GOMS Model, Keystroke Level Model

GOMS Model

Models cognitive processes used when interacting with systems

G oals: state the user wants to reach O perators: cognitive processes, physical

actions necessary to achieve goal M ethods: learned procedures for

accomplishing goals S election rules: used to choose between

available methods

Keystroke Level Model

Variation of GOMS model Provides numerical predictions Uses a standard set of times for main types

of operators used during a task Average time to perform certain tasks Compares times for task using different

startegies

GOMS Pros and Cons

Pros Allows comparative

analysis of different systems easily

Useful estimates to compare efficiency of different systems

Works well for short, well-defined tasks

Cons Only models tasks that

involve routine tasks Designed to predict

experts’ performance Difficult to predict for

average users Only predictions about

predictable behavior

Fitt’s Law

Useful for planning button locations, size and proximity

Predicts time to click on objects on a screen

T = k log2(D/S + .5), k ~ 100 msec– T: time to move hand to target– D: distance between hand and target– S: size of target

Bigger target = easier to reach

Summary

User Testing is the core of Usability Testing Controlled laboratory-like conditions Control independent variable(s) to predict

dependent variable(s) Experimental designs: different participants,

same participants, matched participants GOMS, Keystroke, Fitts’ can be used to

predict expert performance

Documents

User Testing and Modeling