Instructional Technology & Usability

2/16/05 Usability Testing for ISD - Leslie Carter 1

Instructional Technology & Usability

MATTERS

Leslie Carter, Ph.D.


1. How important is usability?

2. What do we mean by “usability?”

3. How do we evaluate usability?


According to Instructional Designers

• 99% Agree: Usability is essential to e-Learning design.

• 96% Agree: e-Learning components should always be tested for usability.

• 96% Agree: User-centered design enhances the effectiveness of e-Learning.

From the E-Learning Guild’s “Usability and e-Learning” survey reported 2/27/2004.


According to Instructional Designers

What are the "Top 5" features you would recommend that someone consider when licensing a live e-learning product?

Ease of Use was cited by more e-learning professionals than any other factor.

From Brandon-Hall.com winter 2004 survey.




3. How do we test for usability?


Usability is“The effectiveness, efficiency, and satisfaction with which specified users achieve specified goals in particular environments.” (ISO 9241)

1. Effectiveness - Accuracy & completeness.

2. Efficiency – Speed & ease of use.

3. Satisfaction - Comfort & acceptability.


What contributes to usability?1. Visual

Design

2. Information Architecture

3. Navigation Design

4. Interaction Design

5. User Interface Design

A. The overall process of designing how a user will interact with a software application or web site, including: requirements analysis, information architecture, interaction design, visual design, user testing, documentation, & help system design.

B. The design of the graphic layout, & the images, icons, fonts selected to support the informational content.

C. Information organization that should help people find & use information efficiently. It’s concerned with things like content chunking, formatting, categorizing, labeling, placement & navigation.

D. The design of the workflow through an application or site, including the dialog between the user & the system, what user inputs are needed & when, as well as, what feedback is appropriate & when.

E. The design and placement of menus & options used to find specific areas of a software application or site.


What contributes to usability?1. Visual

Design

2. Information Architecture

3. Navigation Design

4. Interaction Design

5. User Interface Design

A. The overall process of designing how a user will interact with a software application or web site, including: requirements analysis, information architecture, interaction design, visual design, user testing, documentation, & help system design.

B. The design of the graphic layout, & the images, icons, fonts selected to support the informational content.

C. Information organization that should help people find & use information efficiently. It’s concerned with things like content chunking, formatting, categorizing, labeling, placement & navigation.

D. The design of the workflow through an application or site, including the dialog between the user & the system, what user inputs are needed & when, as well as, what feedback is appropriate & when.

E. The design and placement of menus & options used to find specific areas of a software application or site.


Guidelines for Good Usability

Learning these will help you:• Recognize the cause of problems found

in usability testing • Find solutions to those problems• Reduce the number problems found in

future usability testing

A Comparison of Heuristics from Several “Gurus”

Review the usability heuristics (guidelines) published by researchers & practitioners.

http://immersion.gmu.edu/portfolios/lcarter3/heuristics.html

http://immersion.gmu.edu/portfolios/lcarter3/heuristics.html


Role of Usability in Evaluation

• Increased cognitive load• Wasted time• Frustration & reduced motivation

Poor usability leads to:

Great training content can lead to lousy training results when poor usability is involved!


1. What does the beginning of the field of usability have in common with instructional technology?



4. How do we test for usability?


Conducting a Usability Test

• Determine goal of test • Select evaluation criteria• Select participants• Write a task scenario• Write a testing protocol• Conduct the test• Track the observations• Report the results


Testing Goals1. Determine how to improve a developing

product– Formative: product under construction, could be

even be a prototype– Informal: quantitative measure not required,

qualitative observations needed

2. Compare two different products or versions of same product– Summative: products are mature & ready for the

marketplace– Formal: quantitative measures required


Evaluation Criteria• Most common

– No. of tasks completed– Time to complete task– Errors made in performance– Satisfaction

• Other possible measures– Proportion of time spent determining what to do

instead of doing it– No. of times consulted help or user guide– No. of positive or negative comments from user


Usability is“The effectiveness, efficiency, and satisfaction with which specified users achieve specified goals in particular environments.” (ISO 9241)

1. Effectiveness - The accuracy & completeness.

2. Efficiency – Speed & ease of use.

3. Satisfaction - Comfort & acceptability.


Participants

• Similar to your audience – training, experience…

• Availability – when & for how long.

• Where – your place or theirs, theirs is better if like that of your target audience. Consider remote testing if you can’t get to them.


Tasks Scenario• Select tasks: critical &/or frequent• Develop a realistic scenario for sequencing

those tasks• Start with easy tasks to build user’s

confidence• For each task give user a goal to achieve

instead of steps to perform• Just enough info for user to complete the

task without more info from observers• Make it clear to user when each task is

compete


Sample Scenario Outline

Task 1 - View a webshopTask 2 -Create a webshopTask 3 -Revise a webshopTask 4 -Create a course

(several webshops put together)Task 5 - Revise a course


Sample Tasks

• A page with one large graphic (Image-1large.jpg). Title this page “My First Page.” Add this description “my webshop.” Check to see how this page will look to the site’s viewers before starting on the next one.

• A page with a video clip (TinyMovie3.mov), and an Acrobat PDF file (acrobat6matrix.pdf). Title this page “My Second Page.”

Task 1 – View a WebshopStating from the T/TAC home page, find “Leslie’s Webshop” & view the entire webshop.

Task 2 - Create a WebshopYour goal for our first task is to create a simple webshop made up of several kinds of media. Name this webshop with your name and the date, e.g., “Leslie’s 3/1/04 Webshop.” This webshop will include the sample media we brought & consist of 2 pages:


Testing Protocol

• Intro for participants• Reminders to observers• Copy of the task scenario• Steps required to perform each task• Make a separate copy for each

observer & for each test


Sample Protocol – Participant Intro

• We are not testing your ability to use this site; we are testing the sites’ ability to meet your needs.

• Your feedback will help us ensure that T/TAC Online will meet the needs of T/TAC professionals.

• The site is designed to allow T/TAC staff to upload training content for presentation over the Internet. There are two kinds of presentations we’ll try today. One is a webshop which is a presentation designed to last 15-30 minutes, something a viewer might complete in one sitting. The other is a course which is a set of webshops combined in one course.

• If you would like to take a break at anytime, just let us know.• Please try to think out loud as you use the site. Let us hear

the questions in your mind. We need to know if you can’t find something or if something is not absolutely clear to you. And, let us hear about the features you like about the site.


Sample Protocol – Conducting the Test

Task Steps

1 – View a WebshopStating from the T/TAC home page, find “Leslie’s Webshop” & view the entire webshop.

1. Click on webshop2. Click Continue button

until returned to start

Task 2 - Create a WebshopYour goal for our first task is to create a simple webshop made up of several kinds of media…

1. Online Training tab2. Add a Webshop button3. Enter profile info4. Template button

Reminders:• Don’t tell user the steps, unless they’re very frustrated• Don’t give away the names of the buttons or links• Note start & end times, step where error occurs• Get positive & negative quotes from user


Tracking Results

• It’s very easy to forget or loose site of useful observations

• Need a system for tracking all the issues you want to address

• Track who noticed the issue, when noticed, what kind of issue, what screen, how important, & when fixed.


Reporting on a Test

• MUCH stronger impression is communicated if you have sample user comments from videotapes or Camtasia screen recordings.

• There is an industry standard for usability test reporting from NIST. Very complete. The checklist is great. http://zing.ncsl.nist.gov/iusr/documents/cifv1.1b.htm


Usability Testing in Kuniavsky

1. One day before a test is not enough for the practice test. (p.264)

2. Excellent discussion of dealing with participant recruiters (p.266)

3. Don’t test features. Test user goals. (p.268)

4. Quantitative measures – what are the goals of your test? Don’t forget organizational politics.


Usability Testing in Kuniavsky (cont.)

1. Make it absolutely clear that it’s the interfaces, not the person being evaluated. (p.279)

2. Discount usability testing – Don’t need one-way mirror or cameras. You can do it with remote software & screen capture utilities. (p.286)

3. Define severity ratings to match the bug codes & enter them in the bug tracking system. (p.294)


Resources

http://immersion.gmu.edu/portfolios/lcarter3/resourcesusab.html

http://immersion.gmu.edu/portfolios/lcarter3/ErgonomicKeyboards.ppt

Documents

Instructional Technology & Usability