Usability Testing

Chapter 6

Reliability

• Can you repeat the test?

Reliability

• Huge difference between test subjects– Therefore, only 2 subjects is not a good

reliability check

• Significance levels (e.g., p = .20)• Confidence levels

– When should an usability problem be fixed?

• Error rates have the highest variability– Therefore, more test subjects are needed

Validity

• Does this test reflect what usability issues you want to test?

• Reliability measured with stats but validity requires methodological understanding

• Typical problems– Wrong users or wrong tasks– Confounding effects – text based to GUI

system

Test Plans

• What’s the goal of the test?

• What is it you are trying to prove or show?

• Pilot tests – show the problem areas of the test. E.g., the instructions

Test Plans

• The goal of the test?

• Where and were will the test take place?

• How long is each test session expected to take?

• What computer support will be needed for the test?

• What software needs to be ready for the test?

• Who will serve as experimenters for the test?

• How many test users are needed?

• …

Getting Test Users

• Test subjects (users) should be representative of who will use the system.

• Sales (“demoability”) – is it easy to show

• Asking managers to choose subjects– They will pick the best or the worse

• Formative evaluation– Help improve the interface as part of an

iterative design process

• Summative– Assessing the overall quality of an interface

Novice and expert users

• Most tests should test novice users

• If possible, test for expert users. But test separately from novice users.

• What are the effects of training or not training before a test?

Between-Subject test

• Simplest and most valid

• Different tests subject using different systems

• Possible problems:– Individual differences– Assignment to test groups (volunteer early or

late)

Within-Subject test

• All users are tested on all systems

• Problem: no longer a set of novice users when testing a secondary system

Choosing experimenters

• It’s better to get an usability person to conduct the test but you, the developer, can also run a test

• In fact, it’s good to know the system well• From Usability Engineering (Nielsen)

– “It is possible for computer scientistws to learn user test methods and apply them with good results.”

• However, designers try to explain away problems.

Ethical aspect of testing human beings

• Users should feel as comfortable as possible• Tell the test subjects:

– No information will be revealed– Explain the testing environment. E.g., computer, etc.

• The tester should not allow observers because they tend to influence results.

• The tester should not interfere with the user. Let the user find the solution themselves

Emphasize that it is the system that is being tested, not the user

Test tasks

• Based on a task analysis. What is it the user needs to do.

• Tasks should be small enough to be completed in time that the experiment takes.

• Test tasks should be in writing. The user may refer to the steps.

Stages of a Test

• 1. Prep – is everything in working order?

• 2. Intro - tell the user the purpose of the test.

• 3. Test

• 4. Debrief

Stages of a Test

• Intro - tell the user the purpose of the test.• A reminder that the test is confidential and

should not be discussed with others• A statement that participation is the test is

voluntary and the user may stop at any time• At FIT a human test subject document

needs to be signed by the tester and returned to the university.

Performance Measurement

• Has the usability goals been met? (see page 194)

• Test a set of tasks• The data collected: time to perform and

error rate• Goals are abstract, so break them down.• Make sure you know the start and finish of

a measured test

Thinking aloud

• “verbalizing their thoughts”• Shows how users interpret each interface

item

• Users performed 9 % better using thinking aloud.

• Constructive interaction (co-discovery learning) and coaching methods.

Usability Labs

• Is there really a need?

• Why videotape? Impact analysis – you look after taping for a known problem.

• Convince the manager and developers

• Usability kiosks