User-centered Evaluation

• What is the purpose? – Determine if users can fulfill their goals, – Determine if users can complete tasks required to fulfill their goals– Identify problems users may encounter when trying to accomplish

their goals– Determine issues relative to components of usability – efficiency,

satisfaction, safety, learnability, etc.

• How will the information be used?– “Formative”: Inform the next design iteration– “Summative”: Determine if the system is successful, acceptable

User-centered Evaluation

• Expert evaluation– General description: A non-user explores the system in order to predict

problems that users may have– Examples: Design review, heuristic evaluation, etc.– Strengths: Easy, efficient– Limitations: Difficult to really predict user problems

• User testing– General description: A user is asked to complete tasks using the current

prototype, and the user’s performance and comments are used to identify problems

– Examples: Usability testing, user walkthrough– Strengths: Effective at bringing user issues to light– Limitations: Can be time consuming, but doesn’t have to be

Spending time on evaluation

• Why – Experience with different forms of evaluation

– Feedback to designers, begin iterative cycle

– Practice reviewing from a user perspective

– Desensitization process…

–

• Also– Opportunity for students to see other projects

– Practice presenting your design

Expert evaluation - How

• General– Groups of 3, students represent different project groups – Each student gets to get feedback and give feedback

• Roles:– Designer:

– Reviewer:

What is a heuristic…

Definition• “The term web heuristic is used to refer to all the sets of

process guides, principles, criteria, tips and tricks, and guidelines that are available to support Web designers.”

De Jong, M. and Van Der Geest, T. (2000). Characterizing web heuristics, Technical Communication, Third Quarter, pp. 311-326.

Examples• Speak the users language (Nielsen)• Ensure links indicate that they are links (Farkas)• Put shopping cart in upper right-hand corner of Web

checkout (Adkisson)• Using pictures & illustrations: Decorative pictures (Williams)

Heuristics typically come in sets Consider…

• Williams - Displays

• Spyradakis – Text

• Farkas – Navigation

• Nielsen –Usability

• Quesenbery – Search

• Barnum – Varied…

Also…

• Mullet – Graphic Design

• Simpson – general HCI

• Schneiderman - HCI

• Dumas - general HCI

• Redish – document design

• Horton - documentationStill more …

• Accessibility

• Designing for elderly

• …

How heuristics differ

• Information covered– Specificity– Exhaustiveness

• Validity– Foundations:

• Standards-based, • Theory-based,• Research-based, • Practitioners’ experience

– Novelty– Room for interpretation– Validation research?

• Presentation format– Structure– Formulation of items– Types of answers– Level

• Use of the heuristics– Phase in the design process– Focus of support– Function in the design

process– Assumptions about actual

use

De Jong, M. and Van Der Geest, T. (2000). Characterizing web heuristics, Technical Communication, Third Quarter, pp. 311-326.

Nielsen, 1994

Examples• Visibility of system status• Match between system and real world (speak

users’ language)• User control and freedom (emergency exit)• Consistency and standards• Error prevention• …

Adkisson, 2002• Adkisson, Heidi (2002). Identifying de-facto standards for E-

commerce websites

• Motivating and defining the standards– Nielsen claim: Web design is easy. If you are thinking about how to design

a certain page element, all you have to do is look at the 20 most-visited sites on the Internet and see how they do it.

– De-facto standard: >90% of sites do it one way– Strong convention: 60-90% of sites do things in a single way

• Data Collection– 75 leading e-commerce sites– Explore functions: Return to home, View cart, Manage Account, Get help,

Search– Characterized: Position, Label, Icon

Spyradakis, 2000

• Examples– Group information to help readers create hierarchical

frameworks for storing incoming information.– Use organizational cues to make text visually

accessible and scannable.– Present content in such a way that readers can orient

themselves and access relevant prior knowledge so they can comprehend new information when they arrive on a new page.

– Minimize the amount of information per page.

A challenge of using heuristicsThe heuristic

All of the examples, experiences captured by the heuristic

Nielsen - Speak the user’s language

• How might a system speaks the users’ language?

• How could a system fail to speak the users’ language?

• Implications of failing to speak the users’ language?

• Sources of information about the “user’s language?”

Systematic heuristic evaluation (Usability audit)

1. Preparationa. Identify target: What website, what users, what tasks?b. Identify evaluators: How many and what qualifications?c. Identify heuristics: Which heuristics and why?

2. Conduct evaluationa. Systematically rate conformance to each heuristic

(note: If multiple evaluators, reconcile disagreements on ratings)

3. Explore design implications

Product: – A characterization of the site in terms of each heuristic (a list) – Design insights that result from discussing the design relative to the

heuristics.

Systematic heuristic evaluation: Example: Comparing education websites

1. PreparationIdentify target: Four websites for engineering educatorsIdentify heuristics: Select of comprehensive, navigation, visualsIdentify evaluators: Myself, undergraduate researcher

2. Conduct evaluationIndividually determined conformanceNegotiate disagreements until consensus

3. Explore design implicationsEffective search

Turns, J., Valeriano, M., and Adams, R.S. (2003). Websites for engineering educators: A benchmark study to identify best practices, Proceedings of the 2003 Frontiers in Education Conference, Denver, Colorado.

Systematic heuristic evaluation: Example: Comparing education websites

Heuristics Websites UMich

REE Virtual Labs

1. Indicate link destination (Sec. 1.3, Farkas) 3. Reveal the information structure (Sec. 2.4, Farkas) 6. Provide a search engine or index (Sec. 4.2, Farkas) 8. Minimize information (Sec. 1.3, Spyridakis) 9. Orient Readers (Sec. 1.4, Spyridakis) 10. Create hierarchical frameworks (Sec. 2.1, Spyridakis) 14. Demonstrate credibility (Sec. 4, Spyridakis) 16. Avoid distracting background (Sec. 1.2, 1.3, Williams) 18. Group related elements (Sec. 2.2, Williams) 21. Reveal sequence of information (Sec. 2.5, Williams) 22. Ensure text is readable (Sec. 3, Williams)

Legend: Heuristic Evaluation Rating Scheme: Excellent – Consistent with principle; No usability problem will result. Good – Minor deviation from principle; Slight confusion, impaired productivity, and reduced satisfaction. Fair – Major deviation from principle; Significant detriment to usability. Poor – Principle overlooked; User is expected to experience significant difficult in using the site.

User Testing

• USERS: Work with actual users, make certain to pick representative users

• TASKS: Have users complete actual tasks, “exercise the interface”

• DATA: Collect data – measurements, errors, also user reactions and observations.

• INTERPRETATIONS: What does it mean?

Exercise – User-centered evaluation

• Design and execute an user-centered evaluation of you prototype with at least two users (e.g., usability test, focus group). Prepare a one page description summarizing your evaluation method and results and potential implications for redesign. Bring to class one copy of this exercise for each member of the team and one copy for the instructors.