A Human Factors Evaluation of Illinois Compass Presented at the CITES Educational Technologies Brown Bag by Esa M. Rantanen, Ph.D., on April 13, 2005

A Human Factors Evaluation of Illinois Compass

Presented at the CITES Educational Technologies Brown Bagby Esa M. Rantanen, Ph.D., on April 13, 2005

CITES Educational Technologies Brown BagApril 8, 2005

viationHuman Factors

• What is Human Factors?

• Human-Centered Design philosophy.

• Barriers to Human-Centered Design.

• Human Factors in systems’ development.

• Human Factors evaluation of the Illinois Compass.

• Results: Suggestions for improvement.

• Conclusion.

Presentation Outline



Human Factors: Definitions

• “Human Factors is that branch of science and technology that includes what is known and theorized of about human behavioral and biological characteristics that can be validly applied to the specification, design, evaluation, operation, and maintenance of products and systems to enhance safe, effective, and satisfying use by individuals, groups, and organizations.”

(Christensen, Topmiller, & Gill, 1988).




• “Human Factors is that branch of science and technology that includes what is known and theorized of about human behavioral and biological characteristics that can be validly applied to the specification, design, evaluation, operation, and maintenance of products and systems to enhance safe, effective, and satisfying use by individuals, groups, and organizations.”

(Christensen, Topmiller, & Gill, 1988).




• Human Factors studies human capabilities and limitations as they apply to the design of systems and products.

• Ergonomics (from gk. ergon ‘work’ + nomics) is the application of scientific information concerning humans to the design of objects, systems and environment for human use.

• The goal of human factors is to…– …reduce error– …increase productivity– …enhance safety, and– …enhance comfort



Human (or User-) Centered Design

• A design philosophy, or methodological principle, that centers the design process around the user.

• Three important attributes:1. Focus on the roles of humans in complex systems2. Design objectives are elaborated in terms of roles of humans3. Specific design issues follow from these objectives

• Note: No explicit focus on performance, productivity, economics, or related metrics; they subserve human-centered design issues because they are the consequence of good design rather than means for creating the design.



Human (or User-) Centered Design

• Also known as Usability Engineering.

• Four general approaches to design:1. Early focus on the user and tasks2. Empirical measurement using questionnaires, usability

studies, and usage studies focusing on quantitative performance data

3. Iterative design using prototypes, where rapid changes are made to the interface design

4. Participatory design where users are directly involved as part of the design team



Human-Centered Design: Background

• Systems have increased in size, scale, and complexity, as economic and technological necessities mandate increased performance

People will increasingly become like “cogs of machines.”




• However, automation can never be legally, ethically, and socially responsible for its actions

• Hence, regardless of systems’ scale and sophistication, humans will always have the ultimate responsibility of their operation.




• Because people will inevitably be responsible for system operation, they must…

1. …perceive the nature of these responsibilities, and

2. …have appropriate levels of authority to fulfill them.

• That is, people have to be “in charge.”



Human-Centered Design Objectives

• Design objectives should be to support humans to achieve the operational objectives for which they are responsible

• Hence, the purpose of a pilot is – not to fly the airplane that takes people from point A to point B;– instead, the purpose of the airplane is to support the pilot, whose

responsibility is to take people from A to B.

• The purpose of factory workers is – not to staff a factory designed to achieve some productivity goals;– instead, the purpose of the factory is to support the workers who are

responsible for achieving productivity goals.

• What is the purpose of the Illinois Compass?



Human-Centered Design

• Objectives: Human centered design should…1. …enhance human abilities2. …help overcome human limitations3. …foster user acceptance

• Design Issues: 1. Formulate the right problem -- make sure that system objectives

and requirements are right2. Design an appropriate solution -- excellence in engineering is

necessary but not sufficient to assure that system design is successful

3. Develop the solution to perform well -- operability, maintainability, supportability

4. Assure user satisfaction



Barriers to Human-Centered Design

• Miscommunication– For example: A user “…wanted an operating system that

was stable, up-to-date, and easy to use, one that would enable me to get my work done effectively. It is not clear that my goals were shared by those who created the installation process for the OS upgrade. Instead, they seem to have focused on creating an appealing interface; a simple, automatic installation process; and a means of taking every opportunity to inform the user of all the great system features.”

Alan Hedge in Ergonomics in Design, 11(3), 4-9

• Technology-driven design



Miscommunication in Design

As proposed by the project sponsor

As specified in the project request

As designed by the senior analyst

As produced by the programmers

As installed at the user's site

What the user wanted



Technology-Driven DesignPerformance/productivity

requirementsPerformance/productivity

requirementsTechnological “opportunities”Technological “opportunities”

Designers infuse technology

Designers infuse technology

Maintainers overwhelmed by technology

Maintainers overwhelmed by technology

Managers overwhelmed by Data

Managers overwhelmed by Data

Operators overwhelmed by alternatives

Operators overwhelmed by alternatives

Designers produce technology ‘fixes’

Designers produce technology ‘fixes’

Increased system complexity

Increased system complexity

Increased mis- and disuse of system

Increased mis- and disuse of system

Increased automationIncreased automation

Performance/ productivity shortfallPerformance/ productivity shortfall



Human Factors in Systems’ Development

Operational RequirementsOperational

RequirementsFunctional

RequirementsFunctional

RequirementsSystem

FunctionsSystem

FunctionsFunctional

LogicFunctional

LogicUser

InterfaceUser

Interface

Marketing,Customers

Engineering, HumanFactors

Human Factors

User goals, existing mental models, task- and environment analyses



Systems Development Life Cycle

• Stage 1: Front-End Analysis• Stage 2: Conceptual Design• Stage 3: Iterative Design and Testing• Stage 4: Design of Support Materials• Stage 5: System Production• Stage 6: Implementation and Evaluation• Stage 7: System Operation and Maintenance• Stage 8: System Disposal



…and Associated Human Factors Activities

• Stage 1: Front-End Analysis– User Analysis– Preliminary Task Analysis– Environment Analysis– Identification of User Preferences and Requirements– Input for System Specifications– Make sure objectives and functions match user requirements– Provide ergonomic criteria

• Stage 2: Conceptual Design– Function Allocation– Support for the Conceptual Design Process




• Stage 3: Iterative Design and Testing– Task Analysis– Interface Design– Prototype Development– Heuristic Evaluation (Design Review)– Cost-Benefit Analyses– Trade-off Analyses– Workload Analysis– Simulations and Modeling– Safety Analysis– Usability Testing




• Stage 4: Design of Support Materials– Develop and provide input for support materials

• Stage 5: System Production

• Stage 6: Implementation and Evaluation– Evaluate Fielded System

• Stage 7: System Operation and Maintenance– Monitor System Performance Over Time

• Stage 8: System Disposal



Task Analysis: Definitions

• A collection of techniques for describing how people interact with systems.

• A way of systematically describing a system to better understand how to match the demands of the system to human capabilities

• Function analysis is similar to task analysis except that it focuses on basic functions that will be performed by the human-machine system. – A function is generally composed of many tasks.



Task Analysis

• Task analysis seeks to identify…– User goals and their associated activities

– Tasks and subtasks necessary to achieve the goal

– Conditions under which tasks are performed

– Outcomes of the tasks and subtasks

– Information or knowledge needed to perform the tasks and subtasks

– Communication with others for performing the tasks

– Equipment needed to perform the tasks



Task Analysis (cont’d)

…for– Training requirements– Interface requirements– Process redesign– System reliability– Staffing requirements– Workload estimation

…using methods for– Data collection– Data representation (lists, outlines, and matrices)– Data analysis and modeling– Observable (physical) and unobservable (cognitive) activity



Task Analysis: Examples

• General methods for data collection– Document and equipment analysis– Unstructured and structured interviews– Group interviews and focus groups– Sorting and rating– Observation– Verbal protocol analysis– Questionnaires

• General methods for data representation– Lists and outlines– Matrices (cross-tabulation tables)– Structural and hierarchical networks– Flow- and time-line charts



Task Analysis: Examples

• Specific task analysis methods– Controls and displays analysis– Hierarchical Task Analysis (HTA)– The GOMS model– Critical incident technique– Conceptual Graph Analysis– Activity Sampling– Operational Sequence Diagram

– Cognitive Task Analysis (CTA)



Usability

• User Interface– How the product looks to the user?

• Functional Logic– How the product does what it does?

• Functions– What the product does.

…All should be transparent to the user!

Provide a good conceptual model Make things visible Use natural mappings Provide feedback



Usability Testing

• Definition (and quantification!) of usability– Learnability– Efficiency– Memorability– Errors– Satisfaction



Summary

•‘Know• Thy • User’



Human Factors Evaluation of Compass

• Evaluation of Compass was carried out as a classroom project in the AVI/PSYC 358/IE 340 ‘Human Factors’ course

• Students were encouraged to supply anecdotal usability data by posting their experiences to a Compass discussion forum

• A group of students evaluated Compass and suggested areas for redesign as a semester project.

• Despite the hours expended by participating students, this evaluation was far from thorough and only anecdotal data could be gathered.



Human Factors Issues of Compass

• Compass is a feature-rich system and consequently quite complex

• Complexity results in– Increased learning time and training needs,

particularly with large and constant proportion of novice users

– Under- or disuse of many potentially useful features– Decrease in user acceptance and satisfaction



Selected User Comments

• ‘I used the address book to send e-mail to the class. At first, I thought "save" meant that it would save my selections (as it does in Express webmail), but it displayed the addresses in my "To" box. Because of this, I assumed that I could check any boxes on any page of the address book, and it would all show up as when I hit "Save". I found out that it only does this for one page at a time.’

• ‘There isn't a link to the Illinois Compass calendar from the web page of each course. The only link to the calendar is available at the Illinois Compass Home page. Therefore, if you are in a course home page and you want to use the calendar you have to return to the Illinois Compass Home page each time you want to refer to the calendar.‘




• ‘On the main page of compass, it only notifies you of an update if there is a new email, new discussion post, or new grade (there might be more). So if an instructor for the class adds a new assignment or changes anything inside the class subsection, you won't know unless you explore it. This …means you have to search through each class if you want to make sure you haven't missed something. And if you don't have time, you might miss something important.’




• ‘This happens every time I use the discussion board. Whenever I click on a post that I want to read, it opens it up in a new window. Once it opens, it makes me re-login to compass in the new window. This then brings me back to the main compass page where I have to find the post that I wanted to look at. Once I click it, it opens yet another new window with the post, not requiring me to re-login this time. Essentially I end up with three compass windows. I'm not sure if this is because of my browser, or some setting within my browser. Perhaps they should just have posts viewable without the need open a completely new window.’




• ‘Since a lot of my classes have used compass previously, I have a cluttered up list where I have to search each time I want to find the class I'm looking for. Perhaps it would be even better if we were able to sort the list of classes in some way.’

• ‘When teaching assistants are in a "student view" screen, the user-friendly link to the Illinois Compass Home does not appear on the top right of the screen, which makes getting back to the Home confusing. Ultimately, one must remember to go back to the "build" view for the link.’

• ‘When trying to save lecture notes, you cannot right click and "save target as...". Thus, if your browser is configured to automatically open the powerpoint slides in Explorer, you are not able to save the file or change printing options.’




• ‘I had trouble figuring out how to forward the mail from Compass into my other account. When I got into the message center, I was looking for text similar to "E-mail options" that would allow me to forward my mail. I couldn't find anything like that, so I started browsing the page, looking at the options under the dropdown menus, and clicking the "Options” button, unable to find that option. I eventually used the "Search this tool:” to find out how to forward my mail.’




• ‘Yet another fault of the Illinois Compass system is its poor label design. As denoted by Wickens et. al in Ch. 8 (p.193-4), effective labels are: visible, discriminable, meaningful, and have proper location. In this courseware, however, few labels contain all/most of these features. For instance, why are the important labels of: grades, assignments, and discussions clumped together (lack of discriminability) and hidden under “More Tools” (not visible) whereas “Local Content,” which leads to nowhere if clicked (meaningless), has its own label in plane site on the page?’



General Human Factors Issues

• No link from the UIUC home page; alternatives:– Type the url: https://compass-portal.cites.uiuc.edu/login.html– Search for Illinois Compass

• Incompatibility with browsers; MS Internet Explorer seems to have lots of problems, yet it is the most popular browser (84%)

• Inconsistent functional logic between Compass and web browsers; yet, users expect browser-like controls



Specific Issues: Slowness

• Because Illinois Compass has been implemented using the Java programming language, slow loading time can be expected.

• However, this is heavily dependent on the hardware of the user (e.g., amount of RAM, processor speed), specific browser (IE, Netscape, Firefox, Safari), and internet connection (e.g., dial-up modem, cable).

• An increase in user connections to Illinois Compass causes bottlenecks in the network, which also degrades performance.



Slowness: Example

• Using a dial-up modem:– Time to load home page (from authentication): 1 min 40 s– Time to load a course home page: 3 min 05 s– Time to go to ‘Teach’ page/mode: 1 min 29 s– Time to bring up Grade Book: 3 min 28 s– Time to remove a student and return to main: 3 min 20 s– Time to return to ‘Build’ page: 1 min 17 s– Time to go to ‘Lecture Notes’ folder: 1 min 27 s– Time to upload a PowerPoint file (248 KB): 6 min 12 s– Total 21 min 58 s

• Most of the above times were spent on waiting for pages to load and reload.



Specific Issues: Navigation Problems

• Normal browser functions do not work with the Java-based application, e.g., back-button, shortcut keys; this is contrary to user expectations and leads to recurrent errors.

• Button labels are vague or misleading– Link ‘Illinois Compass Home’ goes to class list; the flower icon links

to CITES page about Compass, not to Compass home page

• Button locations are non-intuitive and inconsistent • Lack of a site map• Even simple actions require navigation through multiple pages.• Slowness of the system is exacerbated by clumsy navigation,

requiring reloading pages after majority of actions.



Specific Issues: Grade Book

• The grade formula cannot be easily edited; adding new graded work often requires retyping the entire formula

• Many useful features from the old electronic gradebook did not make it to Compass



E-mail Forwarding and Replying

• Received:

From: Student [AVI 497 - Special Topics in Aviation - Spring 2005 - Section ER]To: Rantanen, Esa CC: BCC: Subject: Class Sent: Wednesday, March 30, 2005

• Reply:

To: [email protected]

• E-mail forwarding must be set separately for each class student is taking



Suggestions for Improvement

• Overall goal: Reduce navigation through multiple pages when performing simple actions

• Specific suggestions:1. Place Compass website login link on the University website,

www.uiuc.edu The link should take a student or faculty member directly to the login page to save time clicking “Login” multiple times.

2. Clearly differentiate links to Compass home page (class list) and specific course home page with consistent placing of the links.

3. Use of drop-down boxes containing course-specific “requests”, ordered on the left side of the toolbar, from left to right.

4. Add a “GO” button to the right of the drop-down boxes to allow for correction of errors.







Conclusion: Comprehensive Usability Analysis

• A comprehensive usability analysis is essential for successful implementation of human factors in future upgrades of the Illinois Compass.

• However, such analysis is complicated by the number of different computer platforms (PC and Mac), operating systems (versions of Windows and MAC OS), and basic performance characteristics of these (memory, processor speed).

• Furthermore, the complexity of Compass itself makes human factors analysis of the current design and contributions to redesigns very difficult, labor-intensive, and time-consuming.

• Yet, there is no alternative to thorough usability analysis and testing to ensure successful implementation and operation of the system.



Thank You!• Thanks are also due to Shanqing Yin, Jordan Clark,

Patrick Brennan, and Kevin Olson for their term project on Compass usability and to the students of AVI/PSYC 358/IE 340 during the fall semester 2004 and of AVI 447/PSYC 457 during the spring semester 2005 for their thoughtful comments and critique.

Questions?

Documents

A Human Factors Evaluation of Illinois Compass Presented at the CITES Educational Technologies Brown Bag by Esa M. Rantanen, Ph.D., on April 13, 2005