Bevan, N. & Spinhof, L. (2007) Are guidelines and standards for web usability comprehensive? Proceedings HCI International 2007. Springer

Are guidelines and standards for web usability comprehensive?

Nigel Bevan1, Lonneke Spinhof2

1Professional Usability Services, 12 King Edwards Gardens, London W3 9RG, UK 2Centre for Usability Research-K.U.Leuven, Parkstraat 45 bus 3605, 3000 Leuven, Belgium

nigel@nogelbevan.com, lonneke.spinhof@soc.kuleuven.be

A previous paper compared the 110 guidelines in ISO CD 9241-151 with the 187 guidelines produced by the U.S. Department of Health and Human Services (HHS) and found that 76% of the HHS guidelines and 54% of the ISO guidelines were unique. New versions of both the original 2004 documents were issued in 2006, but 71% of the HHS guidelines and 46% of the ISO guidelines are still unique. Neither set of guidelines is easy to use for an expert review of whether a web site complies with the guidelines. A more comprehensive checklist has been developed, based on the HHS and ISO guidelines, but extended to include additional research-based guidelines on privacy and security and e-commerce. It is complemented by a handbook describing each guideline in more detail, illustrated with an example, and with an explanation of how it should be tested and when compliance can be stated.

Comparison of ISO and HHS Guidelines

A previous paper at HCI International 2005 [2] compared guidelines for web usability including:

• The 187 guidelines for effective web design and usability for information-oriented sites produced by the U.S. Department of Health and Human Services (HHS).

• The draft International Standard ISO/CD 9241-151 that contained 110 recommendations for the user-centered design of web user interfaces.

In 2006, these documents were superceded by: • Revised guidelines from HHS [14], including 22 new guidelines, giving a

total of 207 guidelines. • The draft International Standard ISO/DIS 9241-151 that now contains 141

recommendations for the user-centered design of web user interfaces. The revised HHS guidelines are derived from about 500 cited publications. The

guidelines include ratings for importance by 36 website professionals, and are categorized based on a card-sorting exercise by 20 website designers. They cover a wide range of web site design issues, including accessibility, home page design, page

2 Nigel Bevan1, Lonneke Spinhof2

and site navigation, graphics and images, web content organization and effective web content writing. Each guideline contains:

• A brief statement of the overarching principle that is the foundation of the guideline.

• Comments that further explain the research/supporting information. • Citations to relevant web sites, technical and/or research reports supporting

the guideline. • A score indicating the "Strength of Evidence" that supports the guideline.

These range from "Strong Research Support," indicating that there is at least one formal, rigorous study with contextual validity and agreement among experts to "Weak Research Support," indicating limited evidence and disagreement among experts.

• A score indicating the "Relative Importance" of the guideline to the overall success of a web site. These scores range from 1-5 and are intended to help guide usability experts and web designers to prioritize the implementation of these guidelines.

• One or more graphic examples of the guideline in practice. ISO is developing an International Standard to provide recommendations for the

user-centered design of web user interfaces. The recommendations cover much the same scope as HHS, but are documented in a more concise format appropriate for an international standard.

The ISO document distinguishes between design, process and evaluation aspects of web development. However, since the development process and evaluation is already covered by other ISO standards, it focuses on the design aspects, and provides design guidance and recommendations in four major areas:

• High-level design decisions and design strategy. • Content design. • Navigation and search. • Content presentation.

ISO 9241-151 primarily contains material that is unique to the web, so some topics covered by HHS are omitted from ISO 9241-151 as they are covered by more general ISO standards, in particular:

• Design Process and Evaluation: ISO 13407, ISO TR 16982, and ISO 9241-11. • Accessibility: ISO TS 16071 and WAI Guidelines. • Lists: partly covered by ISO 9241-12. • Screen-based Controls (Widgets): ISO 14915-2. • Graphics, Images, and Multimedia: ISO 14915-3.

This means that for complete guidance on the web, readers have to acquire additional standards and identify the parts that are relevant. This is not easy to do, particularly as some interpretation is needed to apply the material in other standards to the web.

The previous paper showed that 76% of the HHS guidelines and 54% of the ISO guidelines were unique.

In 2006, new versions were published of both the HHS and ISO guidelines. The new HHS document has added 22 new guidelines and updated 30 more, and the ISO document has been extensively revised with 31 additional recommendations.

Are guidelines and standards for web usability comprehensive? 3

The 196 HHS guidelines (excluding usability testing) and 141 ISO recommendations are listed in Table 1, showing the ISO topics that appear to be most closely equivalent to each HHS guideline (or category of guideline). Partially corresponding ISO guidelines are shown in italics. Apparently conflicting guidelines are shown in bold.

Table 1. Comparison of HHS and ISO guidelines

* = HHS importance rating * HHS Guideline ISO 9241-151 Recommendation Design Process and Evaluation

5 1:1 Provide Useful Content 7.1.3 Appropriateness of content for the target group and tasks 7.1.4 Completeness of content

5 1:2 Establish User Requirements 6.3 Analysing the target user groups

5 1:3 Understand and Meet User’s Expectations 7.1.2 Designing the conceptual model 7.1.5 Structuring content appropriately 8.3.2 Choosing suitable navigation structures

5 1:4 Involve Users in Establishing User Requirements

4 1:5 Set and State Goals 6.2 Determining the purpose of a Web application 6.5 Matching application purpose and user goals

4 1:6 Focus on Performance Before Preference 4 1:7 Consider Many User Interface Issues 4 1:8 Be Easily Found in the Top 30 3 1:9 Set Usability Goals 2 1:10 Use Parallel Design 1 1:11 Use Personas 6.4 Analysing the users’ tasks 6.6 Recognising the purpose of a Web application 6.7 Prioritising different design goals 6.11 Coherent multi-site strategy 7.2.2 Independence of content, structure & presentation 9.3.5 Visualising temporal status 9.3.12 Consistency across related sites 10.6 Using generally accepted technologies & standards 10.7 Making Web user interfaces robust 10.8 Designing for input device independence Optimizing the User Experience 5 2:1 Do Not Display Unsolicited Windows or Graphics 8.3.11 Avoiding opening unnecessary windows 4 2:2 Increase Web Site Credibility 9.6.4 Text quality 4 2:3 Standardize Task Sequences 4 2:4 Reduce the User’s Workload 4 2:5 Design For Working Memory Limitations 4 2:6 Minimize Page Download Time 10.5 Acceptable download times 4 2:7 Warn of ’Time Outs’

4 2:8 Display Information in a Directly Usable Format 10.1.4 Using appropriate formats, units of measurement or currency.

4 2:9 Format Information for Reading and Printing 4 2:10 Provide Feedback when Users Must Wait 4 2:11 Inform Users of Long Download Times 4 2:12 Develop Pages that Will Print Properly 3 2:13 Do Not Require Users to Multitask While Reading 3 2:14 Use Users’ Terminology in Help Documentation 10.2 Providing help. 3 2:15 Provide Printing Options 9.3.15 Providing printable document versions 2 2:16 Provide Assistance to Users 10.3 Error pages Accessibility 5 3:1 Comply with Section 508 6.8 Conforming to content accessibility standards 5 3:2 Design Forms for Users Using Assistive Technology 5 3:3 Do Not Use Color Alone to Convey Information 9.3.9 Using Colour 4 3:4 Enable Users to Skip Repetitive Navigation Links 4 3:5 Provide Text Equivalents for Non-Text Elements 7.2.3.2 Providing text equivalents for non-text objects

4 Nigel Bevan1, Lonneke Spinhof2

4 3:6 Test Plug-Ins and Applets for Accessibility 10.9 Making the user interface of embedded objects usable and accessible

3 3:7 Ensure that Scripts Allow Accessibility 3 3:8 Provide Equivalent Pages 3 3:9 Provide Client-Side Image Maps 3 3:10 Synchronize Multimedia Elements 3 3:11 Do Not Require Style Sheets 2 3:12 Provide Frame Titles 9.3.10 Using frames with care 2 3:13 Avoid Screen Flicker 6.9 Conforming to software accessibility standards 9.6.7 Making text resizable by the user Hardware and Software 4 4:1 Design for Common Browsers 4 4:2 Account for Browser Differences 4 4:3 Design for Popular Operating Systems 4 4:4 Design for User’s Typical Connection Speed 3 4:5 Design for Commonly Used Screen Resolutions The Homepage 5 5:1 Enable Access to the Homepage 8.4.11 Linking back to the home page

5 5:2 Show All Major Options on the Homepage 8.3.9 Directly accessing relevant information from the home page

5 5:3 Create a Positive First Impression of Your Site 4 5:4 Communicate the Web Site’s Value and Purpose 8.3.8 Informative home page 4 5:5 Limit Prose Text on the Homepage 4 5:6 Ensure the Homepage Looks like a Homepage 3 5:7 Limit Homepage Length 2 5:8 Announce Changes to a Web Site 9.3.4 Recognising new content 2 5:9 Attend to Homepage Panel Width 8.3.10.1 Avoiding unnecessary start (splash) screens 8.3.10.2 Skipping splash screens Page Layout 5 6:1 Avoid Cluttered Displays

5 6:2 Place Important Items Consistently 9.3.2 Consistent page layout 9.3.3 Placing title information consistently

5 6:3 Place Important Items at Top Center 9.3.7 Avoiding scrolling for important information 4 6:4 Structure for Easy Comparison 4 6:5 Establish Level of Importance 4 6:6 Optimize Display Density 4 6:7 Align Items on a Page 3 6:8 Use Fluid Layouts 3 6:9 Avoid Scroll Stoppers 3 6:10 Set Appropriate Page Lengths 9.6.5 Quantity of text per information unit/page 3 6:11 Use Moderate White Space 9.3.16 Use of “white space” 2 6:12 Choose Appropriate Line Lengths

1 6:13 Use Frames When Functions Must Remain Accessible

9.3.11 Providing alternatives to frame-based presentation

9.3.6 Making content fit the expected size of the display area

9.3.13 Using appropriate techniques for defining the layout of a page

Navigation 4 7:1 Provide Navigational Options

4 7:2 Differentiate and Group Navigation Elements 8.4.3 Maintaining visibility of navigation links 8.4.5 Placing navigation components consistently 8.4.7 Splitting up navigation overviews

4 7:3 Use a Clickable ’List of Contents’ on Long Pages 8.4.14 Subdividing long pages

4 7:4 Provide Feedback on Users’ Location 8.2.2 Showing users where they are 8.4.4 Consistency between navigation components and content 10.4 Naming of URLs

4 7:5 Place Primary Navigation Menus in the Left Panel 3 7:6 Use Descriptive Tab Labels 3 7:7 Present Tabs Effectively 2 7:8 Keep Navigation-Only Pages Short

Are guidelines and standards for web usability comprehensive? 5

2 7:9 Use Appropriate Menu Types 2 7:10 Use Site Maps 8.4.8 Providing a site map 1 7:11 Use ’Glosses’ to Assist Navigation 9.4.5 Inferring the link target from link cues

1 7:12 Breadcrumb Navigation 8.2.2 Showing users where they are 8.4.12 Going back to higher levels

8.2.1 Making navigation self-descriptive. 8.2.3 Supporting different navigation behaviours. 8.3.3 Breadth versus depth of the navigation structure 8.3.4 Organising the navigation in a meaningful manner 8.3.7 Superimposing different navigation structures 8.4.6 Making several levels visible 8.4.10 Making dynamic navigation components obvious 8.4.13 Providing a 'step back' function Scrolling and Paging 5 8:1 Eliminate Horizontal Scrolling 9.3.8 Avoiding horizontal scrolling 2 8:2 Facilitate Rapid Scrolling While Reading 2 8:3 Use Scrolling Pages for Reading Comprehension 2 8:4 Use Paging Rather Than Scrolling 2 8:5 Scroll Fewer Screenfuls Headings, Titles and Labels 5 9:1 Use Clear Category Labels 4 9:2 Provide Descriptive Page Titles 9.4.17 Page titles as bookmarks 4 9:3 Use Descriptive Headings Liberally 9.3.1 General page information 4 9:4 Use Unique and Descriptive Headings 8.2.2 Showing users where they are 4 9:5 Highlight Critical Data 4 9:6 Use Descriptive Row and Column Headings 3 9:7 Use Headings in the Appropriate HTML Order 2 9:8 Provide Users with Good Ways to Reduce Options Links 5 10:1 Use Meaningful Link Labels 9.4.7 Using descriptive link labels 4 10:2 Link to Related Content 4 10:3 Match Link Names with Their Destination Pages 4 10:4 Avoid Misleading Cues to Click

4 10:5 Repeat Important Links 8.2.4 Offering alternative navigation paths 8.4.9 Providing cross linking to potentially relevant content 9.4.15 Redundant links

4 10:6 Use Text for Links 4 10:7 Designate Used Links 9.4.8 Highlighting previously visited links

3 10:8 Provide Consistent Clickability Cues 9.4.2 Identification of links 9.4.3 Distinguishing adjacent links from each other

3 10:9 Ensure that Embedded Links are Descriptive 3 10:10 Use ’Pointing-and-Clicking’ 3 10:11 Use Appropriate Text Link Lengths 9.4.14 Link length 3 10:12 Indicate Internal vs. External Links 9.4.13 Distinguishable within-page links 3 10:13 Clarify Clickable Regions on Images 3 10:14 Link to Supportive Information 8.4.16 Dead links 9.4.4 Distinguishing navigation links from transactions 9.4.9 Marking links to special targets 9.4.11 Marking links opening new windows 9.4.12 Distinguishing navigation links from action links 9.4.16 Avoiding link overload Text Appearance 4 11:1 Use Black Text on Plain, High-Contrast Backgrounds 9.3.9 Using Colour 4 11:2 Format Common Items Consistently 4 11:3 Use Mixed-Case for Prose Text 4 11:4 Ensure Visual Consistency 3 11:5 Use Bold Text Sparingly 3 11:6 Use Attention-Attracting Features when Appropriate 3 11:7 Use Familiar Fonts 3 11:8 Use at Least a 12-Point Font 2 11:9 Color-Coding and Instructions 2 11:10 Emphasize Importance

6 Nigel Bevan1, Lonneke Spinhof2

2 11:11 Highlighting Information Lists 4 12:1 Order Elements to Maximize User Performance [ISO 9241-12 5.7.1] 4 12:2 Place Important Items at Top of the List 4 12:3 Format Lists to Ease Scanning 4 12:4 Display Related Items in Lists 3 12:5 Introduce Each List 3 12:6 Use Static Menus 2 12:7 Start Numbered Items at One [ISO 9241-12 5.7.6] 2 12:8 Use Appropriate List Style 1 12:9 Capitalize First Letter of First Word in Lists Screen-Based Controls (Widgets) 5 13:1 Distinguish Required and Optional Data Entry Fields 5 13:2 Label Pushbuttons Clearly 4 13:3 Label Data Entry Fields Consistently 4 13:4 Do Not Make User-Entered Codes Case Sensitive 4 13:5 Label Data Entry Fields Clearly 4 13:6 Minimize User Data Entry 3 13:7 Put Labels Close to Data Entry Fields 3 13:8 Allow Users to See Their Entered Data 8.5.2.8 Search field size 3 13:9 Use Radio Buttons for Mutually Exclusive Selections 9.5 Choosing interaction objects 3 13:10 Use Familiar Widgets 3 13:11 Anticipate Typical User Errors 3 13:12 Partition Long Data Items 3 13:13 Use a Single Data Entry Method 3 13:14 Prioritize Pushbuttons 3 13:15 Use Check Boxes to Enable Multiple Selections 3 13:16 Label Units of Measurement 3 13:17 Do Not Limit Viewable List Box Options 3 13:18 Display Default Values 2 13:19 Place Cursor in First Data Entry Field 2 13:20 Ensure that Double-Clicking Will Not Cause Problems 2 13:21 Use Open Lists to Select One from Many 2 13:22 Use Data Entry Fields to Speed Performance 2 13:23 Use a Minimum of Two Radio Buttons 2 13:24 Provide Auto-Tabbing Functionality 1 13:25 Minimize Use of the Shift Key 8.4.15 Explicit activation Graphics, Images, and Multimedia 4 14:1 Use Simple Background Images 4 14:2 Label Clickable Images 4 14:3 Ensure that Images Do Not Slow Downloads

4 14:4 Use Video, Animation, and Audio Meaningfully 7.2.3 Selecting suitable media 7.2.3.1 Selecting appropriate media objects

4 14:5 Include Logos 6.10 Identifying the site and its owner 9.3.14 Identifying all pages of a site

4 14:6 Graphics Should Not Look like Banner Ads 4 14:7 Limit Large Images Above the Fold 4 14:8 Ensure Web Site Images Convey Intended Messages 3 14:9 Limit the Use of Images 3 14:10 Include Actual Data with Data Graphics 3 14:11 Display Monitoring Information Graphically

2 14:12 Introduce Animation 7.2.3.3 Enabling users to control time-dependent content changes

2 14:13 Emulate Real-World Objects 2 14:14 Use Thumbnail Images to Preview Larger Images 1 14:15 Use Images to Facilitate Learning 1 14:16 Using Photographs of People Writing Web Content

5 15:1 Make Action Sequences Clear 8.3.5 Offering task-based navigation 8.3.6 Offering clear navigation within multi-step tasks 8.3.7 Superimposing different navigation structures 8.4.2 Providing navigation overviews

4 15:2 Avoid Jargon

Are guidelines and standards for web usability comprehensive? 7

4 15:3 Use Familiar Words 9.4.6 Using familiar terminology for navigation links 4 15:4 Define Acronyms and Abbreviations 4 15:5 Use Abbreviations Sparingly 4 15:6 Use Mixed Case with Prose 4 15:7 Limit the Number of Words and Sentences 3 15:8 Limit Prose Text on Navigation pages 3 15:9 Use Active Voice 3 15:10 Write Instructions in the Affirmative 3 15:11 Make First Sentences Descriptive 9.6.1 Readability of text 9.6.2 Supporting text skimming 9.6.3 Writing style Content Organization 5 16:1 Organize Information Clearly 5 16:2 Facilitate Scanning 9.6.2 Supporting text skimming 5 16:3 Ensure that Necessary Information is Displayed 4 16:4 Group Related Elements 4 16:5 Minimize the Number of Clicks or Pages 8.2.5 Minimising navigation effort 3 16:6 Design Quantitative Content for Quick Understanding 3 16:7 Display Only Necessary Information 3 16:8 Format Information for Multiple Audiences 2 16:9 Use Color for Grouping 7.1.6 Level of granularity 7.2.4 Keeping the content up to date

7.2.5 Making the date and time of the last update available

7.2.7 Accepting online user feedback Search

5 17:1 Ensure Usable Search Results 8.5.3.1 Ordering of search results 8.5.3.2 Relevance-based ranking of search results 8.5.3.3 Descriptiveness of results 8.5.3.4 Sorting search results

5 17:2 Design Search Engines to Search the Entire Site 8.5.4.1 Scope of a search 8.5.4.2 Selecting the scope of a search

4 17:3 Make Upper- and Lowercase Search Terms Equivalent 4 17:4 Provide a Search Option on Each Page 8.5.2.7 Availability of search 4 17:5 Design Search Around Users’ Terms 8.5.2.10 Error-tolerant search 3 17:6 Allow Simple Searches 8.5.2.3 Providing a simple search facility 3 17:7 Notify Users when Multiple Search Options Exist 3 17:8 Include Hints to Improve Search Performance 8.5.2.6 Describing the search technique used 2 17:9 Provide Search Templates 8.5.2.1 Providing a search function 8.5.2.2 Providing appropriate search functions 8.5.2.4 Advanced search 8.5.2.5 Full-text search 8.5.2.9 Shortcut to search function

8.5.4.3 Providing feedback on the volume of the search result

8.5.4.5 Showing the query with the results 8.5.5.1 Giving advice for unsuccessful searches 8.5.5.2 Repeating searches 8.5.5.3 Refining searches Usability Testing 4 18:1 Use an Iterative Design Approach 3 18:2 Solicit Test Participants’ Comments 3 18:3 Evaluate Web Sites Before and After Making Changes 3 18:4 Prioritize Tasks 3 18:5 Distinguish Between Frequency and Severity 3 18:6 Select the Right Number of Participants 2 18:7 Use the Appropriate Prototyping Technology 2 18:8 Use Inspection Evaluation Results Cautiously 2 18:9 Recognize the ’Evaluator Effect’ 1 18:10 Apply Automatic Evaluation Methods 1 18:11 Use Cognitive Walkthroughs Cautiously

8 Nigel Bevan1, Lonneke Spinhof2

1 18:12 Choosing Laboratory vs. Remote Testing 1 18:13 Use Severity Ratings Cautiously Privacy and business policies 7.2.8.1 Providing a privacy policy statement 7.2.8.2 Providing a business policy statement 7.2.8.3 User control of personal information 7.2.8.4 Storing information on the user’s machine Internationalization 9.6.6 Identifying the language used 10.1.1 General 10.1.2 Showing relevant location information 10.1.3 Identifying supported languages 10.1.5 Presenting text in different languages Personalisation and user adaptation 7.2.9.2 Taking account of the users’ information needs 7.2.9.3 Making personalisation evident 7.2.9.4 Making user roles evident 7.2.9.5 Allowing users to see and change profiles 7.2.9.6 Informing about automatically generated profiles 7.2.9.7 Switching off automatic adaptation 7.2.9.8 Providing access to complete content

Only 56 of the HHS guidelines are in common (71% of the HHS guidelines and 46% of the ISO guidelines are unique). For the 101 HHS guidelines rated highest for importance, the proportion of unique guidelines drops to 62%. If the topics not covered by ISO 9241-151 are excluded (Design Process; Evaluation; Hardware and Software; Lists; Screen-based Controls; Graphics, Images, and Multimedia; Privacy & Business Policies; Internationalisation; and Personalisation) the percentage of unique guidelines drops to 64% of the HHS guidelines (55% of those of highest importance) and 38% of the ISO guidelines.

While the percentage of unique ISO guidelines in the new documents has reduced from 49% to 38%, the percentage of high priority HHS guidelines that are unique remains about 55% (As some judgments had to be made for what constitutes equivalence, these figures are only approximate.)

Some HHS guidelines are not in the ISO draft because they are beyond the scope of software ergonomics, e.g.:

• Hardware and Software: browser, and operating system (e.g. 4:1 Design for common browsers).

• 5:3 Create a Positive First Impression of Your Site. Other types of HHS guidelines that are not included by ISO include: • Home page design, e.g.: 5:5 Limit prose text on the homepage. • Scrolling & paging: e.g. 8:3 Use scrolling pages for reading comprehension. • Headings, Titles and Labels: window titles and descriptive headings, e.g.:

9:1 Use clear labels for categories of information that summarise the items within the category.

• Appearance, e.g.: 11:4 Ensure visual consistency of website elements within and between web pages.

• Lists: headings, ordering and formatting, e.g. 12:2 Display a series of related items in a vertical list.

• Writing Web Content: jargon, abbreviations, and case, e.g. 15:5 Use abbreviations sparingly.

Are guidelines and standards for web usability comprehensive? 9

• Content Organisation: support scanning and display necessary information, e.g.: 16:1 Organize information clearly: Structure the site to be meaningful to the user.

ISO provides more detail in areas specific to the web such as Navigation and Searching, and includes Privacy and Internationalization that are outside the scope of HHS. In total 65 guidelines are unique to ISO. Examples of apparently important guidelines within the scope of HHS, but unique to ISO include:

• 8.3.10.18.3.6 Avoiding unnecessary start (splash) screens. • 9.4.2 Navigation links should be clearly distinguishable from links activating

some action. • 9.4.9 Links to other file formats should be clearly marked. • 9.4.11 Links that open new browser windows should be clearly marked. • 10.3 Error messages should clearly state the reason why the error occurred. • 8.4.138.4.16 Provide a separate ‘back’ function if the standard function does

not lead to a meaningful previous state. These items may not have been included by HHS either because they were not

included in the original set of guidelines that were reviewed (for example because there was no supporting evidence), or because they were subsequently judged “less important” and therefore eliminated from the published set.

Some differences were noticed in the content of some HHS and ISO guidelines: • ISO 8.2.2 recommends use of breadcrumbs, while HHS 7:12 says that they

are ineffective. • ISO 9.6.5 recommends limiting the quantity of text per information unit/page,

while HHS 6:10 recommends using an appropriate page length, and using longer scrolling pages when reading for comprehension (8:3).

• ISO 9.3.11 warns against using frames, while the HHS guidelines recommend frames in some circumstances (6:13 When functions must remain accessible) and suggest how they should be used (3:12 Use frame titles).

• ISO 9.4.15 warns against using redundant links, while HHS 10:5 recommends repeating important links.

• ISO 9.4.14 recommends that link names should not exceed one line of text, while HHS 10:11 recommends that link names should be long enough to be understood, but short enough to minimize wrapping.

Guideline based inspections

Since the use of design- and usability standards in software development is rising [5] the interest in usability inspections is also rising. Previous research pointed out that the, then existing standards were not very easy for researchers and professionals to use [2,8]. Most current standards are still not readily useful in guideline reviews.

To perform a thorough guideline review on websites can involve using a combination of different sets of guidelines. But when sets are combined ad hoc they are difficult to use and to interpret.

10 Nigel Bevan1, Lonneke Spinhof2

This part of the paper describes: 1) the problems that occurred when the second author tried to use the HHS and ISO documents as a tool in guideline reviews, and 2) how the second author combined the different sets of guidelines into one checklist.

Guideline based inspections are commonly used by usability professionals in the User Centered Design process. ‘Guideline based inspections’ or ‘guideline reviews’ are considered as usability inspection methods [11]. During a usability inspection method the usability related aspects of the interface will be examined. In contrast with formal usability tests here are no end users involved, the examination is performed by some kind of professional, a usability expert, a developer, an experienced user, etc. [11]. The best-known inspection method is ‘heuristic evaluation’, in which an application is checked against a list of quite generally formulated heuristics (e.g. the ten heuristics of Nielsen). The outcome of a heuristic analysis depends highly on the interpretation of the expert that performs the inspection. A Guideline review checks an application against a more concrete set of guidelines (e.g. the HHS guidelines). The outcome of a guideline review depends less on the expertise of the expert (Jordan, e.a. 1996). Guidelines used in a guideline review can have very different abstraction levels, they can vary from quite system specific to more general guidelines that can be applied to different kind of systems. The higher the abstraction level the more insight the expert needs to have in the system [3].

Design- and usability guidelines are becoming more and more popular in the software development processes [5]. The use of and compliance with usability standards is considered as a good way to create a high degree of consistency across and within applications [5,13]. One of the benefits of using guidelines can be increased consistency. Consistency is considered as one of the most important usability principles of Human Computer Interaction. Consistency within and between applications improves the overall usability of the application. Interfaces that are consistent have a higher ease of learning, ensure a smaller number of errors and therefore a higher user satisfaction [10]). More recent sets of usability guidelines such as the HHS guidelines and the ISO 9241-151 also consider other aspects of usability like interaction design and information architecture.

To test an application for compliance with a certain set of guidelines it is useful to develop a checklist, listing all requirements the application should adhere to. A checklist helps the expert who checks for compliance to keep track of the requirements that need to be met. A good checklist consists of an exhaustive list of well-written requirements. A good requirement is necessary, verifiable, attainable and clearly formulated to avoid ambiguous statements [6]. Although usability guidelines tend to be more subjective in nature these rules for good requirements should also be kept in mind while creating a usability checklist. A good set of guidelines is composed of a combination of more specific guidelines for the application at hand and more generic guidelines that refer to more general aspects of the interface. The set of guidelines needs to be well-documented including concrete examples illustrating the different guidelines [5]. The document itself should comply with all guidelines for good document design, such as a thorough table of contents and index, word lists and glossaries [13].

Although the focus of usability guidelines may differ they are all developed to be used by developers when developing new applications and by usability experts when they are inspecting the usability of a system. Existing usability standards tend to be

Are guidelines and standards for web usability comprehensive? 11

quite unusable for developers and even for usability experts who check for compliance with these standards [13]. So users of the different available standards still need to supplement the standards to make them usable in a guideline review.

The two standards discussed in this paper are not directly usable in a guideline review. Ambiguous guideline formulations occur in both sets of guidelines, not all guidelines are verifiable and neither documents provides a ready to use checklist (see Table 2). The ISO standard contains no illustrations to serve as examples for the requirements and no index is provided. So both standards need to be supplemented to be usable as a checklist.

Table 2. Examples of guidelines transformed to make them verifiable

Original guideline from HHS: Let users know if a page is programmed to 'time out', and warn users before time expires so they can request additional time. Unambiguous requirements: Are users warned for a page time out? Can users "ask" for more time to complete a task? Original guideline from HHS: Provide users with appropriate feedback while they are waiting. Verifiable requirements: Is progress feedback provided for processes that take longer than one second? If a process takes maximum 10 seconds, is a visual indication used to indicate the progress? If a process will take more than 10 seconds, is a progress indicator used that shows progress toward completion?

Creating a web usability checklist To perform a usability guideline review on websites we needed a complete checklist covering all usability topics concerning web usability. Neither of the existing sets of guidelines was complete and detailed enough to serve the purpose: a complete conformance test of websites against the existing and commonly used web design and usability guidelines. The HHS guideline set was the most complete and almost ready to use set of guidelines we could find. Therefore we decided to take the HHS-guidelines as a basis to create the usability checklist we were looking for. We extended the scope of the HHS-guidelines with two topics: privacy and security and e-commerce taken from other guidelines found in literature (e.g. [12]). The requirements from the other topics were transformed into necessary, verifiable, attainable and unambiguous guidelines. All requirements are stated as yes/no questions. Some of the topics were combined into one topic and other topics were complemented with some extra requirements that in our experience are important in usability evaluations. All requirements in the checklist are research based. The whole set of guidelines was checked against the ISO/DIS 9241-151 to be sure we included everything from this upcoming international standard. All requirements in the ISO/DIS 9241-151 were covered in our set of guidelines but often differently formulated.

12 Nigel Bevan1, Lonneke Spinhof2

Due to the contextual nature of usability research it was not always easy to formulate requirements that are true for all websites and web applications. We therefore developed a structure where the expert who does the inspection can decide which requirements are applicable in what situation. This was the only way to create a generally applicable web usability checklist that can be adjusted to each specific situation.

Fig. 1. Example from the handbook

Use of the checklist The checklist as it is developed here is intended to be used by usability experts during guideline-reviews. As stated above a checklist alone is not enough to create a usable review-tool. All requirements in the checklist are documented in the “guideline handbook”. Each requirement in the checklist is described in more detail and when possible is illustrated with an example (see Fig, 1). For each requirement there is a description of how it should be tested and when compliance can be stated. A glossary, a table of content and an index are included to make the document usable for the user of the checklist.

Not all requirements have the same impact on the usability of an interface [1], so all requirements have been prioritized on a scale from 5 (highest priority) to 1 (lowest priority). This prioritization is based on the “relative importance” scales used in the

Are guidelines and standards for web usability comprehensive? 13

HHS-guidelines. Adjustments to this scale have been made based on the experience from former research and on feedback from a group of usability experts that used the different drafts of the checklist in usability research.

Further research The complete review-tool (handbook and checklist) need to be user-tested before it can be used as a standalone test-tool. The usability of the tool itself as well as the outcome of the test need to be evaluated. In the next phase the complete review-tool will be tested in parallel with a formal usability test on the same application. By performing such a test the results of the guideline review can be compared with the results of a formal usability test. By asking several usability experts to participate in this test as evaluators, the usability of the tool itself will be evaluated at the same time.

References

1. Agarwal, R. & Venkatesh, V.: Assessing a Firm's Web Presence: A Heuristic Evaluation Procedure for the Measurement of Usability. Info.Sys.Research, 13 (2002) 168-186

2. Bevan, N.: Guidelines and standards for web usability. Human Computer International 2005, Proceedings HCI International 2005. Lawrence Erlbaum (2005)

3. Cockton, G., Woolrych, A., Hall, L., & Hidemarch, M.: Changing Analysts' Tunes: The Surprising Impact of a New Instrument for Usability Inspection Method Assessment. Proc.HCI 2003, Springer Verslag (2003) 145-162

4. de Souza, F. & Bevan, N.: The use of guidelines in menu interface design: Evaluation of a draft standard. In Proceedings of the IFIP TC13 Third Interational Conference on Human-Computer Interaction (pp. 435-440). North-Holland Publishing Co. (1990)

5. Henninger, S., Lu, C., & Faith, C.: Using organizational learning techniques to develop context-specific usability guidelines. In (pp. 129-136). Amsterdam (1997)

6. Hooks, I.: Writing good requirements. In Proceedings of the 3rd International Symposium of theNational Council on Systems Engineering (NCOSE) (1993)

7. Jordan, W. P., Thomas, B., Weerdmeester, A. B., & McClelland, L. I.: Usability evaluation in industry. London: Taylor & Francis (1996)

8. Mosier, J. N. & Smith, S. L: Application of guidelines for designing user interface software. Behavior and Information Technology 5, 1 (1986) 39-46

9. Nielsen, J.: Usability Engineering. San Francisco, CA: Morgan Kaufmann Publishers Inc. (1993)

10. Nielsen, J.: Coordinating user interfaces for consistency. San Francisco, CA: Morgan Kaufmann Publishers Inc. (1989)

11. Nielsen, J. & Mack, L. R.: Usability inspection methods. US: John Wiley & Sons, Inc. (1994)

12. Nielsen, J., Molich, R., Snyder, C., & Farrell, S. E-Commerce User Experience. Fremont, CA, USA: Nielsen Norman Group (2001)

13. Thovtrup, H. & Nielsen, J.: Assessing the usability of a user interface standard. In (pp. 335-341). New Orleans, Louisiana, United States: ACM Press (1991)

14. U.S. Department of Health and Human Sciences: Research-Based Web Design & Usability Guidelines (2006). Available at: www.usability.gov/guidelines/