Validation of a Usability Evaluation Protocol based on

the Heuristic Inspection Method: An Experimental Case

Study in the Web Domain

Freddy Paz1, José Antonio Pow-Sang1, César Collazos2

1 Pontificia Universidad Católica del Perú, San Miguel, Lima 32, Perú

fpaz@pucp.pe, japowsang@pucp.pe 2 Universidad del Cauca, Popayán, Colombia

ccollazo@unicauca.edu.co

Abstract. The usability is nowadays an aspect that can determine the success of

a software product. For this reason, there are methods that allow specialists to

verify if the properties of the system interfaces contribute to achieving the

desired degree of usability. Although the heuristic evaluation is one of the most

recognized methods, there is not a formal process to execute this technique. The

original author only proposes general recommendations. Based on the analysis

of different studies that report the use of this method, we established a new

usability evaluation process that can be used for any professional in Computer

Sciences. This work is focused on the validation of our protocol to conduct

heuristic evaluations. For this purpose, a case study was conducted with the

participation of twenty postgraduate students. The analysis establishes that the

results are more accurate when a formal process is used.

Keywords: Human-Computer Interaction, heuristic evaluation, usability,

experimental case study, assessment protocol.

1 Introduction

Nowadays, one of the most important quality attributes of software applications is the

usability [1]. Given the current market in which there are several alternatives for a

specific type of software, if the system is hard to use, the users will look for another

application that let them to achieve their purpose easily [2]. This fact is even more

relevant for e-Commerce Web sites, where the distance to the competition is as short

that can be located by a simple search on Internet. Companies are progressively

adopting a Web model to reach more customers through the new information and

communication technologies (ICTs). In this scenario, the usability represents a crucial

role that can determine the user’s decision to use certain Web sites instead of others.

The relevance of usability has led to the development of several methods that allow

specialists to evaluate this quality aspect of any technological interface [3]. The

heuristic evaluation is one of the most widely used methods in the context of software

[4]. This recognized technique allows the identification of a high number of usability

problems by the use of minimal resources [5].

Advanced Science and Technology Letters Vol.142 (UNESST 2016), pp.63-68

http://dx.doi.org/10.14257/astl.2016.142.11

ISSN: 2287-1233 ASTL Copyright © 2016 SERSC

Although the heuristic evaluation is a common activity performed by the software

developers that follow a user-centered design, there is no agreement between the

authors on how to carry out this assessment process. The author of the technique only

establishes general recommendations about the procedure [10]. However, there is not

a specific protocol with well-defined activities that can guide step by step the usability

evaluation. In this work, we present the validation of a formal framework to conduct

heuristic evaluations in the context of software products. This usability evaluation

protocol was developed with basis on a previous systematic literature review [6].

2 Usability and Heuristic Evaluation

A general definition of usability is proposed by the standard ISO 9241-11 [7], which

establishes that this feature is related to “the extent to which a product can be used by

specific users to achieve specific goals with effectiveness, efficiency, and satisfaction

in a specified context of use”. This standard establishes the interface must be intuitive

in a way that allows users through the use of the system, to reach the desired goal.

In order to verify if a software system meets an appropriate degree of usability,

several methods were developed. These evaluation techniques allow specialists to

determine systematically if the properties of the system interface contribute to achieve

the desired usability in the software product [8]. The heuristic evaluation is a usability

inspection method that is widely used during the software development process. This

technique involves having a group of specialist who examines the interfaces of the

system to conclude if they were properly designed [5]. This inspection is guided by a

set of principles called “heuristics”. The evaluators verify if there are infringements to

these guidelines, and the results are cataloged as usability problems.

2.1 Usability Evaluation Protocol

The author of the heuristic evaluation method only provides general recommendations

to carry out this technique [10]. For this reason, we proposed a formal process based

on a previous work [6], which involved the conduction of a systematic mapping

review of the usability evaluation methods that were used during the last three years

to develop software products. From these results, only those studies that reported the

use of the heuristic evaluation method were analyzed. According to the papers, the

process can be divided into five phases. This proposal also considers the framework

established by Hurtado et al. [9] for usability inspections:

1. Planning: In this phase, the evaluation manager must choose the specialists

that will examine the system interfaces. This person as well has to decide the number

of evaluators and the profile of the professionals that will participate in the inspection.

Finally, the reviewers must be recruited by these criteria.

2. Training: The phase of training is optional depending on the experience of

the reviewers. If the group of evaluators has performed few usability inspections, it

would be necessary to train them on the basic concepts of the method.

3. Evaluation: After the evaluators have defined the set of heuristics that will

be employed, they proceed to examine the system interfaces individually to identify a

Advanced Science and Technology Letters Vol.142 (UNESST 2016)

64 Copyright © 2016 SERSC

list of problems. During this time, the purpose is to determine infringements to the

principles. These issues will be cataloged as usability problems.

4. Discussion: Once the individual lists are identified, the inspection team has

to reach a consensus and prepare a single list of usability problems. The team must

organize a meeting, in which, each identified issue must be discussed.

The process of discussion to determine a single list of usability problems.

5. Report: Once the evaluators have established the single list, they have to rate

the severity and frequency of each problem individually. For this activity, the

team can use the ranking proposed by Nielsen [2]. Subsequently, the team

4.1. The moderator verbalizes the usability issue.

4.2. The evaluator who identified the issue provides more details about the

problem, even with the help of screenshots.

4.3. The team will discuss if the issue is indeed a usability problem. For this, each

evaluator will give an opinion about the identified problem.

If the group agrees that it is a usability problem then,

4.3.1. The team has to determine if the issue was identified by more than

one evaluator.

If the issue was identified by more than evaluator then,

4.3.1.1. The evaluators who identified the issue must discuss to

determine if they have indeed identified the same incidence or if

they are doing reference to different aspects.

If they are doing reference to the same issue then,

4.3.1.1.1. They must collaboratively prepare a definition and

description of the issue.

Else

4.3.1.1.2. The new incidence should be classified as

usability problem to discuss afterwards it.

4.3.1.2. The team has to determine if the issue, in the way it was

written, express indeed the incidence to which it is doing reference.

If the problem is written properly then

4.3.1.2.1. The problem is included in the final list.

Else

4.3.1.2.2. The team collaboratively must prepare a

definition and description of the identified problem.

Else

4.3.1.3. Go to step 4.3.1.2.

Else

4.3.2. The evaluator who identified the issue should decide after having

heard the different opinions.

If the evaluator persists that it is a usability issue then,

4.3.2.1. Go to step 4.3.1.2.

Else

4.3.2.2. The problem is discarded.

Advanced Science and Technology Letters Vol.142 (UNESST 2016)

Copyright © 2016 SERSC 65

must calculate the average of this rates. Finally, the team must prepare a report

in which they should include possible solutions to each identified problem.

3 Experimental Case Study

The validation of the protocol was performed with the participation of twenty post-

graduate students from Pontifical Catholic University of Peru (PUCP), specifically

from the Master Program of Informatics Engineering. They were selected from a

mandatory course of Human-Computer Interaction. In order to obtain the validation of

the new formal protocol, the students were requested to participate as evaluators in the

execution of a heuristic inspection. Given that one of the goals of the course was the

instruction of this method, they were trained in the process. For this activity, they

were divided into four groups of five students. Two groups were trained according to

the original recommendations proposed by Nielsen [10]. The other two groups were

trained with basis on our new evaluation approach. Two e-commerce Web sites were

selected: latam.com (Web site of an airline) and aliexpress.com (website of a retail

store). Teams A and C performed the heuristic inspection of this Web application

employing the ten usability principles for user interface design [11]. In contrast,

groups B and D organized a meeting to determine the set of usability that they should

use for the evaluation. They concluded that an appropriate set of heuristics was the

fifteen principles proposed by Paz et al. [12] for transactional Web sites. The

distribution for this experimental case study is presented in Table 1.

Table 1. Instrument-Subject Distribution for the Heuristic Evaluations

Evaluation

Team

Web Site that was

assigned Evaluation Process Usability Heuristics

Group A www.latam.com Nielsen’s traditional

approach Nielsen’s heuristics

Group B www.latam.com New Evaluation Protocol Paz’s heuristics

Group C www.aliexpress.com Nielsen’s traditional

approach Nielsen’s heuristics

Group D www.aliexpress.com New Evaluation Protocol Paz’s heuristics

4 Analysis of the Results

Both approaches were compared by the number of usability issues that were identified

in each scenario. The results are presented in Table 2, and establish that when our new

proposal is used to carry out a heuristic evaluation, more issues are detected. In

addition, the severity of the problems is higher, considering that this value varies

between 0 and 4, where the maximum value of four represents a catastrophic problem.

The protocol allows specialists to conduct the assessment in a more structured way, in

which all findings are discussed to the obtain accurate results. The establishment of

the proper heuristics is another factor that can modify the outcomes of the inspection.

Advanced Science and Technology Letters Vol.142 (UNESST 2016)

66 Copyright © 2016 SERSC

The evaluation process has been in a way that it can be used by any professional in

the field of the Computer Science. Table 3 and Table 4 show five of the most severe

problems that were only identified by those teams that used our approach. The results

establish that these issues are relevant and should not be overlooked in a usability

evaluation. However, when an evaluator only follows general recommendations to

perform the assessment, these features are not considered. The quality of the results

increases if a formal protocol is employed to carry out this type of inspection.

Table 2. Results of the heuristic evaluations

Team Number of identified problems Average of Severity of the Problems

Group A 22 2.37

Group B 37 2.65

Group C 17 2.38

Group D 28 2.80

Table 3. Five of the most severe problems that were only identified by the evaluation team B

that used our new approach

ID Problem Definition Severity

P17 The system displays a blank screen when the user logs off. 3.25

P26 The button that allows users to return to the previous step during the

payment process with VISA cancels the entire procedure. 3.25

P13 There is no help system to understand the interface design. 3.00

P35 There is unwell written information. 3.00

P19 The system does not support different Web browsers. 3.00

Table 4. Five of the most severe problems that were only identified by the evaluation team D

that used our new approach

ID Problem Definition Severity

P04 The option to log out is not intuitive. 3.3

P02 The help system displays the information only in English. 2.8

P03 There is not an option to display the prices in the local currency. 2.3

P04 The system interfaces have a different design. 2.0

P10 Many options of the system force the user to register. 2.0

5 Conclusions and Future Works

The heuristic inspection is a recognized method to evaluate the usability of software

products. However, there is not a formal procedure that allows specialists to execute

this technique in a structured way. For this reason, we have proposed a protocol that

can be used by software developers and professionals in the field of Computer

Science to conduct a usability evaluation based on this technique. In this work, we

present the validation of a new process to measure the degree of usability of software

Advanced Science and Technology Letters Vol.142 (UNESST 2016)

Copyright © 2016 SERSC 67

systems. For this purpose, we requested the participation of postgraduate students

from the Master Program of Informatics Engineering of the Pontifical Catholic

University of Peru, who voluntarily agreed to contribute with this research. The

results establish that more usability problems can be identified when a particular

procedure is used. The issues that were detected by the teams who employed the new

approach are relevant. These findings may be due to the fact new usability principles

were selected to carry out the evaluation. After a meeting, the teams agreed that the

Nielsen’s ten traditional heuristics fail to cover new aspects of the current e-

commerce Web sites such as real-time processing and sophisticated designs. For this

reason, they chose to use another heuristic proposal. Although the results are further

accurate when a formal protocol is followed, it is still necessary to perform more case

studies to demonstrate its usefulness and applicability in other contexts and scenarios.

References

1. Dhouib, A., Trabelsi, A., Kolski, C., Neji, M.: A classification and comparison of usability

evaluation methods for interactive adaptive systems. In: Proceedings of the 9th

International Conference on Human System Interactions (HSI 2016), pp. 246-251,

Portsmouth, United Kingdom (2016)

2. Nielsen, J.: Usability Engineering. Academic Press, San Diego, CA, USA (1993)

3. Paz, F., Pow-Sang, J.A.: Current Trends in Usability Evaluation Methods: A Systematic

Review. In: Proceedings of the 7th International Conference on Advanced Software

Engineering and Its Applications (ASEA 2014), pp. 11-15, Hainan, China (2014)

4. Paz, F., Villanueva, D., Rusu, C., Roncagliolo, S., Pow-Sang, J.A.: Experimental

Evaluation of Usability Heuristics. In: Proceedings of the 10th International Conference on

Information Technology: New Generations (ITNG 2013), pp. 119-126, Las Vegas, NV,

USA (2013)

5. Nielsen, J.: Usability inspection methods. In: Proceedings of the Conference Companion

on Human Factors in Computing Systems (CHI '94), pp. 413-414, Boston, Massachusetts,

USA (1994)

6. Paz, F., Pow-Sang, J.A.: A Systematic Mapping Review of Usability Evaluation Methods

for Software Development Process. International Journal of Software Engineering and Its

Applications. 10(1), 165 - 178 (2016)

7. ISO: Ergonomic requirements for office work with visual display terminals (VDTs). ISO

IEC 9241-11, International Organization for Standardization, Geneva, Switzerland (1998)

8. Fernandez, A., Abrahão, S., Insfran, E., Matera, M.: Further analysis on the validation of a

usability inspection method for model-driven web development. In: Proceedings of the 6th

International Symposium on Empirical Software Engineering and Measurement (ESEM

2012), pp. 153-156, Lund, Sweden (2012)

9. Hurtado, N., Ruiz, M., Orta, E., Torres, J.: Using simulation to aid decision making in

managing the usability evaluation process. Information and Software Technology 57, 509-

526 (2015)

10. Nielsen, J.: How to conduct a heuristic evaluation, (1995)

https://www.nngroup.com/articles/how-to-conduct-a-heuristic-evaluation/

11. Nielsen, J.: 10 Usability Heuristics for User Interface Design, (1995)

https://www.nngroup.com/articles/ten-usability-heuristics/

12. Paz, F., Paz, F. A., Pow-Sang, J.A., Collantes, L.: Usability Heuristics for Transactional

Web Sites. In: Proceedings of the 11th International Conference on Information

Technology: New Generations (ITNG 2014), pp. 627-628, Las Vegas, NV, USA (2014)

Advanced Science and Technology Letters Vol.142 (UNESST 2016)

68 Copyright © 2016 SERSC